texts: Ad click prediction: what are the negative examples?

texts:
I am analysing the log of a website and I would like to build a classifier to predict the users that are likely to click on an Ad.

The Ad can be displayed to the visitor several times.

To build any classifier I need positive and negative examples:

The positives are the visitors who clicked on the Ad (easy).
The negatives are the visitors who saw the Ad but didn't click (not very obvious).

Question

Is there a convention about how/when to consider a user as a negative example?

I presume that I should define a threshold of impressions (views) per user, if the user reaches this threshold without clicking on the Ad, I consider him/her as as negative example?

Any reference or guidance is highly appreciated.


texts: Transfer learning VGGish (AudioSet). Impact of zero padding to fit the input size

texts:
I am trying to train a network on top of the VGGish architecture (https://github.com/tensorflow/models/tree/master/research/audioset/vggish), using (transfer learning) finetuning.

I initially started out with just finetuning the embedding layers, and training from scratch a simple MLP with some regularization techniques. (2 hidden layers FC with ReLU, BatchNorm and dropout, final output currently is FC to 4 classes). It is able to get an accuracy of 1 on the training data.

The input of the VGGish network are spectograms. My collected audiofiles are pretty low quality OGG/vorbis with a sample rate of 8000Hz. This is lower than the audio used in VGGish, therefore, if I create the log-mel spectogram using their code, every spectogram is missing some bins. (see bands 48 and up in the image below, y-axis) note that this happens in every image, so my hypothesis is that this shouldn't really matter. It is essentially just padding to fit the input size of the VGGish network.

Will this have any negative consequences on the ability of the network to learn using my dataset? Do I also need to finetune the ConvLayers earlier on in the network?

Any other thoughts on how I can work around this (potential) issue?It is not feasible to increase the frequency of the recordings, as they are being streamed from low power devices.

I do not have enough (annotated) data to fully retrain, so training a model from scratch is not feasible.

Suggestions on other pretrained networks are also welcome!

Thanks :)


texts: Minimize absolute values of errors instead of squares

texts: Calculating absolute values is much more efficient than calculating squares. Is there any advantage, then, to using the latter as a cost function over the former? Squares are easier to treat analytically, but in practice that doesn't matter.


texts: Gracefully removing observations with outliers in N fields

texts:
I have a function.

remove_outliers <- function(x, na.rm = TRUE, ...) {

    #find position of 1st and 3rd quantile not including NA's
    qnt <- quantile(x, probs=c(.25, .75), na.rm = na.rm, ...)

    H <- 1.5 * IQR(x, na.rm = na.rm)

    y <- x
    y[x < (qnt[1] - H)] <- NA
    y[x > (qnt[2] + H)] <- NA
    x<-y

    #get rid of any NA's
    x[!is.na(x)]
}

Given a dataset(numbers) like this:

  x
  5
  9
  2
  99
  3
  4

The functioning is obvious

remove_outliers(numbers)

means I now have this:

  x
  5
  9
  2
  3
  4

However, what if I have an ID that I want to retain, such as:

number_id    numbers
12              5
23              9
34              2
45              99
56              3
67              4

How do I remove the outlier(99) with the remove_outliers function(or another, better suited function), to get this data:

number_id    numbers
12              5
23              9
34              2
56              3
67              4

(note the entire observation with the outlier has been removed)

And how can I scale this solution to handle n more variables?

I can do it very ungracefully by taking out each column separately and building a new data frame with loops, but it's hardly readable and a mess to debug. Is there a more graceful way?


texts: Accept any suggestion to create training data from correlation matrix to find odd one out to identify difference in variation

texts:
I have N time varying feature vectors obtained by recording different parameters over time.This results in N*N similarity matrix which contains one to one correlations value for each feature. We need to consider only the upper triangle matrix since it's symmetric, so the correlation value for one feature is low corresponding to all other feature. Is there any learning method that can identify that feature autonomously.
For Example the matrix looks something like this

             Feature 1   Feature2 Feature 3 Feature 4
   Feature1  1           0.91      0.81       0.44
   Feature2  0.91          1       0.98       0.31
   Feature3  0.81       0.98       1          0.32
   Feature4  0.44       0.31       0.32        1

So we know Feature 4 is behaving differently . Any learning method that can learn this difference and identify the corresponding feature.Sorry, if this is a very trivial question, I'm new to data analysis.


texts: Mechanical engineer having troubles with neural network. Am I going in the right direction? Is my data set insufficient?

texts:
I’m doing a project in which the suitability of Neural Networks (NN) shall be assessed and I am fairly new to the subject. More precisely the NN should replace an analytical model. This model calculates forces and torques applied on a structure based on displacement data. The task of the neural network is to approximate forces/moments via regression.
Some Authors state that a neural network consisting of input, one hidden layer and output layer might is sufficient for this purpose. Therefore I constructed a NN with these characteristics. The NN has 12 inputs, 8 hidden and 6 output neurons. The 12 inputs represent the deformation in two directions measured on 6 different points. The 6 Outputs represent the applied forces and torques (Fx,Fy,Fz,Mx,My,Mz).
For training and validation of the model 52 measurements, each one consisting of 12 deformations and 6 forces/torques have been used. Afterwards the data was extended to 174 measurements. In both cases the NN could not generalize well and the training “loss” was huge. I tried changing various parameters of the neural network (such as number of hidden neurons, activation function, optimizer, scaling of the data…). None of this led to any significant change. Afterwards I tried to reduce the complexity of the problem by reducing the number of output variables. Still with little success.
My main concern is that the amount of data available for training is way too small. Do any of you have any tips or suggestions? Could anyone verify my concern?


texts: Workflow for Iteratively Capturing and Improving Machine Learning Models

texts:
I am working with multiple machine learning methods (support vector regression, random forest regression, and knn regression through scikit-learn) and want to know how to determine which method is most appropriate to pursue given incrementally increasing my training sample by 10,000 records.

My training data size is about 80k records, and my test dataset is approximately 30k records. I start with 10k records to train each of my models because the system I am working in has very limited cores (and runs on a VM on a different continent) so I can't simply start a training session and walk away for a few hours.

As I increase the training size, I expect that my training validation scores (RMSE, MAPE) will decrease on the full training dataset, and my testing validation scores will increase.

What are the metrics that I should capture about each run in order to make informed decisions about the complexity, speed, and long term viability of these machine learning models?

(This post does a great job of explaining the general steps associated with the *entire* data science workflow. However, I am particularly interested in exploring what would be considered the ideal set of metrics that I should capture after having trained, tested, and validated a machine learning model.)


texts:
I have built a recommendation engine using cosine similarity. When I want to find all the records similar to a given record that is already present in the dataset it works. Consider a case, a user enters a new record which does not exist in the similarity matrix, to find and recommend similar records I am adding the new data point into the dataset built the similarity matrix and then pick the similar records, Is there a better way to handle this problem?
Adding more details
data=["document1", "document2", "document", ... "documentN"]
count_matrix = count.fit_transform(data)
cosine_sim = cosine_similarity(count_matrix, count_matrix)


texts: About the maximum likelihood, when we convert the maximization problem into minimization, why we take the negative?

texts:
On page 12, we take $log$ on both side.

$\max_{\boldsymbol{w}}L\boldsymbol({w})=\max_{w}\displaystyle\prod_{n=1}^Np(t^{(i)}|x^{(i)};\boldsymbol{w})$

$\ell(\boldsymbol{w})=-logL(\boldsymbol{w})$

$\ \ \ \ \ \ \ =-\displaystyle\sum_{i=1}^Nlog\ p(t^{(i)}|x^{(i)};\boldsymbol{w})$

The $log$ function is increasing as $\boldsymbol{w}$ increase. Why we have to take the negative?


texts: Unstable Training when combining Graph Neural Networks for Graph Classification Tasks

texts:
I have been combining Graph Convolutional Layers and Graph Pooling layers to define a neural network architecture for Graph Classification tasks. Specifically, using the Graph Convolutional Layer proposed by:

Kipf, T.N. and Welling, M., 2016. Semi-supervised classification with
graph convolutional networks. arXiv preprint arXiv:1609.02907.

And the Graph Pooling layer based-upon the Graclus algorithm from:

Defferrard, M., Bresson, X. and Vandergheynst, P., 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, pp.3844-3852.

My graph contains 100 nodes, with 2 features per node. Where for the first feature we find -10<=x_1<=10 and for the second feature -30<=x_2<=-30. Nodes correspond to actual subregions of high-dimensional spaces, and edges can be introduced to model the neighborhood relationships between these different subregions. Notably, the feature matrix is rather sparse. Based upon labeled sets of feature matrices, I want to classify these matrices accurately, while exploiting knowledge about the neighborhood relationships between the nodes.
Therefore, I decided to setup a neural network architecture as follows:

Input Layer
Graph Convolution Layer (filters= 25, adjacency_matrix = A[0])
ReLU Activation
MaxPooling1D (pool_size=2)
Graph Convolution Layer (filters= 16, adjacency_matrix = A1)
ReLU Activation
MaxPooling1D (pool_size=2)
Flatten
Dense (nodes= 1024)
ReLU Activation
Dense (nodes= 256)
ReLU Activation
Dense (num_classes)
Softmax Activation

Note that the Graclus algorithm adds fake nodes to the original graph and reduces the size of the adjacency matrix. Such that one needs to pass the new adjacency matrices to every Graph convolution operation. However, I find that no matter how I train this network, the accuracy turns out be wildly fluctuating.

I achieve much better results when just using a MLP and neglecting the adjacency matrix, i.e. the neighborhood relations between subregions/nodes, completely. I also achieved very competitive results to the MLP when using a 2-channel CNN instead (by converting the graph into a 2d euclidean map using a self-organized map beforehand). The Graph Neural Network architecture not working is counter-intuitive and I am wondering whether or not one can identify a reason for this to be the case. I do not suppose feature normalization is a problem, as it has not been for the CNN.


texts: What is the name of this similarity distance metric?

texts:
def distance_metric(seed, base):
    num = 0.0
    den = 0.0
    num = sum(numpy.minimum(seed,base))
    den = sum(numpy.maximum(seed,base))
    dist = round(1.0 - 1.0*num/den,4)
    return dist

The metric is used to gauge similarity in the context of locality sensitive hashing.

Items within a bucket are kept if their distance is < 0.16.


texts: Why is there a trade-off between bias and variance in supervised learning? Why can't we have best of both worlds?

texts: The bias-variance trade-off is like a law in machine learning. You cannot have the best of both worlds. What is it about supervised learning in machine learning that makes it impossible to satisfy the two at the same time?


texts: Shaping columns (and column headers) into multi-index rows using pivot/pivot_table

texts:
Problem
I am trying to shape a dataset which includes stacked dates/data as rows and half-hourly data as columns (Trading periods, eg. TP1, TP2), into a chronological half-hourly order with the stacked data becoming columns (see image below for the output format) and the column headers (TP1, TP2, etc) becoming indices along with the dates.
I am new to Python and Pandas (C and MATLAB programming experience), initially I thought to loop through the data and append the relevant parts of each row into a new data-frame. It seems to me however that either the df.pivot() or pd.pivot_table() would be a more suitable way to achieve this as it would allow some aggregation at the same time.
Sample dataset:
df = pd.DataFrame({'POC_Code': ['ARA', 'ARA', 'ARA', 'BPE', 'BPE', 'BPE'],
                   'Trading_date': ['1/04/2018','2/04/2018','3/04/2018','1/04/2018','2/04/2018','3/04/2018'],
                   'TP1':[10120,11760,25930,3545,12749,11358],
                   'TP2':[10170,11790,25890,4329,13793,15448],
                   'TP3':[10200,11750,25860,3465,13943,16132]})

  POC_Code Trading_date    TP1    TP2    TP3
0      ARA    1/04/2018  10120  10170  10200
1      ARA    2/04/2018  11760  11790  11750
2      ARA    3/04/2018  25930  25890  25860
3      BPE    1/04/2018   3545   4329   3465
4      BPE    2/04/2018  12749  13793  13943
5      BPE    3/04/2018  11358  15448  16132

My Attempt
Using the following I get close what I want (the image below), but I am still not sure how to turn the TP1, TP2, into indices:
piv = df.pivot_table(values=['TP1','TP2','TP3'], index=['Trading_date'], columns='POC_Code')

                TP1           TP2           TP3
POC_Code        ARA    BPE    ARA    BPE    ARA    BPE
Trading_date
1/04/2018     10120   3545  10170   4329  10200   3465
2/04/2018     11760  12749  11790  13793  11750  13943
3/04/2018     25930  11358  25890  15448  25860  16132

I tried to access the column headers like this, but it's not valid which I get: piv = df.pivot_table(values=['TP1','TP2','TP3'], index=['Trading_date', df.columns.values[2:5]], columns='POC_Code')
Desired Output
I need the output to be a chronological format (by date & trading period) such as the multi-index way below, so that I can combine it into a single index of the form 2018-04-01-01 for export to another program. If this combination of the indices can be done in the same step, even better.

Is this possible with pivot/pivot_table, or would something like this (https://stackoverflow.com/questions/37430940/python-pandas-converting-column-headers-into-index) be a better approach (or a combination of both)? The full dataset includes 65 POC_Codes and 50 Trading periods, so I would like to use a pivot table as I also plan to sum these into various categories.
Full dataset can be found at: https://www.emi.ea.govt.nz/Wholesale/Datasets/Generation/Generation_MD/201804_Generation_MD.csv


texts: Many questions training unbalanced and duplicated data

texts:
I'm a DS student. I have like 30.000 of bank statements, all labeled with a specific category(cat1, cat2, ...). With that data I'm trying to train a classification model but I found several problems:

The category that has more statements has 10k, the one with less 100 (unbalanced dataset)
Many rows are repeated with the same text ( for example "Buy for a total of X$ in the supermarket Y", but the repeated rows appears in each label)

Should I train with repeated statements? if I do so the model will be trained for some specific cases and the predicted results will not be real because for example if I have repeated statements with the same label:

A, B, B, B, B, B (labels)
and I take only one row with label A and B, since the statement will be repeated the accuracy predicting the others B's will be 100%.

If I do not train with repeated statements am I not deleting important information about my data and can affect the prediction later?
Should I undersampling or oversampling? It's correct for this kind of data? I will have more repeated rows in the case of oversampling, and undersampling I'll lose a lot of information.

I do not know if what I'm thinking has some sense. I really need some tips in order to train my data.


texts: How to get a rebalance strategy with a cost matrix?

texts:
In the case of a classification problem where a cost matrix is used to maximize the model performance, it is common to do a rebalance technique.

Let's say for example that I have the following costs for the two classes.

C(a,a) = 0, C(b,b) = 0, C(a,b) = 2, C(b,a) = 1.

Then, with a Rebalancing technique, I would need examples of class b twice as the examples of class a.

But, what should by rebalancing strategy will be when there is a cost for (a,a) or (b,b)?

For instance,

C(a,a) = 0, C(b,b) = 2, C(a,b) = -2, C(b,a) = -10

How should I handle those cases?


texts: Predicting High-School test scores after a disciplinary action

texts:
I'm somewhat new to machine learning and have learned to apply many of the basic regression and classification methods using python and various packages.  However, approaching this problem has me stumped.  To illustrate the problem, I created a fictitious scenario where a guidance counselor wants to predict test scores for a student after disciplinary action.  Suppose they have data available like the mock-up below:

Column definition:
Student - Student Identification #
Gender - Male/Female
Age - Current Age
Athlete - Sport that student plays (0-None, 1-Basketball, 2-Football, 3-Soccer)
Online - Student takes classes online (0-No, 1-Yes)
Before_Disciplinary_Action_Scores - Sequence of their last n test scores prior to discipline (in date order)
Discplinary_Action - Action Counselor Took (0-None, 1-Assigned Tutor, 2-Guardian Meeting, 3-Study Program, 4-Ineligible for game)
After_Disciplinary_Action_Scores - Sequence of their next n test scores after discipline (in date order)

Assume the fictitious school system is quite large and there are about 80k records in total.  All students have a total of 12 test scores from Before/After discipline action, but the number of test scores varies based on when the disciplinary action was given.  For simplicity, you can assume the scores are given on a weekly basis for a single course.

I can compute mean before/after scores and create a fairly good classification model, but I'd like to take it a step further and predict the post-discipline scores.

I've tried using Prophet and LSTM models to predict the after scores from a time series approach with poor results and the varying number of scores makes it difficult.  I can see that the features around Athlete, Transfer, and Online are important components and I tried adding them as regressors, but that too has failed.  I appreciate any guidance you can give.


texts: How to select 'cutoff' of classifier probability

texts:
I have recently used xgboost to conduct binary classification in an nlp problem. The idea was to identify if a particular article belonged to an author or not, pretty standard exercise.
The results are outputted as a probability between 0 and 1, and there is the ocasional article that is completely misclassified.
I would like to know if there is a statistical approach that gives me a confidence interval for the probability outputs (for example if I consider all articles with prediction of 0.4 I will get 95% of the articles that belong to the author), or something that helps me make decisions regarding the cut-offs.


texts:
I'm following an AI course and we've just entered the deep learning chapter. Speaking about the difference between classic machine learning models and deep learning, it turns out one of the most relevant points in favor of a neural network is that it doesn't need the feature extraction phase. That's because it can implicit learn what are the most important features directly on data we give it as input. Conversely, a machine learning system heavily relies on feature extraction before it can start training.
I've found some useful questions about this topic on this site, but still I'm a bit confused in understanding what feature extraction actually is. So I've asked my teacher to provide me an example.
He gave me this example: suppose we have a huge dataset of images and we want to classify them based on the presence or absence of at least one tree in the picture. So, an image will be classified as 1 if it contains a tree, -1 otherwise.

in the case of a neural network, we simply feed the images to the net with their labels. That's all. The network will learn what a tree is and will optimize its weights in order to detect them
on the contrary, if we want to use a machine learning algorithm, we also have to tell the model what a tree actually is, in addition to the labeled images. This part is the feature extraction phase

I don't understand how can I tell to a model what a tree is inside an image. I thought discover it is part of the task of the machine learning algorithm.


texts: How to randomly split the data set into multiple different sets:(training 70%:validation 10%(optional):testing 20%) in R?

texts:
I have a dataset with 4 predictor variables X1, X2, X3, X4, and one response variable Y. I have been asked to check the correlation between these variables and see how they are related and then use a linear model to fit them.

No split of training set: test set is given. I have one data set with 10000 samples. I was planning of splitting this data set in the ratio 80:20 for training and testing respectively.

I'd like to know how to do the same in R programming.

Also in general, we will split it into multiple combinations of training:testing set right? or? Please correct me if am wrong. I am a newbie to ML, so kindly help me out here.


texts: How to continue incremental learning when a categorical variable has been assigned additional category labels?

texts:
Please help answer this question or point me to any resource.

There is a model in an environment where training happens with new data and the data is discarded after training is completed.
This keeps on happening in cycles. Hence we dealing with models allowing "partial fit"

In this model, there is a training variable X , which is a categorical variable.
X has some previously known categories C and its encoded as "one hot vector" of length D .

Now suppose we are now observing a new category of X, so we need to extend encoding of C to length D+1 .

The question is, how to do this without losing all the previously learned knowledge by model.

Training again is not an option since we have discarded all the previously held data and only new data can be used for training.

There is already a question Updating One-Hot Encoding to account for new categories
However the answer is either to use entire data to create the "one hot encoding" (which is not possible in this environment since new data is in future) , or to omit the usage entirely (thus loosing some information).


texts: How to statistically prove that a column in a dataframe is not needed

texts:
I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic and domain knowledge I can judge that the columns that show dimensions of the fabric is not necessary to analyse the data I am looking for.

I tried to run a correlation and print a heatmap but it doesnt show any output for most of the columns. Only white color is printed out for most of the column correlation. This makes it difficult for me to judge.

Please advice if there is a better way to statistically prove the relationship between the columns and justify the reason to drop irrelevant columns


texts: Why are Convolutional Networks not using cross-correlation

texts:
To my knowledge cross-correlation is used to measure the similarity of certain values, like to images. Same applies to the process of feature extraction in CNNs, where input matrices are multiplied by filters. So it seems odd to me that they are call Convolutional Networks.

The Pytorch documentation for Conv2d even says that it is using the cross-correlation operator.

So why are CNNs called convolutional when they are actually using cross-correlation?


texts: Multi-label compute class weight - unhashable type

texts:
Working in a multi-label classification problem with 13 possibles outputs in my neural network with Keras, sklearn, etc...

Each output can be an array like [0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1 ,0].

I have an imbalance dataset and i trying to apply compute_class_weight method, like:

class_weight = compute_class_weight('balanced', np.unique(Y_train), Y_train)

When i try to run my code, i got Unhashable Type: 'numpy.ndarray':

Traceback (most recent call last):
 File "main.py", line 115, in <module>
   train(dataset, labels)
 File "main.py", line 66, in train
   class_weight = compute_class_weight('balanced', np.unique(Y_train), Y_train)
 File "/home/python-env/env/lib/python3.6/site-packages/sklearn/utils/class_weight.py", line 41, in compute_class_weight
  if set(y) - set(classes):
  TypeError: unhashable type: 'numpy.ndarray'

I know that is because i working with arrays, already tried add some dict,

i.e.:

class_weight_dict = dict(enumerate(np.unique(y_train), class_weight))

Well, i don't know what to do, tried others strategies, but no success... Any ideas?

Thanks in advance!


texts:
I am looking for a relatively large amount of jazz (and other) solos to use for data science purposes - probably somewhere up to a 100 transctipts. Although that is not that much and many pdf transcripts are available, I want the data to be in a format where I can easily access all the information in Python (which note was played and at what beat). Manually writing all that data would, however, be very tedious and time-consuming.

Thus my question: does anyone know of any software that automatically reads pdf transcripts and puts it in a different file format, or alternatively of a website with transcripts already in a format in which I can access all the data easily from Python?


texts: I want to create a weighted random graph (in contrast to the unweighted Erdős–Rényi model). I have a list of weights (derived from a real-world network, very skewed distribution that most weights are 1 and the rest of them are between 6-8). My plan is to first create an Erdős–Rényi model and randomly reassign the weights according to my list of weights. Does it sound like the right approach or my new network essentially lost the key characteristics of a random graph?


texts: Similarity between bordermode and zero padding in Keras

texts: In Keras, the border_mode = 'valid' doesn't zero pad the input. Thus, we subsequently get an output feature map that is not the same size as the input. Likewise, setting border_mode = 'same' gives output feature maps as the input. My question is if we set border_mode = 'same', should we also perform zero padding using the zero_padding function?


texts:
I'm trying to create a logistic regression model in jpmml, then write the PMML to a file. The problem I'm having, is that I can't find any way to create a custom tag, such as "shortForm" and "longForm" in the following example:

<MapValues outputColumn="longForm">
  <FieldColumnPair field="gender" column="shortForm"/>
  <InlineTable>
    <row><shortForm>m</shortForm><longForm>male</longForm>
    </row>
    <row><shortForm>f</shortForm><longForm>female</longForm>
    </row>
  </InlineTable>
</MapValues>

Here's what I have so far:

MapValues mv = new MapValues("output")
  .withFieldColumnPairs(
        new FieldColumnPair( new FieldName("gender"), "shortForm" )
  ).withInlineTable(
        new InlineTable().withRows(
                new Row().with???( new ??? )
)))

In short, I am asking for an API call I can use to instantiate the "shortForm" element in the example, and attach it to the "row" object. I've been all through the API, examples, and Google/SO, and can't find a thing.

Thanks for your help!


texts: Reverse engineering a distance metric from the output of a k-NN

texts:
Suppose that someone has trained a nearest-neighbor algorithm based on some unknown metric. I have a large dataset of $N$ observations and $P$ features. For each observation, I am given $K$ indices corresponding to the $K$ "nearest" points in the (same) dataset.

What is the general type of strategy to learn this metric? Does this type of problem have a standard name? Are there any Kaggle-type competitions with such a task, or is this unusual?


texts: Found array with dim 3. Estimator expected <= 2

texts:
I am using LDA over a simple collection of documents. My goal is to extract topics, then use the extracted topics as features to evaluate my model.

I decided to use multinomial SVM as the evaluator.

import itertools
from gensim.models import ldamodel
from nltk.tokenize import RegexpTokenizer
from nltk.stem.porter import PorterStemmer
from gensim import corpora, models
from sklearn.naive_bayes import MultinomialNB

tokenizer = RegexpTokenizer(r'\w+')

# create English stop words list
en_stop = {'a'}

# Create p_stemmer of class PorterStemmer
p_stemmer = PorterStemmer()

# create sample documents
doc_a = "Brocolli is good to eat. My brother likes to eat good brocolli, but not my mother."
doc_b = "My mother spends a lot of time driving my brother around to baseball practice."
doc_c = "Some health experts suggest that driving may cause increased tension and blood pressure."
doc_d = "I often feel pressure to perform well at school, but my mother never seems to drive my brother to do better."
doc_e = "Health professionals say that brocolli is good for your health."

# compile sample documents into a list
doc_set = [doc_a, doc_b, doc_c, doc_d, doc_e]

# list for tokenized documents in loop
texts = []

# loop through document list
for i in doc_set:
    # clean and tokenize document string
    raw = i.lower()
    tokens = tokenizer.tokenize(raw)

    # remove stop words from tokens
    stopped_tokens = [i for i in tokens if not i in en_stop]

    # stem tokens
    stemmed_tokens = [p_stemmer.stem(i) for i in stopped_tokens]

    # add tokens to list
    texts.append(stemmed_tokens)

# turn our tokenized documents into a id <-> term dictionary
dictionary = corpora.Dictionary(texts)

# convert tokenized documents into a document-term matrix
corpus = [dictionary.doc2bow(text) for text in texts]

# generate LDA model
#ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=2, id2word=dictionary, passes=20)

id2word = corpora.Dictionary(texts)
# Creates the Bag of Word corpus.
mm = [id2word.doc2bow(text) for text in texts]

# Trains the LDA models.
lda = ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=4,
                               update_every=1, chunksize=10000, passes=1)

# Assigns the topics to the documents in corpus
a=[]
lda_corpus = lda[mm]
for i in range(len(doc_set)):
    a.append(lda_corpus[i])
    print(lda_corpus[i])
merged_list = list(itertools.chain(*lda_corpus))
print(a)
    #my_list.append(my_list[i])

sv=MultinomialNB()

yvalues = [0,1,2,3]

sv.fit(a,yvalues)
predictclass = sv.predict(a)

testLables=[0,1,2,3]
from sklearn import metrics, tree
#yacc=metrics.accuracy_score(testLables,predictclass)
#print (yacc)

when I run this code it throws the error mentioned in the subject.

Also this is the output of LDA model(topic doc distribution) that I feed to SVM:

[[(0, 0.95533888404477663), (1, 0.014775921798986477), (2, 0.015161897773308793), (3, 0.014723296382928375)], [(0, 0.019079556242721694), (1, 0.017932434792585779), (2, 0.94498655991579728), (3, 0.018001449048895311)], [(0, 0.017957955483631164), (1, 0.017900184473362918), (2, 0.018133572636989413), (3, 0.9460082874060165)], [(0, 0.96554611572184923), (1, 0.011407838337200715), (2, 0.011537900721487016), (3, 0.011508145219463113)], [(0, 0.023306931039431281), (1, 0.022823706054846005), (2, 0.93072240824085961), (3, 0.023146954664863096)]]

my labels here are : 0,1,2,3


texts: Aggregating multiple encoded categorical values

texts:
I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables.

I am currently using a dataset with a feature CATEGORY which has a cardinality of ~20,000. One-hot encoding does not make sense has it would increase the feature space by too much.

Each observation in my dataset can take multiple values for the CATEGORY feature, for instance, row 1 could have the value a but row 2 could have the values a, b, c, d

I have managed to encode each individual value in the feature but am unsure how to aggregate these values for each row.

How should these encoded values be combined?


texts: How to handle "year" variable for Machine Learning models

texts:
I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?

Thanks in advance,


texts:
Question
What is the loss function for each different activation function?
Background
The choice of the loss function of a neural network depends on the activation function. For sigmoid activation, cross entropy log loss results in simple gradient form for weight update z(z - label) * x where z is the output of the neuron.
This simplicity with the log loss is possible because the derivative of sigmoid make it possible, in my understanding. The activation function other than sigmoid which does not have this nature of sigmoid would not be a good combination with the log loss. Then what are the loss function for ReLu, ELU, SELU?

References

Derivative of Binary Cross Entropy - why are my signs not right?


texts:
I see many Data Science (DS) tutorials done on MACS, and many DS blogs recommend MACS as the best developing platform, thus the quote "Data Science is statistics on a Mac" came more than once into my mind.

I'm quite fascinated by MACS (be it iMAC or MacbookPro) but I never could get a valid reason to why Data Scientists in particular use them (if that is true, of course)

All who I asked said: "because it has Unix", but I hardly manage to accept it as a valid reason (at least for me). The hardware is very weak for Machine Learning processes, and the OS is inefficient and cumbersome (no file path, no drag and drop photos and music files, windows don't fully open by default nor close, etc)

The only benefit I found so far in MACoS (or Linux for this matter), is that when running parallel processes, the RAM is being shared, unlike on windows, but quoting one of my lecturers: "you need to be mad to switch OS for that reason only"

Would be grateful if you could enlighten me on the subject.


texts:
Is there any other materials that derives the LSTM back propagation and carousel of error except the original paper? (I could not understand it, sorry).

I tried deriving and got stuck, and asked the following question: LSTMs - Data Science Stack Exchange question, however, it doesn't seems that there are much people interested in hand derivation of LSTMs.

Thanks


texts: finding optimal solution $w$ and classification accuracy

texts:
Suppose you are given $6$ one-dimensional points: $3$ with negative labels $x_1 = −1$, $x_2 = 0$, $x_3 = 1$ and $3$ with positive labels $x_4 = −3$, $x_5 = −2$, $x_6 = 3$. In this question, we first
compare the performance of linear classifier with or without kernel. Then we solve for
the maximum margin classifier using SVM.
Consider a linear classifier of form $f(x) = sign(w_1x+w_0)$. Write down the optimal
value of $w$ and its classification accuracy on the above 6 points. There might be more than one optimal solution, writing down one of them is enough.

My attempt:
I understand that the data isn't linearly separable and that there will be some error, but I don't get how to get the optimal value of $w$. Do I minimize $f(x)$? But how do I take the derivative of $f(x)$? Any guidance would be appreciated, I'm a little lost.


texts: 3D CNN model always predict the same values

texts:
I am currently trying to build a 3D-CNN model to predict video quality.
When I trained the model with a small dataset (300 videos) it predicts different values, but when I use a huge dataset for training it predicts the some values. Can you help me please?
The model:
def get_model():
    # Define model
    model = Sequential()
    model.add(Conv3D(32, kernel_size=(3, 3, 3), input_shape=(
        16,224,224,3), activation='relu'))
    model.add(MaxPooling3D(pool_size=(1, 2, 2)))
    model.add(Conv3D(64, kernel_size=(3, 3, 3), activation='relu'))
    model.add(MaxPooling3D(pool_size=(1, 2, 2)))
    model.add(Conv3D(128, kernel_size=(3, 3, 3), activation='relu'))
    model.add(Conv3D(128, kernel_size=(3, 3, 3), activation='relu'))
    model.add(MaxPooling3D(pool_size=(1, 2, 2)))
    model.add(Conv3D(256, kernel_size=(2, 2, 2), activation='relu'))
    model.add(Conv3D(256, kernel_size=(2, 2, 2), activation='relu'))
    model.add(MaxPooling3D(pool_size=(1, 2, 2)))

    model.add(Flatten())
    model.add(Dense(1024, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(512, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(1, activation='linear'))
    model.compile(loss='mean_squared_error',
                  optimizer=Adam(), metrics=['mse'])
    model.summary()
    #plot_model(model, show_shapes=True,
    #           to_file='model.png')
    return model


texts: Topic Segmentation - should it be done in Raw, TfIdf or Semantic Space?

texts:
Let's assume we have a collection of documents and wish to perform some unsupervised topic segmentation.

As always, we will perform some preprocessing (including tokenization, accent-removal, lowercasing, lemmatizing and such) and transform the lists of tokens into either raw-counts or tfidf-vectors. We'll call this matrix M.

Now we have several possible approaches to perform some simple bag-of-words topic segmentation:

Apply a matrix decomposition method (LSI, LDA, NMF) directly to M and use the resulting components as the topics.
Embed each vector of M into a semantic space (LSI, word2vec) and then apply a matrix decomposition method on the semantic space.
Apply a clustering method (kM, DBSCAN, MSC, GMM) directly to M.
Embed each vector of M into a semantic space and then apply a clustering method on the semantic space.

I have two questions:

Are there any other alternatives to bag-of-words topic segmentation that I have not considered yet?
What are the conceptual differences between the methods described above and which one(s) are recommended?

Thanks in advance!


texts: How to determine the number of the training images in Keras after data augmentaion?

texts:
I want to create a CNN model and I am using data augmentation. I want know the number of augmented images in Keras.

How to determine the number of the training images in Keras after data augmentation?

For example, if I have 2000 images for training and I am applying data augmentation, what is the number of the images after augmentation?


texts: Construction of a three variable function from a set of given data points

texts:
For my problem a function $\phi$ takes three variables $f(\mu,\nu,\rho)$. Suppose, we know the range of the parameters value of $\mu=[-3,3]$, $\nu=[-3,3]$ and $\rho=[-3,3]$. We have functional value of $\phi$ for $\mu=-2$, $\nu=-2$, $\rho=-2$ to $\mu=2$, $\nu=2$, $\rho=2$ within 1 unit gap i.e. we have 125 points, however, we do not know the functional form. I have drawn the data points in the above figure and we can think as $$\frac{d\phi}{d\mu}=f(\mu,\nu,\rho)$$ I also observe a pattern of $\phi$ i.e. for each subplot $\nu+\rho$ is fixed and varied from $-4$ to $4$. My aim is to find out the function $\phi$ in terms of $\mu$, $\nu$, and $\rho$. Can you please assist about some learning method which helps to create the function with sufficient accuracy?
Thank you!


texts: What is the meaning of spherical dataset?

texts:
In the following article, one of the statement is as follows:

  The K-means algorithm is effective only for spherical datasets

What does spherical dataset mean?


texts:
Is it possible to predict the products which are going to be out of stock in next coming days/weeks.?

If Yes, what could be the possible data points that I should collect?

How is this problem different from sales prediction?


texts: Using neural networks to solve polynomials

texts:
Is it possible to train a neural network to solve a polynomial equation? What about any non-linear single variable equation?
Here are possible methods that I am thinking of trying:

Training a neural network to model some polynomial $P(x)$ and setting the output to $0$ then maybe devising some sort of algorithm with produces an approximation of the corresponding input.
Doing (1) but make a new NN which is the inverse function of the first NN. Then finding the root from inputting $0$ into the second NN.

I thought this might be an interesting science fair topic with many applications and possibilities for future study but I would like to know if it is doable. I will first attempt cubics, then successively higher degree polynomials.


texts:
I'm doing a project which involves a small dataset (4GB). I'm trying to upload it to Paperspace to do some analysis but it's taking an absurd amount of time.

Using Gradient, it would upload maybe 40MB in 10min then stop uploading. This was done using the "upload" button in the jupyter notebook.
Using a P400 hourly computer I'm transferring at 800MBPS over SSH (well, through Nautilus), but it's still taking hours to transfer this small dataset.

What am I doing wrong? Is there a faster way?


texts: Scikit MultinomialNB predict_proba Gives Same Label Multiple Times

texts:
I'm training a fairly simple model...

category_texts = [
    'this is a string here',
    'another string here',
    'once more with the words',
    'finally this string',
    'some other class with sentence in it'
]

category_labels = [
   'hasstring',
   'hasstring',
   'just words',
   'hasstring',
   'sentenceword'
]

I'm getting X by running it through TfidfVectorizor.

I'm training it with the MultinomialNB

clf = MultinomialNB()
clf.fit(X,category_labels)

Now when I run

 clf.predict_proba('vectorizer.transform('a sentence with the word string')

I'm getting an output like:

hasstring     0.9823232421
hasstring     0.6234394343
sentenceword  0.3234394343

Is getting the same label with separate probabilities expected behavior?


texts: Output shapes of Keras AdditiveAttention Layer

texts:
Trying to use the AdditiveAttention layer in Keras. On manual implementation of the layer from tensorflow tutorial https://www.tensorflow.org/tutorials/text/nmt_with_attention
import tensorflow as tf

class BahdanauAttention(tf.keras.layers.Layer):
  def __init__(self, units):
    super(BahdanauAttention, self).__init__()
    self.W1 = tf.keras.layers.Dense(units)
    self.W2 = tf.keras.layers.Dense(units)
    self.V = tf.keras.layers.Dense(1)

  def call(self, query, values):
    query_with_time_axis = tf.expand_dims(query, 1)
    score = self.V(tf.nn.tanh(
        self.W1(query_with_time_axis) + self.W2(values)))
    attention_weights = tf.nn.softmax(score, axis=1)

    # context_vector shape after sum == (batch_size, hidden_size)
    context_vector = attention_weights * values
    context_vector = tf.reduce_sum(context_vector, axis=1)
    return context_vector, attention_weights

The shape of the context_vector is (batch_size, units)
Whereas using the same AdditiveAttention layer from keras built-in
from tensorflow.keras.layers import AdditiveAttention
the shape of the context_vector = [batch_size, Tq, dim]
Any suggestions on what is causing this OP shape difference will be useful.


texts: Loading and querying a Spark machine learning model outside of Spark

texts: I'm currently working on a project where we're building a data pipeline. We have spark setup and have generated models. Sadly, loading the model in Spark and querying it isn't fast enough for us. What's the most straightforward way of exporting a model and loading it into memory on a local server? I've researched PMML and some libraries and that appears to be one path.


texts: Correct output labelling for RNNs/ANNs with sequential data and single label

texts:
I have a labelled time-series dataset; for example, a sequential input length nn from kk sensors corresponds to class '1' (see below for n=50,k=2n=50,k=2). How can I correctly classify it?

If I use RNN with infinite input length, the input layer will have 2 units (2 channels), but what about the output? Label '1' applies to the full sequence, not just a single input.

Therefore, there are two related questions: 1) how do I design an output unit? It has to reflect the input of the full sequence, 2) how do I compute an error? It also needs to account for the error of the whole sequence.

If I use ANN with a sliding window, the problem remains virtually the same - different classes can have comparable values in the window, so the output must correspond to the full sequence.

In other words, how do I match a time series to just 1 number?

EDIT:

For example, there is an $n \times k$ array labelled '1' (this can be for example readings off $n$ sensors, which match 1 class, like a gesture: 'snap'), and another $m \times j$ array labelled '0', where $n$ is the number of features/input neurons and $k \neq j$ is the length/size of the input data. If I use an RNN to feedforward the first array I do it $k$ times, obviously. But how do I label it? Do I use a unity vector length $k$, get an error each time and them average out to get the sample error and backprop? I don't think so, but not too sure.


texts:
I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked for my dataset.

Dataset

The dataset consists of 9 different classes with around 3 000 videos with labels:

Björktrast (Fieldfare)
Koltrast (Blackbird)
Nötväcka (Nuthatch)
Pilfink (Tree sparrow)
Mus (Mouse)
Blåmes (Blue tit)
Rödhake (Robin)
Katt (Cat)
Inget objekt (No object)

Here is a video of a sample dataset containing the different classes that my deep learning model should be able to predict:

Method

Currently I have used this codebase for temporal segment networks to train my model and the resulting accuracy for my own dataset was about 87% (only with RGB and not using optical flow). Although depending on weather condition and lighting, the accuracy can be very different.

I have expanded the dataset to around 30 000 videos using the following techniques:

Adding RGB noise to videos already in the training dataset.
Adding random distortions and warping to videos in the training dataset.

Problems

I have a few questions about doing video classification to predict bird species:

Are there better techniques for doing video classification compared to using temporal segment networks?
Are there any preprocessing steps that should be done to improve accuracy?
How to train the model to generalize better to be less sensitive to lighting and weather?
How can the model be trained to classify unknown species?
Are there other ways to expand the dataset except those mentioned above?

In short: What is the best deep learning model for video classification for bird species?

Update 2018/04/17: I discovered that the accuracy of the model is significantly lower with objects obstructing the camera which have similar colours to the classes of birds to be recognized. This often causes the model to label the video incorrectly and setting a high confidence.


texts: Gradient Descentfor non-linear predictor?

texts: From I what I have read so far, if a loss function $\mathcal{l}(y,\hat{y})$ is convex where $\hat{y}$ is the (to be estimated) decision function, then gradient descent tries to minimizes the loss function w.r.t $\hat{y}$. This looks alright in theory but in practice of all algorithms I have seen assumes $\hat{y}$ to be linear function (and hence GD find the slope and intercept as as the optimization problem). My concern is what if I desire an non-linear predictor. How would I implement it in an algorithms?


texts: Getting different chi square values than sklearn function

texts:
I have a dataset which looks like this-

I am trying to use chi2 as a feature selction algorithm on it.Here is the code

dataset = pd.read_csv('small_dataset.csv')
print(dataset.shape)

X = dataset.iloc[:, 1:106].values
y = dataset.iloc[:,0].values

print(sklearn.feature_selection.chi2(X, y))

I tried to validate the answers from sklearn and the formula which is

so I took the first feature and computed the A B C D which are 0 19 73 73 and N is 146.So the X2 value coming is 21 but in sklearn its coming 579.

What am I doing wrong?


texts: Generating rules from data for classification

texts: I have a set of unlabeled numeric data and i want to generate rules from it for classification purpose.  Which technique can i use?


texts: Is a test set necessary after cross validation on training set?

texts:
I'd like to cite a paragraph from the book Hands On Machine Learning with Scikit Learn and TensorFlow by Aurelien Geron regarding evaluating on a final test set after hyperparameter tuning on the training set using k-fold cross validation:

"The performance will usually be slightly worse than what you measured
using cross validation if you did a lot of hyperparameter tuning
(because your system ends up fine-tuned to perform well on the
validation data, and will likely not perform as well on unknown
datasets). It is not the case in this example, but when this happens
you must resist the temptation to tweak the hyperparameters to make
the numbers look good on the test set; the improvements would be
unlikely to generalize to new data."
-Chapter 2: End-to-End Machine Learning Project

I am confused because he said that when the test score is WORSE the cross validation score (on the training set), you should not tweak the hyperparameters to make the testing score better. But isn't that the purpose of having a final test set? What's the use of evaluating a final test set if you can't tweak your hyperparameters if the test score is worse?


texts:
I am trying clustering for the first time trying to separate my user into three categories (or the categories I though that they will should fall in).

First of all I have two tables that describe states and times.
The state table looks like that:

    [[7059992, 7082352, 7083172, 7085480, 7095742],
     [7443343, 7453591, 7463815, 7474055, 7484295],
     [3122958, 3133217, 3143449, 3153658, 3163918],

each row is a user going through some states. This type of features can not be used for clustering directly so I had to create some features.

What I did was to mark the no change between states with a zero
and any change with a timestamp between the last step when a transition was made.

so this gave me a table like that

    [ 3205.,     0.,     0.,     0.],
    [    0.,     0.,     0.,     0.],
    [10005., 16620.,  4626., 26503.],

so user 1 goes through 3205,0,0,0 which means that at the beginning there was a transition that happened 3205ms time before the last change.

user 2 goes through 0,0,0,0 that means that no change happened

and user 3 goes through 10005, 16620,  4626, 26503 which means that the user went through four changes with their respective timestamps.

I thought then that these can be now features for clustering and I tried k-means to get some labels. (I also did normalization to reduce the effects of zero values). Last column is the assigned cluster id.

 0.567415  0.347368  0.673178  0.185788         1
 0.000000  0.000000  0.000000  0.000000         0
 0.413984  0.000000  0.000000  0.439690         2

I want my clustering technique to try to find three categories that have these distinct characteristics.

First class: no transitions, pure zeros
Second class: few transitions (one or max two nonzero values, but each of them is a large value.)
Third class: many transitions (three or four non zero values, but small values for each of them).

K-means seems to merge many time second and third class since the zeros and the large values seem many times the same with many transitions and small values.

If there any clustering technique that can work better with this type of data?
If not can I also denote the no transition instead of 0, with some other value, like NaN? That might help to differentiate better the second and third class.


texts: Dealing with optional data in an Artificial Neural Network

texts:
I want to train an Artificial Neural Network on some data however some of the fields are optional, dependent upon the values contained in other fields. The data in these optional fields is missing, but correctly so. I am wondering how I should factor this into my neural network.

I feel like what I might need to do is turn certain inputs on/off dependent upon the values contained in other inputs but then I am wondering how this would affect weighting in the hidden layer. Could I have some kind of conditional hidden layer or something for only when these optional values are 'activated'?

What would be the standard approach to this? It seems surprisingly hard to find any material on this online!

Thanks in advance for any help!


texts: Are there any metrics or methods for assessing stain normalization techniques?


texts: Which type of model to use and what's my target variable

texts:
I am a beginner in Data Science field, so sorry if my question is too basic.

The task is to build an ad bidding model for online marketing which allows you to deliver targeted ads to the right people.
A part of the given data is

I don't have any additional info about the task. Is my target variable the variable spent ?
And if yes which is the best method to follow to predict the target variable?


texts:
I'm workig on multi-label classification problem. I'm facing issue while saving the TF-IDf verctorizer and as well as model using both pickle and joblib packages.

Below is the code:

vectorizer = TfidfVectorizer(min_df=0.00009, max_features=200000, smooth_idf=True, norm="l2", \
                         tokenizer = lambda x: x.split(), sublinear_tf=False, ngram_range=(1,3))
x_train_multilabel = vectorizer.fit_transform(x_train)
x_test_multilabel = vectorizer.transform(x_test)

classifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)
classifier.fit(x_train_multilabel, y_train)
predictions = classifier.predict(x_test_multilabel)

Error Message while saving the TF-IDF vectozier.

Any suggestions ?
Thanks in advance.


texts: Tensorflow API: What does the metric `tf.keras.metrics.TopKCategoricalAccuracy` do?

texts:
According to the API doc, this metric

  "Computes how often targets are in the top K predictions."

But how come the following codes prouce the result 1?
0.95>0.9>0.8>0.1>0.05, both 0.95 and 0.8 lead to 1 in prediction, shouldn't the result be 2?

m = tf.keras.metrics.TopKCategoricalAccuracy()
m.update_state([[0, 0, 1], [0, 1, 0]], [[0.1, 0.9, 0.8], [0.05, 0.95, 0]])
print('Final result: ', m.result().numpy())  # Final result: 1.0


texts: How to interpret KDE distribution graph?

texts:
I would like to know how to interpret this distribution graph. I have been doing an exercise from the book called 'Python for Finance Cookbook' by Eryk Lewinson. It does not give an in-depth explanation on this graph and i have watched videos on YouTube and it does not explain the comparison of distribution.
I would also like to know if this is positively skewed.
This is the distribution graph.

Thank you for your help!


texts: How to detect cardboard boxes using Neural Network

texts:
I'm trying to train a Neural Network how to detect cardboard boxes along with multiple classes of persons (people).

Although it's easy to detect persons and correctly classifies them, it's incredibly hard to detect cardboard boxes.

The boxes look like this:

My suspicion is that box is too simple of an object, and the neural network has a hard time detecting it because there are too little features to extract from the object.

The division of the dataset looks like this:

personA: 1160
personB: 1651
personC: 2136
person: 1959
box: 2798

Persons are wearing different safety items, based on the items are classified, while detected as whole person, not just the item.

I tried to use:

ssd300_incetpionv2
ssd512_inceptionv2
faster_rcnn_inceptionv2

All of these are detecting and classifying persons much better than boxes. I cannot provide exact mAP (don't have it).

Any ideas?

Thanks.


texts: Pre-processing mixed data prior to clustering

texts: I am new to hierarchical clustering, and wish to perform clustering on mixed data. I am slightly confused on the necessary pre-processing steps. I understand how to pre-process purely continuous data, what I haven't been able to identify is what pre-processing steps are necessary for mixed data? Do I just scale my continuous variables, impute missing data, and leave the categorical variables alone? Or do I need to perform transformations across all of my variable types?


texts:
For my program on BCH decoding I had to polynomially divide the corrected BCH code with the generator matrix for the final decoded input. I did the same as follows:

for j=1:num_blocks,
        op4(:,j)=mod(deconv(op3(:,j)',gen),2)';
end

where op3 is my corrected BCH code matrix (each block is a column) and gen is my generator polynomial.

This tends to be slow though as my input bit stream is pretty large (~10^6).

I've read that deconvolution can also be performed through the ifft of the element wise division of the ffts of op3 (its columns) and gen. However for a 15 point fft of both op3 and gen (gen is padded), the deconvolution is also 15 points.

What I need is for a (15,11) code to recover my block size 11 input on deconvolving the 15 point BCH code with the 5 point generator.

Any suggestions?


texts: RBF neural network python library/implementation

texts:
I want to use a Radial Basis Function Neural Network for my thesis.

Is there any library that implements it? And in the negative case, which is the best library to implement it?


texts: How do write the following piece of code in R?

texts:
I am trying to write a piece of code for R to fetch data from an api with the following command:

data <- read.csv("https://...../api/dataservice?sql = select * from db where name IN (?)")

The parenthesis are where I need to input 218 different name tags. How can I do this programmatically? It is possible to write down few of them, but over 200 is not possible. Is there a way this? Can I use a character vector containing the names?


texts: One-class classification in Keras using Autoencoders?

texts:
I am trying to develop a model based on one-class classification approach. For example, the model should to identify if a given picture contains a cat or not. Keep in mind that my training dataset only contains pictures of cats and nothing else. Sort of like a anomaly detection problem

I have seen some suggestions on using autoencoders for unary classification but I couldn't find a concrete example on the net? Can someone point to an implementation of this approach using Keras. Is there any other way to about solving this problem?


texts: How to use multiple cross-section observations per subject for churn prediction?

texts:
Recently I have started to teach myself about machine learning and I have ran into a dataset, which got me a bit confused.
Dataset: The subjects of the dataset are university students (student ID == "Key" feature), and each observation is a summary of their semester (grade averages, ECTS taken and completed, etc.) plus their general programme-related data (enrollment and scholarship status, date of enrollment, programme code, etc.). The data is in hungarian, but in the context of the issue, it is not important to understand the meaning of the feature names and values. Below is an example of an observation:

My goal: I want to build a model, which predicts student churn.
Problem: The dataset contains a single or multiple observations per student, based on the number of university semesters, and the observation periods are not consistent between the students, since it is based on the individual date of enrollment.

In the example picture above, you can see, that student no.1 has 7 observations (=7 semesters completed) and started his programme on 2009.09.10 (Képzés jogviszony kezdete == Date of programme enrollment), while student no.2 has 3 observations and started his programme on 2008.09.12.
I am wondering, should I use only one observation (e.g.: the last completed university semester) per student ,or does it make sense to use all observations per student?
Thank you for the feedbacks in advance!
(Also, I am new on the forum, so if you have any constructive criticism regarding the content and the format of my question, please, share with me.)


texts: Can I used a pre-trained auto encoder as an embedding layer in the model?

texts:
The dataset has an ordinal target variable [0,1,2]. Each observation in the dataset has multiple time series. In addition, each observation have various tabular features.

The purpose is to build a model that predicts whether the target is either 1 or 2 (binary classification), but a higher penalty should be assigned to misclassifying 2 than 1.

An embedding layer of some sort will be used to convert the time series into a tabular format (I am using TensorFlow). My question is whether it makes sense to train an autoencoder to reconstruct the data for where target==2? and then use that as the embedding layer. The purpose would be to optimize the embedding for the high-cost misclassifications. Or is there reason why I should train the embedding layer with the rest of the model with the appropriate sample weights?

I know the answer various from dataset to dataset. Any past experience or knowledge is appreciated.


texts: Picking a model to go ahead with for a WGAN

texts:
I have been trying to train a model using WGAN loss functions, working different learning rates to choose my hyper parameters based on advice. I was told to try looking into keeping everything simple and handle my hyper parameters first. So I set my model to having all my layers with just one filter.

Batch size=64 and a latent dim=512 as they worked best.

When I try working with my learning rates though, I have tried using 0.005, 0.0005, 0.00005 and 0,000005

None of them seem to be working even remotely well.

My losses are being generated/ plotted as follows:

for epoch in range(epochs):
        start = time.time()
        disc_loss = 0
        gen_loss = 0
        for images in train_dataset:
            #images=(images-127.5)/127.5
            #images=np.squeeze(images)
            images=np.expand_dims(images, axis=0)
            disc_loss += train_discriminator(images)
            a1.append(disc_loss)

            #a1[i]=disc_loss
            #a2[i]=gen_loss
            i=i+1

            if disc_optimizer.iterations.numpy() % n_critic == 0:
                gen_loss += train_generator()
                a2.append(gen_loss)

and then plt.plot(a1) and a2.

For a learning rate of 0.0005 these are my losses

https://imgur.com/gallery/IlxnNre

(Not sure of best way to upload)

The generator loss really bothers me as do the images. The other learning rates don't seem to be much better. Even when I beefed up my network (the heaviest layers having 512 filters, these problems persisted)

How do I work this out as no matter what I do, my models refuse to converge


texts:
I know that

  In artificial intelligence, an expert system is a computer system that
  emulates the decision-making ability of a human expert.

Does the architecture of an expert system same as that of knowledge based system or is there any difference?

What are the stages in development of expert system?


texts: Value error while using linear regression | ML

texts:
With reference to the below screen, I received the error:

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

For reading purposes, i'm appending screen shot. Please advise where I have gone wrong. Thanks for your assistance.

My Code:


texts: Incorrect multi-variate anomaly detection - Isolation Forest Python


texts: How to handle such a large class imbalance in text data?

texts:
I am working on a multi class text classifier. The total number of class that are there is 265 and total number of rows is 20,000. The class with largest number of occurrences  has 6000 samples and there are many classes with exactly 1 sample as well. Preliminary analysis of the data led me to put a cutoff of 10 samples to be their for a class to be recognized and I made a separate miscellaneous class of less than 10 samples. Now I am reduced to 27 classes. And there is still class imbalance as can be seen from the figure with the class having largest number of samples has around 6000 and the lowest has 10.

How do I deal with such large class imbalance? Are their algorithms which are better suited to handle such large class imbalance?

]1ps://imgur.com/a/8hJw9MJ)


texts: Weighting of words in lexicon based sentiment analysis

texts:
I have a a question regarding my current project, i am trying to do a lexicon based sentiment analysis on my data, where i calculate the sentiment score as following:

$$ Score = \frac{\sum_{i}{word_i}}{\mid words \mid} $$

So according to the score the word will be classified in either negative or positive. But i have also calculated for every word in the article its salience and frequency and would like to know if its possible to use them in my sentiment analysis formula above.

 words| salience| frequence
 sad    0.8       3
 happy  0.5       2


texts:
I'm trying to draw the keras model with the plotmodel.

Setup:

I installed graphviz binaries with:

choco install graphviz

added path to the bin folder, and then I did:

pip install pydotplus
pip install graphviz

Code:

Getting this error when I try to execute:

from keras.models import Sequential
from keras.layers import Dense
from keras.utils.vis_utils import plot_model
model = Sequential()
model.add(Dense(2, input_dim=1, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)

Error:

  ImportError: Failed to import pydot. You must install pydot and graphviz for pydotprint to work.

Can someone suggest some alternative to draw the keras model? I need to implement this for a CNN.


texts: Extract all the data of a particular month from dataset of different years

texts: In a dataset containing temperatures of different years and I want to extract data of particular month from all different years in single liner code what is the syntax? To extract all the data from a single month the syntax is :df['Temperature'].loc['2010-08'] It is always year followed by month.What if I want to extract 2010 & 2011 years temperatures in August in single line.


texts:
Background Information

I work for a fire department in Florida and the fire chief posed a question to me; At any given moment in time during the calendar year 2018, how many fire trucks are busy,  how many are available, how many are standing by and how many are out of service? Using the SQL Server database I was able to distill this information into tabular data.

Software available to me.

Also, because the data is in SQL Server I can parse out the date time stamp by hour, weekday, month, etc. Then Using Python or Excel I can generate statistical tests and univariate statistics and charts. I have modest experience with R. Therefore my tools are SQL Server, R, Python and Excel.

The data

For the first record with the primary key 21546912 at the time of january 1, 2018 at 00:02:31.800 there was 1 fire truck working, 9 fire trucks are available for work and 0 are standing by or out of service. For the entire year I have 104,179 observations showing at exactly that time stamp how many are working, available or standing by or out of service.
I have enclosed 30 records as a comma separated values file below my question for you to see a sample.

The Question !

So what do I do with this data? I have already generated histograms showing the frequency of FireTrucksWorking, etc. Is there a statistical test such as regression that I can perform on this data? How do I bring meaning to this dataset? Are there any general patterns to the dataset that I can discover? Can I create a probability model out of this data such as; at 08:00 during the month of July there is x% probability of N fire trucks working ? Would you use queueing theory on this dataset?

I am open to all suggestions. At this point I have a large dataset but it is really just a jumble of numbers, how do I generate meaning out of these numbers?

This is cross-posted in Mathematics Exchange. But here I make the following revision.
I am looking for some type of knowledge discovery that will show me something about my data set the naked eye cannot see or the human brain cannot discern. For example: "there is a much higher likelihood of working during the hours of 08:00 to 12:00 than the rest of the day". "Are there patterns in my data I do not recognize"?
Do you have any thoughts on this?

Thirty sample records of data

PrimaryKey,DateTimeStamp,FireTrucksWorking,FireTrucksAvailableForWork,FireTrucksStandingBy,FireTrucksOutOfService
21546912,2018-01-01 00:02:31.800,1,9,0,0
21546921,2018-01-01 00:04:46.720,1,9,0,0
21546950,2018-01-01 00:09:39.400,1,9,0,0
21546951,2018-01-01 00:09:47.320,2,8,0,0
21546955,2018-01-01 00:11:16.780,3,7,0,0
21546959,2018-01-01 00:12:04.840,2,8,0,0
21546962,2018-01-01 00:12:09.030,3,7,0,0
21546963,2018-01-01 00:12:14.470,3,7,0,0
21546966,2018-01-01 00:12:17.790,3,7,0,0
21546967,2018-01-01 00:12:21.240,2,8,0,0
21546970,2018-01-01 00:12:40.240,2,8,0,0
21546973,2018-01-01 00:12:46.720,2,8,0,0
21546990,2018-01-01 00:14:24.610,2,8,0,0
21547002,2018-01-01 00:16:03.130,2,8,0,0
21547036,2018-01-01 00:19:59.450,1,9,0,0
21547043,2018-01-01 00:21:21.950,0,10,0,0
21547064,2018-01-01 00:24:50.470,0,10,0,0
21547065,2018-01-01 00:25:13.000,0,10,0,0
21547165,2018-01-01 00:43:31.130,0,10,0,0
21547344,2018-01-01 01:15:00.980,1,9,0,0
21547361,2018-01-01 01:16:58.320,1,9,0,0
21547383,2018-01-01 01:21:38.130,1,9,0,0
21547421,2018-01-01 01:30:42.250,0,10,0,0
21547436,2018-01-01 01:33:30.320,1,9,0,0
21547442,2018-01-01 01:33:48.470,1,9,0,0
21547449,2018-01-01 01:33:58.780,1,9,0,0
21547474,2018-01-01 01:37:30.550,1,9,0,0
21547484,2018-01-01 01:39:12.350,1,9,0,0
21547487,2018-01-01 01:40:41.290,0,10,0,0
21547510,2018-01-01 01:47:02.420,0,10,0,0


texts: Right way to Fine Tune - Train a fully connected layer as a separate step

texts:
I'm using Fine Tuning with caffenet and it works really well but then I read this in Keras blog entry on Fine Tuning (They use a trained VGG16 model):

"in order to perform fine-tuning, all layers should start with properly trained weights:
 for instance you should not slap a randomly initialized fully-connected network on top of a pre-trained convolutional base.
This is because the large gradient updates triggered by the randomly initialized weights would wreck the learned weights in the convolutional base.
 In our case this is why we first train the top-level classifier, and only then start fine-tuning convolutional weights alongside it."

So as a separate step in Fine tuning they save the output of the last layer before the fully connected layer (the "bottleneck features") and then they train a "small fully-connected model" on those features and only then they put the newly trained fully connected layer on top of the whole net and train the "last convolutional block".

Is this always the right way to do Fine Tuning?

thanks


texts: Unsupervised Anomaly Detection in Images

texts:
I would like to detect defects/anomalies in images. Due to the lack of images with anomalies, I try to solve the problem in an unsupervised manner. Until now, I trained a variational autoencoder together with an generative adversarial network with “good” images. Now, I have an encoder/decoder network which is able to produce images without anomalies. Here are two examples:

original image | cropped image & input of the encoder | output of the decoder

In those images, if the object is rotated (not vertical), then it is an anomaly (like the second image). To measure the difference between the input/output of the encoder/decoder network, I tried the structural similarity metric SSIM. However, the result is not satisfying enough as many images without an anomaly also have a low similarity value. Also the pixel-wise MAE or MSE metrics are not good as they do not look specifically on the structure and are very sensitive to the brightness. Here is the result with the SSIM metric (green are the good examples, red are the anomalies):

What I want to avoid is to add specific rules for the detection, in this example to detect vertical lines because there are many other objects with different anomalies which I would like to detect.

Do you have a suggestion for another image similarity metric? Am I on the right path or are there other/better methods to do unsupervised anomaly detection?

PS: There are only a few images with anomalies (<10 images), but many without an anomaly..


texts: ValueError: shape mismatch: objects cannot be broadcast to a single shape

texts:
I want to plot a plt.bar but my code won't do it?

My code is as follows. It's getting triggered at line:

plt.bar(t[1:], gr, label='Quarterly growth rate')

  shape mismatch: objects cannot be broadcast to a single shape


texts: Feasibility: train a model to learn how to extract data from documents

texts:
I am searching for an approach for solving the following problem:

Given I have a large amount of printed and scanned documents. I am already able to detect text and the corresponding bounding boxes in that document. I have the texts and the coordinates of the bounding boxes. Now, if I want to extract specific data like "insurance contract number" or "recipient" and stuff like that, I would have to manually program that (to some degree).

I was wondering if there is an approach to show the computer what is the correct data extraction. Is it possible to train a model against a large amount of documents and the correctly extracted data as labels. So the features would be the extracted text plus the coordinates and the model should learn which data to extract depending on the training examples with correctly extracted text.

All algorithms I came across so far only "allow" one label. From my understanding this apporach would need like 20 labels for every training example. And also the prediction is also only "correct", if all labels are matching.

Anyone has an idea how to approach that?


texts: How to Ascertain Sine Wave and Harmonics

texts:
I have some data that I've noticed conforms to a sine wave. Details of the data is unknown. My task is to approximate it as closely as possible.

From some experimentation in Excel, I noticed the data follows a sine wave, so attempted to plot in the base sine wave (orange) and am now trying to approximate the blue line. I believe this is a simple job of harmonics being added to the base sine wave, however; the quantity and their respective amplitude and harmonic number (2nd harmonic being 2 * f, 3rd harmonic being 3 * f etc) is what I now need to calculate. I'm also guessing the amplitude of the resulting wave is modified by possibly a sine wave too.

Is there software which could automate this and approximate it through some analysis (Fourier analysis maybe) or possibly brute force, or is it something I have to write myself?


texts: Will the MAE of testing data always be higher than MAE of training data?

texts: On the Kaggle Course Page the chart below shows that MAE of testing data is always higher than MAE of training data. Why is this the case? Is it only limited to DecisionTreeRegressor model? Or the graph is wrong and in practice the MAE of testing can be lower than MAE of training?


texts: Python: Handling imbalance Classes in python Machine Learning

texts:
I have a dataset for which I am trying to predict target variables.

Col1    Col2    Col3    Col4    Col5
  1      2       23      11     1
  2      22      12      14     1
  22     11      43      38     3
  14     22      25      19     3
  12     42      11      14     1
  22     11      43      38     2
  1      2       23      11     4
  2      22      12      14     2
  22     11      43      38     3

I have provided a sample data, but mine has thousands of records distributed in a similar way. Here, Col1, Col2, Col3, Col4 are my features and Col5 is target variable. Hence prediction should be 1,2,3 or 4 as these are my values for target variable. I have tried using algorithms such as random forest, decision tree etc. for predictions.

Here if you see, values 1,2 and 3 are occurring more times as compared to 4. Hence while predicting, my model is more biased towards 1 2 and 3 whereas I am getting only less number of predictions for 4 (Got only 1 predicted for policy4 out of thousands of records when I saw the confusion matrix).

In order to make my model generalize, I removed equal percentage of data that belongs to 1,2 and 3 value randomly. I grouped by each value in Col5 and then removed certain percentage, so that I brought down the number of records. Now I could see certain increase in percentage of accuracy and also reasonable increase in predictions for value 4 in confusion matrix.

Is this the right approach to deal with (removing the data randomly from those groups on which the model is biased)?

I tried for in-built python algorithms like Adaboost, GradientBoost techniques using sklearn. I read these algorithms are for handling imbalance class. But I couldnt succeed in improving my accuracy, rather by randomly removing the data, where I could see some improvements.

Is this reduction is undersampling technique and is this the right approach for under-sampling?

Is there are any pre-defined packages in sklearn or any logic which I can implement in python to get this done, if my random removal is wrong?

Also, I learnt about SMOTE technique, which deals with oversampling. Should I try this for value 4? And can we do this using any in-built packages in python? It would be great if someone helps me in this situation.


texts: Dimension change of convolutional layer after applying the feature map

texts: In this lecture I don't understand how the output has 1 layer after applying the feature map to the 3 layer input.


texts:
I have a dataset which contains both text and numeric features.

I have encoded the text ones using the TfidfVectorizer from sklearn.

I would now like to apply logistic regression to the resulting dataframe.

My issue is that the numeric features aren't on the same scale as the ones resulting from tfidf.

I'm unsure about whether to:

scale the whole dataframe with StandardScaler prior to passing to a classifier;
only scale the numeric features, and leave the ones resulting from tfidf as they are.


texts: How exactly do probability distributions help modeling/making decisions?

texts:
I am an elementary/wanna-be statistician/data scientist from South Korea. I have been studying a variety of theories of mathematical statistics and different probability distributions. (I apologize for bad English if there is one!)

I am wondering how exactly these distributions help a curious data scientist derive any inference or make decisions when modeling.

For example, if a variable of a dataset is estimated to follow Poisson distribution with a specific value of parameter lambda. What can it actually tell you? What kind of inference can I learn? What can it imply when I am looking into a couple of specific data points or areas?


texts: Learning a distribution of functions instead of a single function?

texts:
It is easy for a neural network to learn to approximate an analytic function such as f(x) = x^2 or f(x) = x^2 + 1.

I am struggling to learn a parametrized version of that (ie. f(x) = ax^2 + bx + c).

Given a context of 1000 x and f(x) pairs uniformly sampled from a quadratic with random a,b,c, I want to provide a target x_t and have the network predict f(x_t).

My architecture is below. input_1 is the context vector which consists of 1000 x and f(x) pairs. input_2 is x_t, the position of the point I want to predict. dense_6 is a linear layer with one neuron, supposed to represent f(x_t).

To train, I have generated 100 random quadratics. For each quadratic, I generate 100 training examples. A training example consists of a random context (the 1000 points fed to the first network), and a random target.

However, I am unable to get this to work at all no matter what I do. The validation loss barely decreases. It either memorizes the training set and overfits or underfits.

Any ideas on how I can achieve this? What should I try?


texts: Why are results of cross-validation before splitting is different from results after splitting?

texts:
All data in my binary classification problem are represented by X and y.  Now, I do stratified cross-validation on this data as follows:
scoring = {'accuracy' : make_scorer(accuracy_score),
           'precision' : make_scorer(precision_score),
           'recall' : make_scorer(recall_score),
           'f1_score' : make_scorer(f1_score)}

model=RandomForestClassifier(n_estimators=50,random_state=10)

results = cross_validate(estimator=model, X=X, y=y, cv=10, scoring=scoring)

If run the code, I will get the following results:
Accuracy : 0.5436815489342804
Precision : 0.020165565854870747
Recall : 0.11013513513513513
F1_score : 0.03315023853741518

Now, I split X and y as follows:
#Test, training data split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, stratify = y)

#Split the training data into validation set
X_val_train, X_val_test, y_val_train, y_val_test = train_test_split(X_train, y_train, test_size = 0.1, random_state=0, stratify=y_train )

Now, if I do the same cross-validation procedure like before on X_train and X_train, I will get the following results:
Accuracy : 0.8424393681243558
Precision : 0.47658195862621017
Recall: 0.1964997354963851
F1_score : 0.2773991741912054

I do not understand why results are so different and why this happened.


texts: Is there a way to add new feature to existing machine learning model and retrain it?

texts: Suppose a machine learning model is trained on two features (f1, f2). If there comes a new feature(f3) is there any way to pass this new feature to existing model and retrain instead of training a model again on the three features?


texts: Semantic segmentation with greyscale images

texts:
I'm trying to reproduce a research with greyscale images instead of colour images.
I have found that there are pre-trained networks, like VGG16, with ImageNet. But that dataset has colour images, and I can't use it because I'm going to use greyscale images.
Is there any pre-trained network with greyscale images?
Failing that, I can also train a network with a greyscale image dataset but I can't find any.


texts: I have a dataset with 3 independent variables [city, industry, amount] and wish to normalize the amount. But I wish to do it with respect to industry and city. Simply grouping by the city and industry gives me a lot of very sparse groups on which normalizing (min-max, etc.) wouldn't be very meaningful. Is there any better way to normalize it?


texts: Data preprocessing for time series prediction

texts:
I have a dataset that has the following structure
[
 [
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 1
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 2
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 3
                               :
 ],
 [
  [ product 2 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 1
  [ product 2 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 2
  [ product 2 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 3
                              :
 ],
                              :
]

My goal is to predict for a newer product say product_n predict if it will be sold (3 hours earlier).
My question is how do I process it for a Recurrent Neural network  since the vector of prediction was sold? is available for each hour.
To say that in detail, since
[
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 1
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 2
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 3
                               :
 ],

is one observation  for the RNN how do I assign was sold? to it? Since len(X) should be equal to len(y)
was sold? is available for each observations, do I take max for 3 hours and asssign it to the obervation?
Like
X = [
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 1
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 2
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 3
                               :
 ],
and
y = [max(was sold?)]


texts: Variational auto-encoders (VAE): why the random sample?

texts: Why do people train variational auto-encoders (VAE) to encode means and variances (regularised towards 0 and 1), and then sample a random Gaussian, rather that simply encode latent vectors and regularise them to follow a standard N(0,I), which would appear as a more natural choice?


texts: Data repository on employee turnover based on Position

texts:
We need a data-set that has information regarding the career progression and preferably the salary of the employee.

For example:

People move from the position of junior nurse to head nurse (junior developer to senior developer to lead developer etc.) in on average "x"-years. They change job after "y"-year and on average receive increment of "z"-% etc.

Do you guys know any public or private repository of such data.


texts: Do I need to convert strings before using LSTM?

texts:
I have a dataset which includes one column with a URL and another column with value 0 or 1 indicating if it is a phishing link.

I want to process this dataset using LSTM. Do I need to first convert the string column into some other data type?

The model I'm hoping to use would be something like below:

# Model building using the Sequential API
model3 = Sequential()
model3.add(Dense(10, activation='relu',
          kernel_initializer='uniform',input_dim=x3.shape[1]))

# Add a LSTM layer with 128 internal units.
model3.add(layers.LSTM(128))

# Add a Dense layer with 10 units.
model3.add(layers.Dense(10))


texts:
I am interested in finding out how decision trees chose the order in which they split. I understand that splitting is based in information gain. The attribute with the lowest information gain is chosen as the root node.

If I had a data set with columns:

credit standing
age
income,
marriage status

and I was interested in finding out what determines a person to have a good or bad credit status, Am I correct in saying, I calculate entropy and information gain for each of these categorical attributes against the independent attribute i.e what I am investigating (credit standing), and that the calculation with the lowest information gain is chosen as the root node.

For example, if this root node (first split) was Age, Is the entropy and information gain of Age (the new independent attribute) against the remaining attributes (marriage status and income) calculated, and the calculation with the lowest information gain is then chosen as the second split node, and so on?

ie.

information gain:

credit standing vs age    = 0.01
credit standing vs status = 0.1
credit standing vs income = 0.2

Age is chosen as root (first split) node.

then, information gain:

age vs status = 0.2
age vs income = 0.1

income is chosen as second split node.

Am I understanding this correctly?


texts: How to do data augmentation for Machine Learning on statistical data?

texts:
I am training machine learning classification model. I have data in csv format with lets say 5 features(or columns) and 100 such observations(or rows).

I want to add more similar data to improve my model accuracy. I do not have more data from original source.

I did data augmentation on images in deep learning where I performed brightness(-/+), rotate, flip. But I do not know how to do such exercise in ML.

Please help.


texts:
I would like to build an LSTM network for text classification with PySpark, but I don't find any library or function about it.

Is it possible to implement this kind of network in Spark?


texts: OpenAI Gym equivalent for supervised and/or unsupervised learning

texts:
OpenAI Gym has really normalised the way reinforcement learning is performed. It makes it possible for data scientists to separate model development and environment setup/building and to focus on what they really should be focusing on.
Quoting from Gym website:

Background: Why Gym? (2016)
Reinforcement learning (RL) is the subfield of machine learning
concerned with decision making and motor control. It studies how an
agent can learn how to achieve goals in a complex, uncertain
environment. It’s exciting for two reasons:

RL is very general, encompassing all problems that involve making a sequence of decisions: for example, controlling a robot’s motors so
that it’s able to run and jump, making business decisions like pricing
and inventory management, or playing video games and board games. RL
can even be applied to supervised learning problems with sequential or
structured outputs.
RL algorithms have started to achieve good results in many difficult environments. RL has a long history, but until recent
advances in deep learning, it required lots of problem-specific
engineering. DeepMind’s Atari results, BRETT from Pieter Abbeel’s
group, and AlphaGo all used deep RL algorithms which did not make too
many assumptions about their environment, and thus can be applied in
other settings.

However, RL research is also slowed down by two factors:

The need for better benchmarks. In supervised learning, progress has been driven by large labeled datasets like ImageNet. In RL, the
closest equivalent would be a large and diverse collection of
environments. However, the existing open-source collections of RL
environments don’t have enough variety, and they are often difficult
to even set up and use.
Lack of standardization of environments used in publications. Subtle differences in the problem definition, such as the reward
function or the set of actions, can drastically alter a task’s
difficulty. This issue makes it difficult to reproduce published
research and compare results from different papers.

Gym is an attempt to fix both problems.

I was wondering if there is something equivalent for supervised or unsupervised learning. I believe such a tool will be useful for the same reasons. Am I missing something? Or is there there something already out there?


texts: Naive/ Persistent Models & 7 Day Forecasts

texts:
When determining the baseline performance using a persistent or naive model I understand it to be using the value of the previous time step as the prediction for the next time step and then determining the accuracy.

My question is how would one do this when determining the baseline for a 7-day forecast? Would you calculate the accuracy using the previous time step (t-1) to predict only the next time step (t), or would you use the previous time step (t-1) to predict all 7 future time steps (t,t+1,t+2 etc).


texts: Pandas replace column values by condition with averages based on a value in another column

texts:
I have a dataframe with people's CV data.
Among others, there's a column with years of experience, and a column with age.
Some people stated their age and experience in a way that experience > age. For example age=30 and experience=40, which is a problem.
I am trying to perform a following task:
If experience > age, replace the value of experience with an average experience of people of the same age. For example if age=30 and experience=40, replace experience with an average experience of all 30-year olds.
The problem is that I don't know how to obtain the age of a specific person in the slice df[df['experience'] > df['age']].
AFAIK it could be done by something like this:
mask = (df['experience'] > df['age'])

df['experience_cleaned'] = np.where(mask, df[df['age'] == age]['experience'].mean() ,df['experience'])

I would appreciate if you show me how it can be done, because I could use the technique to also preprocess more data in a similar way.


texts: Confusion regarding the Working mechanism of Activation function

texts:
For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted output value to be a number between 0 and 1.

My confusion is that is there a binary step function in the output layer which squashes the values of the linear combination of weights and and inputs to 0 or 1? Does classification means always applying a thresholding function on top of a linear or non-linear function which is in the hidden layer?

Say the predicted output value is 0.75 and actual Y is 0. Then, how is 0.75 converted to 1? The loss function would calculate the error as actual - predicted = 0-0.75 = -0.75

Can somebody please explain the math or point out some links where the working steps are shown? Thank you.


texts: What is the best way to propose an item from a set based on previous choices?

texts:
The goal of this question is to be able to propose a user further choices based on his past experiences: like Amazon's book advices.

From a set of mp3 files, I assume that a set of mp3 tags data is already filled, based on the music he/she has alredy listened to  : what is the easiest way to  implement a machine learning that is able to propose a list of music choices  based on the user's set ?

NB : I'm a Machine learning novice I'd appreciate if the answer could be based on Orange, Weka or these kind of tools.

Update: removed the classification tag as recommanded.

For new comers as me:
- the book Predictive Analytics For Dummies is a nice general introduction about this subject
- a next step would be the the paper Finding Clusters of Similar Artists which is really interesting especially for the K-means approach
- The millions songs dataset which is a gold mine with its huge dataset as well as tutorials with Python codes to use with it
- special thanks sheldonkreger for his answer with neo4J graph usage idea wich is really interesting


texts:
I am trying to find some sort of encoding algorithm that would allow to transform system file paths eg. "c:/users/file1/subfile2/targetfile" into a feature that I could use in machine learning like a float value. It would be best if the folders/files located in the end of the path would have a smaller impact on the output value.
I am thinking maybe some combination of hashing and arithmetics but not sure for now.
I tried looking for something like that in scientific papers and on the internet in general but with no success.
Have anyone ever done something like that or similar?
Sorry if there are some stupid errors in my question. I am just starting my data science adventure.


texts: Tensorboard with pytorch dont display a graph

texts:
I am trying to visualize a model I created using Tensorboard with Pytorch but when running tensorboard and going to the graph tab nothing is shown, im adding my code for reference, also im adding a screen-shot of my conda env for all the dependencies

from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter()
images, labels = next(iter(trainloader))
grid = torchvision.utils.make_grid(images)

writer.add_image('images', grid)
writer.add_graph(model=model, input_to_model=images, verbose=True)
writer.close()


texts: Which PreProcessing method should be used?

texts:
I have a dataset that consists of a poisson distribution, a exponential distribution, categorical variables, and my target variable is a numerical bimodal variable. This is a regression model. I wanted to clarify the type of model best to use and any transformations required for my variables.
Model 1. Random Forest. Since RFs are scale invariant, no transformations are required for either target and independent variables.
Model 2. Gaussian Regression. Categorical Variables would be One Hot Encoded. The Poisson and Exponential Variables could be transformed with a Box-Cox Transformer. How would I transform my target bimodal variable? I already tried abs(y-mean(y)), which when tested was not normal. Some things I have read, say to fit to a Gaussian Mixture Model, but I do not understand what they mean because I have only used GMMs with clustering.
Is there a different type of model that may work better here?


texts: same or different number of epochs for dataset of different sizes?

texts:
I am trying to fit a DL model on a dataset, after a while of parameter tuning, I have determined the optimal number of epochs.  Now suppose there are new data available and I would like to include them to re-train the model, would I need to set the number of epochs to be exactly the same, or make it smaller such that I have equal number of gradient updates?  Both seem to make sense to me, but I am not sure which would usually work better.
I understand that a better way would be to use train-test splits to determine best epochs, but since data are super limited, I would prefer not to reserve any data if possible.
Any idea or suggestion would be much appreciated, thank you!