texts: matching results with sklearn average_precision_score

texts:
I have written a (convenience) class for binary classification in python which takes several models and compares their ROCs, AvPrecisions, and creates several charts (distribution of scores, accuracy/sensitivity/specificity, ROC, PR, etc.).

A key difference that I am observing is in the computation of average precision.  For some examples, using average_precision_score(true labels, prob score) I get a value which is 0.1% to 0.2% more than the average precision computed as

xaxs,yaxs = df['sens'].values[::-1],df['ppv'].values[::-1]
ap = yaxs[:-1]*(xaxs[1:]-xaxs[:-1])

or equivalently

xaxs,yaxs = df['sens'].values,df['ppv'].values
tmp = yaxs[1:]*(xaxs[:-1]-xaxs[1:])

I do the same for AUC and that matches to second place of decimal.  Based on the formula given here the above calculation seems to be correct.

In the figure shown, the values computed by scikit-learn are 0.8977 and 0.8722.  Whereas the values compute by the numpy code above are 0.895 and 0.8697.  I am using 10,001 thresholds between 0 and 1 so I would think that the bumps in the left corner of the chart should be adequately captured.

I would welcome input on how to change my calculation to match the one from skl. Thx.


texts:
I made a front end where I would like to make REST calls to an AWS Lambda interfaced with AWS API Gateway.

I dumped my model as a pickle file (and so my encoders) which I initially trained locally. I then stored these files in a S3 bucket.

The problem is that I cannot import libraries such as pandas and sklearn to make model predictions because the lambda console is unable to find them.

Does anyone have any suggestions to help solve this issue?


texts:
I am building a Convolutional Neural Network with Keras. I want to use my own dataset. This dataset consists of matrices of specified size and filled with 0's which contain a submatrix of specified size and filled with 1's. The submatrix is optional and the goal is to train the NN to predict whether a matrix contains the submatrix or whether it doesnt. To make it more difficult to detect, I am adding various types of noise to the matrices.

Here is a picture of what the matrices look like, the black parts are 0's and the white part are 1's. There is a 1:1 correspondance between the pixels of the image and the entries in the matrix.

Now I want to create a CNN which can classify these images. The easiest approach so far was to use

test_set = test_datagen.flow_from_directory('dataset/training_set',
                                            target_size = (imageSize, imageSize),
                                            batch_size = batchSize,
                                            class_mode = 'binary')

And then I fill the directory training_set with 1000's of images. This works well, but has one major problem:

It is really inefficient, as I'm first creating images, saving them on a hard-dsik, then loading them again. These are a lot of unecessary steps, considering that keras transforms the images back into matrices (or tensors, idk), and my images were initially matrices.

Thus I was thinking about improving the efficiency by having the matrices in a .csv or even having my matrixgenerator.py "feed" a list or array directly into keras. I did some research and found the method flow from Keras which specifies as parameter an input matrix.

Unfortunately this is where I am stuck. I do not understand how to get my matrices into the form keras wants. I also don't know the exact shape the matrices and labels need to have to be able to use the flow method.

It would really help if someone could explain me how to use the flow method/tell me where I find more information on it, or give me a better option to feed my dataset into a keras CNN. Thank you for your help. I'm also ok with using a lower-level tensorflow implementation as long as I understand it.


texts: Why joint probability in generative models?

texts: I have been reading about generative models for past few days and there's one question that's bugging me. Many sources(Wiki, Google dev article) mention that generative models try to model the joint probability distribution p(x, y) and that to generate new samples we sample from this distribution. My question is why model and sample from the joint distribution and not the conditional distribution p(x|y)?


texts: Finding clusters of 3D points where not all the points belong to a cluster

texts:
In my research, I'm trying to find clusters of brain activity, like in the following image:

Where the red dots represent the origin of the activity (where the green pole is the direction). So basically, I would like to find clusters of 3D points. I know the number of clusters (4 in this example), and that some of the points do not belong to any of the clusters (noise). One approach can be using GMM (Gaussian Mixture Model), where one can increase the number of the components until finding the desired output. The big disadvantage of this approach is that the desired output should be well defined. I thought to use a more sophisticated approach, that can deal better with noise, like RANSAC (Random sample consensus). Using skimage.measure.ransak, one can define the model_class for a 2D circle (CircleModel), but not for a 3D sphere. Is anyone familiar with a 3D sphere model for RANSAC? Or any other solution that can handle the noise?


texts: Help with algorithm approach for computer vision

texts:
I hope this is the right forum to ask.

I had a client approach me with a demand for a vision system for their assembly line.

The problem they are facing is that the operator sometimes forgets to put all three parts of the product together. They want a vision system that can spot if any of three main components are missing (only missing nothing else).

I have the knowledge to build a covnet in keras but I think that would be a way to big hammer for this nail and I have to go to the client any produce images myself (ergo I can sit there for a day or two and take photos the products). So I might end up with a data set of maybe 200 images. I will try and make some artificially bad images, taking of the different components and photoing that. I don't think I'm allowed to show what the product looks like but I made a sketch.

What algorithm can I read up on that would solve this problem in a good and time efficient manner? Demands on the camera and pc for the any suggested algo would also be very much appreciated (resolution etc).


texts:
I am relatively new to data science and have an exercise task. This consists of the classifications of excerpts of texts. However, the texts are obfuscated such that one cannot read the words, spaces etc. But the "patterns" are preserved. I have a training set of the following form. One .txt file with the text excerpts looking like

shdbcjhbjhbefbhbwhbkbehbwbwbfhwb /
wbhbwtjnwkjbrfbqlenk /
wjnfkjebrkbrghkbibgibib /
tberbtewtwbkwtjbrkbwkbtwrbt /
.
.
.

and one .txt file with the labels of the texts looking like

1 /
4 /
11 /
0 /
.
.
.

This means that each string of symbols belongs to a certain text and one knows to which one.

The task is now to set up a classifier, to train it on above training data and to test it on test data which also consists of text excerpts.

My basic idea was to interpret the data as images and to set up a deep learning NN with e.g. TensorFlow. However, I am not sure about the shape of the data which is needed to feed the data to the NN.

Do I have to create label-folders containing all texts with a certain class-label or is there a more direct way (so far I created vectors in R consisting of the texts and the labels)? Must the excerpts all have the same length? How do I have to manipulate the data until I can feed it to the NN?


texts: If deep-learning learns features, aren't we saying it can learn association rules?

texts: If deep-learning learns features, aren't we saying it can learn association rules ?


texts:
I have large text files on HDFS... I would like to label some text in those files to improve text analysis?

Do you know of any tool like that?


texts: Renaming deep learning layers causes bad results

texts:
I created two CNN models using Keras. The models give good accuracy results. To concatenate these models together to create a multi input model, I have to change the names of the layers to be unique. So, I changed the layers names in the second model using the following code:

for layer in model2.layers:
    layer.name = layer.name + str("_2")

After changing the names of the layers, the second model gives a bad results.
When I remove the code of renaming, the second model gives a good accuracy.

How can I solve this problem?

Is there another way to change the layers names?

The complete code here:

model = vgg16.VGG16(include_top=False, input_shape=(64,64,3))
inp = model.input
out = model.output

model2 = vgg16.VGG16(include_top=False, input_shape=(64,64,3))

for layer in model2.layers:
    layer.name = layer.name + str("_2")

inp2 = model2.input
out2 = model2.output

merged = concatenate([out, out2])
merged = Dense(1024, activation='relu')(merged)
merged = Dense(num_classes, activation='softmax')(merged)

model_fusion = Model([inp, inp2], merged)
model_fusion.summary()


texts: What does the embedding mean in the FaceNet?

texts:
I am reading the paper about FaceNet but I can't get what does the embedding mean in this paper? Is it a hidden layer of the deep CNN?
P.S. English isn't my native language.


texts:
I want to test if using additional information of weekdays would improve my NN. Therefore, I just converted the weekdays numerically such as

Monday -> 0
Tuesday -> 1
...
Sunday -> 6

but my NN fails totally with that alongside to other 16 variables. Without it's ok. Now I wonder if I maybe have too few data or if the conversion does not make sense at all but I guess I do not need something like one-hot-encoder, or do I?


texts:
Often when i get a new project in machine learning the client always ask me either to do a particular task like a prediction for one thing or
give me data and ask me to find what i can do with it.

i've read the book hands-on machine learning with scikit-learn & tensorflow where you can see a full process for starting a project , basically drawing plots and search in correlation matrices what is interesting.

Do you guys have a special way to do this ? what is the step by step process you use ?

Thank you.


texts: In word2vec why is the implementation of likelihood function multiplication of probabilities of finding a neighbouring word given a word? I didnt get why the probabilities should be multiplied.Is there a reason/intuition behind it ?


texts: Solving Bayes Theorem equation -> I can't calculate proper result

texts:
I am solving questions for an edx course on Machine Learning.
One particular question is giving me a problem:

Assume a patient comes into the doctor’s office to test whether they have a particular disease.
The test is positive 85% of the time when tested on a patient with the disease (high sensitivity):
P(test+|disease)=0.85
The test is negative 90% of the time when tested on a healthy patient (high specificity):
P(test−|heathy)=0.90
The disease is prevalent in about 2% of the community:
P(disease)=0.02
Using Bayes' theorem, calculate the probability that you have the disease if the test is positive.

My solution:

I have created a table

   sick   |   healthy
     2%   |   98%
+   90%   |   15%
-   10%   |   85%

From this I calculated bayes theorem like this:

                      (0,02*0,9)
P(A|B) = -----------------------------------------------------
               (0,02*0,9) + (0,15*0,98)

I get P(A|B)=0,109
however this answer is wrong,
where did I do mistake?


texts: Advice on where to continue in the field of data engineering and machine learning

texts:
I finished a 28 hours Machine learning with python (Basic course) on Udemy, and it was very beneficial.
My aim, is to be able to understand what is ML and how to use its concepts while working with data.
I am confused about where to continue. My goal is to be a data engineer/analyst or at least data engineer.
I am searching for courses on coursera, because of its credibility.
I found a course from University of Michigan: Applied data science with Python.
But I am confused if I should take something else in order to start my data science job search. I know it is going to be a long process to finish.
Any suggestions on how to proceed in this journey as I am really confused.
Feel free to remove my question if it was not supposed to be asked here.


texts: (Pattern-Matching) what is the best measurement can be used to match drug names from different countries

texts:
I need to compare the price of the local drug-list of my country with prices of drug-lists of other countries (CSV format), I have a manually-matched table whic is my target and I use python via Jupyter on windows.

In the following, there are two records of two different drug-lists

one drug in list 'A'

Name:   40MG ABSEAMED*

Form:   injectable solution

Dosage: 0.9%

matched drug in list 'B'

Name:   ABSEAMED SOLUTION FOR INJECTION 40MG/0.4ML

Form:   SOLUTION FOR INJECTION

Dosage: 25mg/ml

My simple process is to clean data from undesired symbols and spaces, then to count occurances of features.

Problems:

using simple distribution function (intersected words in both) / (words of list 'A') the name of drug (ABSEAMED) is considered as priority as other features ('FOR', 'SOLUTION', ...)
By removing the Form words from Name words, some names becomes null
after cleaning data, I get noisy features (25mg/ml ===> 25 mg ml) and this dropped the accuracy of matching from 56% to 24%

  What are the basic recommended techniques/tools (Automates, POS_tagger, ...) of Natural Language Processing that may give a high-accuracy results ?

I thought about creating my own POS tag to match tags before contents

(e.x. the drug name '0.225% W/V SODIUM CHLORIDE (1/4 NORMAL SALINE)

can be tagged as \dosage \symbol \name \quote*

but I'm not sure enough that it can improve my result

Any help is appreciated.


texts: How to measure deviance resulting from different random seeds in machine learning?

texts: I'm running an xgboost model to predict probabilities to a binary classification problem. Then I aggregate the results based on the Age variable (what is the aggregated risk of getting the sickness for Age x). I made a mistake and did not set the seed number to be constant so when I rerun the model I get slightly different aggregated results. Could you give me a reference why it is not substantial? So I can avoid running the model on several seed settings to get a confidence interval? I would not want to do it because the learning process takes a few hours. Thank you in advance!


texts: Bias variance tradeoff boosting (xgboost) vs random forest (randomized bagging) which to use when?

texts:
I was looking up differences between boosting an bagging and I see this quoted everywhere

If the classifier is unstable (high variance), then we should apply
Bagging. If the classifier is stable and simple (high bias) then we
should apply Boosting.

also

Breiman [1996a] showed that Bagging is effective on ``unstable''
learning algorithms where small changes in the training set result in
large changes in predictions. Breiman [1996a] claimed that neural
networks and decision trees are examples of unstable learning
algorithms.check last lines

Aren't gbdt's the most preferred/recommended learners in xgboost ?  Which is contradictory considering how decision trees are apparently unstable learners.


texts: Modeling a time series quantity by modeling its constituent time series

texts:
I have a time series target, let's say $Y_1$. This quantity depends on two other time-series quantities deterministically, $Y_2 \text{ and } Y_3$. That is, we have some function which takes $Y_2$ and $Y_3$ and computes $Y_1$. This function was verified by carrying out simultaneous lab sampling for $Y_1, Y_2,$ and $Y_3$.
Now, I have created a Dynamic Partial Least Squares Regression model for determining the values of $Y_2$ and $Y_3$ in between lab samples, using lab sample data for training the model. The R-squared of fit for $Y_1$ was 0.78 and for $Y_2$ was 0.59.
Next, I try to calculate $Y_1$ using the predicted values of $Y_2$ and $Y_3$ by the function mentioned above (Modelling $Y_1$ directly gives bad results). However, the R-squared for $Y_1$ for even this type of calculated model was only 0.05, even though the calculation is deterministic and accurate. I am failing to understand why this is happening. Is this simply a matter of propagation of errors for the two models for $Y_2$ and $Y_3$? Based on the advice here, I am looking to either completely abandon this approach of modeling $Y_1$ indirectly, or look into it further.


texts: Methods to improve neural network to distinguish red from blue?

texts:
I have a lot of data where im trying to distinguish red data from blue, but all the features look like this basically (see imgur).

They overlap a lot, and I can only get roc auc score of 0.65 .
So far in terms of data-processing ive only used class weights to balance it. I have tried different kinds of dense/dropout networks, with the different features but only 0.65 is the best I can get, most hover around 0.63-0.65, no matter how many layers or epochs I add.
For example a simple DNN/dropout yields 0.65:
"""
Dense(512, activation='relu'),
Dropout(0.3),
Dense(512, activation='relu'),
Dropout(0.3),
Dense(512, activation='relu'),
Dropout(0.3),
Dense(512, activation='relu'),
Dropout(0.3),
Dense(512, activation='relu'),
Dense(1, activation="sigmoid")
"""
Some resnet modification I found online also only provided 0.65.
So i have tried CNN and dense layer networks only, so far.
Any hints or ideas I could do to get better AUC scores?


texts: Multi-label classification using output quantization

texts:
Problem statement

It's a fact that in order to train the network for multilabel-dataset, a one-hot-vector output is usually used.

Example:

  dog         [1 0 0]

  cat         [0 1 0]

  rabbit      [0 0 1]

Consequently, we're increasing size of the weight matrix as well as the required training time.

Question: Is there an approach that we can use one output and quantize it for different classes?

Example:

  0 <= output < =1

  dog [0.0 - 0.33]

  cat [0.33 - 0.66]

  rabbit [0.66 - 1]


texts:
First off, I am new to machine learning, so these questions may be trivial.

Basically I am trying to tune an object with numeric knobs and numeric outputs. By doing a brute force tuning (permutations), there is a solution to find ideal values of the outputs, but it takes time. I'm trying to take a stab at using ML to at least shorten the process of tuning.

I have a dataset for a large number of good units that were tuned successfully, at the various number of attempts though.

For example:
   passing requirement: X = near 10, Y = near 5, Z = near 4

Object 1:

Try 1 => A = 1, B = 2, C = 3 ; X = 1, Y = 0, Z = 1  => not good
Try 2 => A = 1, B = 1, C = 1 ; X = 10, Y = 5, Z = 4 => good enough

Object 2:

Try 1 => A = 1.4, B = 2.6, C = 3.8 ; X = 10, Y = 5, Z = 3.9 => lucky!!!

Object 3:

...
Try 10 => A = 1.4, B = 2.6, C = 3.8 ; X = 10, Y = 5, Z = 3.9 => took a while!!!

I'm wondering how do I prepare the data for training and testing with this type of problem, as each object had a variable number of tries before it got successfully tuned.
Should I just take the last successful combination for each object, and maintain the same columns (A, B, C, X, Y, Z). Or take them all (multiple rows per object)?

Or, for each object record, append another set of columns, such that per object, I have only one row. For example (A1,B1,C1,X1,Y1,Z1, A2,B2,C2,X2,Y2,Z2,... An,Bn,Cn,Xn,Yn,Zn)

As for the algorithm choice, I can only articulate that this isn't a classification problem (or a binary outcome). Something like regression, or decision tree, or hill climbing if there's such a thing?


texts: What's the advantage of multi-gpu training in real?

texts:
The decreasing speed of training loss is almost the same between one gpu and multi-gpu.

After averaging the gradients, the only benefit from multi-gpu is that the model seems to see more data in the same time.

But why average the gradients?
Is it that the model is indeed feed with more data in the same time?


texts: Steps to fit a Machine learning model for prediction of up and down market movement

texts:
I have around 5 years of data of an index containing many features on a daily basis. I want to classify whether the index will move up or down the next trading day (up or down movement is determined by next day open/close price). I am using an SVM classifier for this classification. What could be some essential steps which need to be followed? I suppose since I am using financial data, there would be some deviation from the traditional method of applying machine learning. I have the following steps in mind:

Prepare data containing all features at-hand and direction variable.

Feature engineering: That is creating various features using transformations like log, % change on various available features. I have not incorporated this step but will do it later when I have a decent running model.

Feature Selection: How to go about this? I first am using various features which I think have predictive power but how to go about it systematically.

Walk forward Modelling: After selecting features I have the data which I will be using for my model. I am using the first 500 days of data to train the model and testing it on the next day and then using data from day 2 to day 501 to train the model and then testing it for day 502, and so on until I reach the present day. I make sure that I scale my training data in the range 0-1 and using the same scaler for test data before running the model. I use the default parameters of SVM in sklearn right now.

I then check performance of my model by using various metrics obtained from confustion matrix like accurace, F1 score, etc.

For applying SVM I am using this reference which tells to use cross-validation to choose C and gamma. How can we use CV here if we have financial time series data? What else should I include in my steps?


texts: How to know when to stop trainning a deep network?

texts: I've been training several auto encoders containing two GRUs as encoder and decoder during last year. It occurred to me that while training there were sequences of epochs which their loss did change just a little bit(less than 0.005 or so). Every time I took this as a sign that my model is not going to be better and stopped training. Now that I think about it I wonder what if I continued to train for more epochs and loss of model started to decrease again? Is there a factor to decide how long and how many epochs are enough to train recurrent networks?


texts: How to improve regex while fetching record dynamically

texts:
The data is as follows:

COL1    COL2

 12    :402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:22:hhhh
 57    :42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:hhhhhh

I am creating a third column field_info like:

 COL1  COL2                                                    field_info

 12   :402:agsh,hhjd,:45:hghghgruru,:12:fgh,ghgh,:22:hhhh      fgh,ghg
 57   :42:agshhhjd :57:hghgh,gruru:120:fghghgh :12:hhhhhh    hghgh,g

I am using a regex function as follows:

df.loc[:,'field_info']=df.col2.replace(regex=r'.*'+ df.col1.astype('str') +':(.{15}).*',value="\\1")

I have 2 columns col1 & col2. col1 has some value which I am searching in col2 dynamically and extracting the next 15 characters from that. However, it's taking a lot of time. Can anyone suggest a faster way of doing this?


texts:
The parameters of the network are changed to minimize the loss on the mini-batch, but usually the loss on the mini-batch is just the (weighted) sum of losses on each datum individually. Loosely, I would represent this as
$$ dT = \frac{1}{\text{batch_size}} \sum_{i \in \text{batch}} dT_i$$
Where $dT$ is the update of the net parameters for the batch and $dT_i$ is only for one training example.
Why can't $dT$ be calculated 'on-line' then, where the only RAM needed is on the partial sum for $dT$ and whichever $dT_i$ you are working on at that moment?


texts:
Im using this book to make a very barebones neural net framework to make arbitrary sized networks. But im confused about the back prop algorithm provided and was hoping someone can explain the little things im confused by.
The algorithm in the book is as follows:

What I am mainly confused about is the nested loop for subsequent layers.
First, the $\Delta_{j} \leftarrow g'(in_{j}) \sum_{i} W_{i,j} \Delta_{i}$
The book says earlier that $\Delta_{i} = Err^e_{i}$ x $g'(in_{i})$ So will it always be that or should each iteration update the $\Delta_{i}$ to become $\Delta_{j}$ after?
The second line in that loop $W_{k,j} \leftarrow W_{k,j} + \alpha$ x $I_{k}$ x $\Delta_{j}$ the book says it's for between input and the next hidden layer. Where $I_{k}$ is that input value. But then what rule do I use for between hidden layer and hidden layer? Do I use that rule but instead of $I_{k}$ being an input would it be that neurons activation value instead?


texts: How to derive false positive and false negative from top-k accuracy?

texts:
I am working on the following "equality identification" problem and become quite confused on how to reasonably define false positive and false negative in my case.
Problem:
Suppose I have a large set of graphs $G$, where for every graph $g \in G$, there is one and only one graph $g'$ that is identical to $g$. I am training a model $M$ which, given two graphs $g, g' \in G$, can decide how similar these two graphs are. The output of $M$, $o =M(g, g')$, denotes a similarity score of these two graphs. $o$ is a floating number, ranging from $[0, 1]$.
To evaluate the performance of $M$, I am measuring the top-$k$ accuracy of its predications in the following very standard way:
for g \in G:
    for t \in G:  // suppose here t != g and therefore comparison is meaningful
        accuracy[(g, t)] = M(g, t)

correct_prediction = 0
total_prediction = 0
for g \in G:
     total_prediction += 1
     sort_accuracy_array_on_graph (g); // we sort all comparisons on $g$ with other graphs \in G
     if g' in top-$k$ most similarity graphs with g:
        // g' is the ground truth, the "identical" graph with g
        correct_prediction += 1

correct_prediction / total_prediction // the average top-$k$ accuracy

OK, so this seems very standard approach to computing the top-$k$ accuracy of $M$. However, currently I want to take one step further and somewhat measure the false positive and false negative of $M$. I understand that FP and FN are both very standard metrics in data mining, but just somewhat become very confused on how to define it in my problem. For instance, I can define False Negative in the following way:

False Negative of $M$: conceptually, FN denotes that $g'$, the identical graph of $g$, does not appear in top-$k$ most similarity graph of $g$. Then, perhaps FN is interchangeable with top-$k$ accuracy in my case?

False Positive of $M$: but if we accept that FN is interchangeable with top-$k$ accuracy in my case, then how to define FP of $M$? FP denotes that $M$ aggressively treats graph $t$, where $t != g'$, as the "similar" graph of $g$ and appear in its top-$k$ most similar set. Then isn't it indicating that FP is ALSO interchangeable with top-$k$ accuracy in my case? That seems suspicious because in my understanding, FP should not equal to FN? There must be something wrong.

Am I clear on this confusion? Any suggestion would be appreciated. Thank you!


texts: In a list, find the numbers where they are bigger than the next numbers

texts:
I'm using Rstudio. I have a list like below, I want to find all the numbers that are bigger than the next numbers (i.e. numbers on their right hand, I make them bold).

7  7  9  9  9 10 10 1 51 51 53 58 58 58 58 59 59
62 64 67  0  0  3  3 12 12 12  1  4  4  5  5  9 21 25 25 27
27 29 33 33  49 50  0  0  0  3  4  4  8  8  8 21 21 21 21 21 24 25 25 26 27

Thanks anyone who kindly offers an idea!


texts:
Could any one help me know about different approaches, methods or algorithms to build a model to forcast a time serie which has only two values ( 0, 1 ) but they last over time each time .
basically I've some on/off data, it tell me if there is an object in a place and how much time that object was in that specific place . So the output is a rectangular wave but it's not periodic.

so if we have the object in that place field we will get 1 every x seconds then when there is nothing we will get 0 .
Thanks in advance of your response!


texts: How is this tensorflow command interpreted?

texts:
I am reading an introductory tutorial on tensorflow here, and I'm confused about the code that defines an input layer for a word2vec embedding:

# Headline input: meant to receive sequences of 100 integers, between 1 and 10000.
# Note that we can name any layer by passing it a "name" argument.
main_input = Input(shape=(100,), dtype='int32', name='main_input')

Afaik, there are 10.000 words in the library, and the input is a string of 100 words (a news headline). One word is represented as a 32 bit int.

But it seems to me that too little information is passed to this Input layer:

What does "shape=(100,)" mean? why does it omit the second argument?
how does it know how to deal with "dtype='int32' "? If I saw that without context, I would guess that it interprets each of the 100 32bit ints as numbers rather than one-shot word vectors, so I would guess it would create a 100 neuron layer, where each neuron receives 1 input of 32 bits which would be the input to the neuron's activation function. But what we really want is a neural net with 10000 neurons (one for each word) times 100 (for each word in the sequence), right?

What am I missing?


texts: Should we first remove the unwanted independent variable or split the data into test train data?

texts: I wanted to know if we are dropping any independent variable as it has too many missing values(~75% or more) from the data then should we do it before splitting the data or after splitting the data


texts:
I feel like this is a rudimentary question but I'm very new to this and just haven't been able to crack it / find the answer.

Ultimately what I'm trying to do here is to count unique values on a certain column and then determine which of those unique values have more than one unique value in a matching column.

So for this data, what I am trying to determine is "who" has "more than one receipt" for all purchases, then determine the same information based on each product category.

My approach so far:

We have a dataset like this:

receipt,name,etc,category
1,george,xxx,fish
1,george,xxx,cat
2,george,xxx,fish
3,bill,xxx,fish
3,bill,xxx,dog
4,jill,xxx,cat
5,bill,xxx,cat
5,bill,xxx,cat
5,bill,xxx,dog
6,george,xxx,fish

So then I can do this:

df.set_index(['name','receipt'])

And get the more interesting

                etc category
name   receipt
george 1        xxx     fish
       1        xxx      cat
       2        xxx     fish
bill   3        xxx     fish
       3        xxx      dog
jill   4        xxx      cat
bill   5        xxx      cat
       5        xxx      cat
       5        xxx      dog
george 6        xxx     fish

At this point it feels to me like the data is easy to work with, but I haven't figured it out.

One thing that is interesting to me is that if I sort the data by name before indexing it, the data displays grouped by name. In both cases the index is the same, so I don't know how to play with the representation of the data after indexing.

It is easy to find the data by category using

>>> orders.loc[orders['category'] == 'fish']
                etc category
name   receipt
george 1        xxx     fish
       2        xxx     fish
bill   3        xxx     fish
george 6        xxx     fish

But what I can't figure out is how to tell pandas "Find me the list of names that have more than one receipt".

Smaller questions:

What is the "pandas way" to get the length of the names part of the index? I'm supposing I could just turn the name column into a set and get the length of that. But I'm curious about indexes.

Edit / Update

Thanks for those answers! Here is clarification on what I am looking for:

I'm trying to find "repeat customers": people with more than one receipt.

So my set of all customers would be:

names: ['george','bill','jill'], ratio: 1.0

My repeat customers:

names: ['george','bill'], ratio 0.66

All 'fish' customers:

names: ['george','bill'], ratio: 0.666

My repeat 'fish' customers:

names: ['george'], ratio: 0.333

I think the examples given look helpful, but feel free to add anything.


texts:
I want to try different methods of conditioning the decoding process of the Variational Autoencoder Models of the Google Magenta project for my own research project. As far as I can tell, MusicVAE has already been conditioned by the authors on chords (e.g., for the 'hier-multiperf_vel_1bar_med_chords' model). I want to also try other methods, like style tags or diatonicity etc.
However, I am having a hard time of figuring out where the respective tensors (one-hot-encoded chords I think) are used during training in the hierarchical decoder. Are the same conditioning tensors concatenated to every decoding step? Or only the first? Or something else? Since it is difficult to figure this out by looking at the code and the paper (https://arxiv.org/pdf/1803.05428.pdf) does not mention this architectural concern, I thought maybe a person involved could clear this up for me. Here is the picture of the MusicVAE architecture as depicted in the just mentioned paper.

P.S.: I also posted this very same question here https://stackoverflow.com/questions/63029684/magenta-musicvae-groovae-conditioning , I can delete it if that is against the rules.


texts: NLP Text Summarization - which metrics to use in evaluation?

texts: I'm trying to implement Text Summarization task using different algorithms and libraries. To evaluate which one gave the best result I need some metrics. I have read about the Bleu and Rouge metrics but as I have understand both of them need the human reference summaries as a reference. Is it so that no automatic evaluation can be done in text summarization, i.e human generated summary is needed beforehand? What is the common practice for the tasks like this? How often is the text summarization accuracy measured at all?


texts: Huggingface Library - Multi-document summarization

texts:
Can BART, PEGASUS ... etc. API in huggingface library be used to directly perform multi document summarization?
(e.g. here: https://huggingface.co/transformers/model_doc/bart.html)


texts: looking for public dataset for stock market

texts: I want to do some modelling and data visualization on historical stock data, including price, volume, financials, etc. Is there an public dataset available for stock price history? I looked at a few, but either they have a high cost, or not sure they would be reliable. Free would be preferred, also established and reliable. If not, what are some good options for collecting the data myself? Maybe web scraping, or public api's etc.


texts: Which algorithm to use to classify plots based on graphical features?

texts: I have plots which I need to classify based on some features. For example, I need to differentiate between the following plots having smooth features or 'valleys' at certain x values. Which machine learning algorithm would be most appropriate to do so? I was thinking a combination of anomaly detection, clustering and classification. Any help is appreciated. Thank you!


texts: What are the differences between Glueviz and Orange that come installed with Anaconda and how do they compare on performance and workflows?

texts: I've been using Anaconda for quite a while now and these two libraries glue and Orange have sparked interest lately. I've checked their webpages and some documentation and FWIW, it looks as if both share more similarities than differences. I could be wrong there and hence this question. Which one is better? I've used matplotlib and some seaborn as my go-to visualisation tools so far.


texts: Terminology to distinguish between ML methods and optimization methods (PSO, ACO..)

texts:
I am currently writing a scientific thesis which consists of two parts. In the first part I am building ML models with neural networks, support vectors etc. and the second part is about finding global minima for optimization objectives with optimization methods like Particle Swarm Optimization, Ant Colony Optimization, Simulated Annealing etc.
When writing about the parts in the introductory part, I would write machine learning methods when referring to the first part and metaheuristic optimization methods when referring to the second part.
Is it possible to distinguish them like this?


texts: How does LSTM fights vanishing gradient?

texts:
Vanishing gtadient causes too small values which are come from non-linear functions like sigmoid, tanh. LSTM architecture was developed to fight these issues.
But how can it avoid a problem if there are 3 sigmoid, 2 tanh functions and, moreover, they are combinated via multiplication?


texts: Preparing Dataset Minority Class vs Majority Class

texts:
I'm currently doing a binary classification for sentiment prediction. Currently I have the majority class (~90% of the data) as my positive class (labelled 1) and the minority class (~10% of the data) as my negative class (labeled 0). What I'd like to maximize in this experiment is the detection of negative sentiments, hence I'd like to maximize the precision (and recall) of my minority class.
However, in many similar datasets (in terms of prioritizing the detection of minority class) out there like credit card fraud detection, cancer detection, usually the minority class is set as the positive class and the majority class set as the negative class.
My question is: Does it matter if the minority class is set as the positive or negative label in relation to performance of training a model or affecting a loss function such as cross entropy?


texts: Looking for a data set that shows hospital patients' illnesses and their vital signs, during their stay

texts:
I'm looking for a data set that shows hospital patients' vital signs (body temperature and/or heart rate, etc.) and their illnesses, over time. Is there such a data set?

For example:
Patient #1, Male, age:35, ..., Diagnosed w/: Heart failure
22:32:10: Temperature:32.1
22:32:20: Temperature:34.2
22:32:30: Temperature:33
22:32:40: Temperature:35

Or:
Patient #2, Female, age:35, ..., Diagnosed w/: Migraine Attack
22:32: Heart Rate:68
22:33: Heart Rate:75
22:34: Heart Rate:69
22:35: Heart Rate:70

I don't have any insistence for the data to come from a hospital. I already have searched some sources but with no luck.


texts: Training accuracy is ~97% but validation accuracy is stuck at ~40%. What does it imply?

texts:
Training accuracy is ~97% but validation accuracy is stuck at ~40%.

I can not understand the meaning of two concepts and their relationship.


texts:
Initially, I have a dataset where for each row there is user_id and product_ids he bought.

In that dataset there are 478191 products bought by different users.

In order to discover frequent items that are bought together, I will use association rules, apriori algorithm.
As apriori algorithm expects to have the features one-hot encoded.   I need the product_ids one-hot encoded.

Scikit-learn one-Hot encoding (sparse matrix=True) resulted in memory error. What other methods can I try? (using Python)


texts: Predicting Item Ratings with the Log Likelihood Ratio

texts:
I'm trying to infer prediction ratings from an item-item similarity matrix where the similarity score is calculated via the log-likelihood ratio (LLR). I'm using this code snippet to calculate the LLR (taken from this medium article):
import numpy as np

def xLogX(x):
    return x * np.log(x) if x != 0 else 0.0

def entropy(x1, x2=0, x3=0, x4=0):
    return xLogX(x1 + x2 + x3 + x4) - xLogX(x1) - xLogX(x2) - xLogX(x3) - xLogX(x4)

def LLR(k11, k12, k21, k22):
    rowEntropy = entropy(k11 + k12, k21 + k22)
    columnEntropy = entropy(k11 + k21, k12 + k22)
    matrixEntropy = entropy(k11, k12, k21, k22)
    if rowEntropy + columnEntropy < matrixEntropy:
        return 0.0
    return 2.0 * (rowEntropy + columnEntropy - matrixEntropy)

def rootLLR(k11, k12, k21, k22):
    llr = LLR(k11, k12, k21, k22)
    sqrt = np.sqrt(llr)
    if k11 * 1.0 / (k11 + k12) < k21 * 1.0 / (k21 + k22):
        sqrt = -sqrt
    return sqrt

rootLLR(100, 10, 10, 50000) # 36.18239222320234

To get a better understanding of the LLR check out this article by Ted Dunning. The issue I'm having is the LLR scores are not bounded (higher is better). For example, using the cosine similarity, the scores are bound from -1 to 1. Whereas with LLR this is not the case so calculating the weighted sum for rating prediction is difficult.
I've seen this paper (page 6) which normalises the LLR score but does not state how. The table from the paper is shown below.


texts: Is it correct to use non-target values of test set to engineer new features for train set?

texts:
Suppose, I have a dataset with a feature_1 value and a target value. Now, I want to engineer a new feature by creating relative value by subtracting mean from each value.

Question: Can I (1) use feature_1 value of test set to calculate mean or (2) should I use only the train set values?

If (1) is correct than I can use the same mean for test set and train set by calculating the mean of feature_1 for all dataset. I'm not sure it's legal, because here we use information from the test set in the train set. On the other hand, we don't use target value, so it might be ok.

If (2) is correct, then, I suppose, we can't use test to calculate the mean for train set, but we can use train set feature_1 values to calculate mean for test set. But then train and test sets' means might be different and influence the correctness of the model for test set. I could use the mean of train set for test set, but again I'm not sure it's correct.

It might be irrelevant for a large dataset, because influence of each value for mean is negligible, so I suppose (1) is ok her. But if I have very small dataset of, say, less than 30 samples or, if e.g. I want to generate new feature by calculating relative value of feature_1 for each category in some categorical feature_2 by calculating mean of all the samples belonging to the same category. Then, it might turn out to be just a few samples in some category of feature_2, so that each sample would influence mean greatly.


texts: Anomaly detection in structured textual data

texts:
Pls refer screenshot for sample data. As can be seen most of the fields in data are textual and highly correlated but each row has unique values and hence won't be right to call it categorical. I tried to break down column 1 in two tokens to ABC and X1 and tried KModes clustering where number of clusters = unique values in column1. However each cluster do not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.
I want to have unsupervised algo where I can force it to use Key1+Key2 as composite primary key for clustering. Algo should then ensure data follows some pattern in each cluster e.g. (alias1, alias2 moves in unison. Multipliers are in close range.)
Kindly suggest what would be the best anomaly detection algo and how to approach this problem.


texts: k-means python implementation: unclear\wrong result

texts:
I have implemented two type of clustering in python, using SciPy: one with hierarchical approach, and the other one with k-means.

In each cases I have used as input a two dimensional array X (documents, binary features) of 628 * 492.

#Hierarchical:

Z = linkage(X, 'ward')
clusters = fcluster(Z, max_k, criterion='maxclust').tolist()

#k-means:

clusters = kmeans2(X, k, iter_, minit="points")[1].tolist()

In the first case (hierarchical one) I got an array of size 628 (an array which the i-th element is the i-th document associate cluster)

In the second one I got an array of size 627. Why?

I got also this warning in the k-means execution (using different values for k):

/usr/lib/python3/dist-packages/scipy/cluster/vq.py:600:
UserWarning: One of the clusters is empty.
Re-run kmean with a different initialization.
warnings.warn("One of the clusters is empty. "

The cluster empty is the one associate to the missing document?
If the document missing would be in the "middle" of the list this situation wouldn't led to incoherent results? (list indexing wrong)


texts: Twitter POS and NER: What is state-of-the-art?

texts: What is the current state-of-the-art for pos tagging and named entity recognition for twitter data? Are industrial-strength programs like Spacy and SparkNLP accurate for such texts? How about FlairNLP and Stanford's CoreNLP accuracy measures?


texts:
I'm building a system able to classify emails into different categories (positive, negative, out of office, etc...) and I'm looking for a dataset of already classified emails to avoid hand classification on a 70k database.

I know it exists the Enron email dataset but do you know if it exists a version of this dataset with classified emails ? Or any other already classified email dataset ?


texts: When would C become nescessary to do an analysis or manage data?

texts: I use python in my day to day work as a research scientist and I am interested in learning C.  When would a situation arise where python would prove insufficient to manipulate data?


texts: How to aggregate features to a group level as a feature in machine learning model?

texts: I am building a model to predict some behavior at a household level. I could roll up income or number of cars etc so that I can take everyone into consideration. But how can I roll up something like age? I don't know if it makes sense to add up ages or education. Is there a cleaver way to still use those features at a aggregated level?


texts: How to implement clustering on categorical feature vectors in R?

texts:
I have a dataset like:

 id color  body  eyes
 1  A  blue  slim green
 2  B black   fat  blue
 3  A black  slim black
 4  C green  slim  blue
 5  D black medim black

whereas each id represents an individual with his individual physical characteristics.

Reproducible:

 structure(list(id = structure(c(1L, 2L, 1L, 3L, 4L), .Label = c("A",
"B", "C", "D"), class = "factor"), color = structure(c(2L, 1L,
1L, 3L, 1L), .Label = c("black", "blue", "green"), class = "factor"),
body = structure(c(3L, 1L, 3L, 3L, 2L), .Label = c("fat",
"medim", "slim"), class = "factor"), eyes = structure(c(3L,
2L, 1L, 2L, 1L), .Label = c("black", "blue", "green"), class = "factor")),
 .Names = c("id",
"color", "body", "eyes"), class = "data.frame", row.names = c(NA,
-5L))

Then number of the characteristics is fixed (color: blue/black/green, body: slim/fat/medium, eyes: green/blue/black).

My aim is to cluster those individuals.

My conceptual question regards the approach:

A simple correlation could be a first step. A question could be: how the combination of these characteristic is likely to appear in groups of individuals?

A more complicated approach. Maybe k-means clustering. How can address this given that these are categorical variables? should I convert them into dummies?

I'm new to this kind of analysis and any hint/reference to the implementation in R is highly appreciated!
Thanks


texts: Azure automl time series forecasting error

texts:
I'm using Microsoft Azure automl to try and generate models for time series forecasting but I keep getting an error:
Error: Could not determine the data set time frequency. All series in the data set have one row and no freq parameter was provided. Please provide the freq (forecast frequency) parameter or review the time_series_id_column_names setting to decrease the number of time series.

My data set looks like:
          Date,       Temp,
1,2019-05-07 13:51:00,25.19,
2,2019-05-07 13:51:58,25.14,
3,2019-05-07 13:53:00,25.14,
4,2019-05-07 13:54:00,25.14,
5,2019-05-07 13:55:00,25.1,
6,2019-05-07 13:55:58,25.1,
7,2019-05-07 13:57:00,25.1,
8,2019-05-07 13:57:58,25.11,
9,2019-05-07 13:58:58,25.11,
10,2019-05-07 14:00:00,25.07,
11,2019-05-07 14:01:00,25.07,
12,2019-05-07 14:02:00,23.11,

I followed the steps on the ml.azure.com gui, I don't see a place for freq or time_series_id_column_names
Seems to be an issue with the data set, any ideas on how I can prevent/fix this error would be great, thanks!


texts: How to choose appropriate epsilon value while approximating gradients to check training?

texts:
While approximating gradients, using actual epsilon to shift the weights results in wildly big gradient approximations, as the "width" of the used approximation triangle is disporportionately small. In Andrew NG-s course, he is using 0.01, but I suppose it's for example purposes only.
This makes me wonder, is there a method to chose the appropriate epsilon value for gradient approximation based on e.g. the current error value of the network?


texts: is gradient descent also used during feed forward propagation in Neural Network?

texts: To the best of my understanding , weights are updated during back propagation only using gradient descent and no gradient descent is used during feed forward propagation. is it correct ?


texts: How to obtain a count of the classes of a categorical var within a certain time interval for a time-stamped data?

texts:
I have a dataframe with several categorical variables. But for simplicity let's assume there is only 1 categorical variables with 3 classes. I want to obtain the counts of these classes within a certain time-intervals, say 15mins. To make it easier to understand what I am looking for, here is a toy example and the output I am looking for.
                    _time       AN
0 2019-04-09 16:00:00.050        a
1 2019-04-09 16:00:00.050        a
2 2019-04-09 16:00:00.050        b
3 2019-04-09 16:00:00.050        a
4 2019-04-09 16:00:00.050        b
5 2019-04-09 16:02:38.992        a
6 2019-04-09 16:06:41.884        c
7 2019-04-09 16:15:00.051        a
8 2019-04-09 16:15:00.051        b
9 2019-04-09 16:15:00.051        a

The output that I am looking for is below:
_time               AN
2019-04-09 16:00:00 a                4
                    b                2
                    c                1
2019-04-09 16:15:00 a                2
                    b                1

So for this toy example, the timeline has two 15mins intervals. For the first one, 'a' appears 4 times, 'b' appears 2 times and 'c' appear 1 time. In the 2nd 15min interval, 'a' appears 2 times and 'b' appear only 1 time and 'c' doesn't appear at all.
I obtained this result by running this code:
c.resample('15T',on = '_time').agg({'AN':'value_counts'})

However, when I am running this same code on the entire dataframe, df, I am getting the following error:
ValueError: operands could not be broadcast together with shape (10267,) (4315,)
I am not sure why I am getting this. Is there a different method to obtain the same result? or is there any suggestion to fix this error?
Thanks in advance.


texts: How to get feature importance from a keras deep learning model?

texts:
In case of scikit-learn's models, we can get feature importance using the relevant attributes of the model.
I've been working on a RNN, using LSTMs for text embedding.
Is there any way to get feature importance of various features from the finalized model?


texts:
I am new to machine learning and I am trying to build a classifier. My problem is that I am not able to import the dataset I need. In particular, I put my dataset in the Desktop and what I did is:

#pakages
import numpy as np
import pandas as pd
import jsonlines                   #edit

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import *
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

print('Libraries imported.')

now, after these imports I want to use the function

training_set = pd.read_json('\Desktop\training_dataset.jsonl')  #edit
print(training_set.head())

to import my dataset. The problem is that what i get is:

ValueError                                Traceback (most recent call last)
<ipython-input-16-f789503c3c7c> in <module>
----> 1 training_set = pd.read_json('\Desktop\training_dataset.json')
      2 print(training_set.head())

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in
read_json(path_or_buf, orient, typ, dtype, convert_axes, convert_dates,
keep_default_dates, numpy, precise_float, date_unit, encoding, lines,
chunksize, compression)
    590         return json_reader
    591
--> 592     result = json_reader.read()
    593     if should_close:
    594         try:

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in read(self)
    715             obj =
self._get_object_parser(self._combine_lines(data.split("\n")))
    716         else:
--> 717             obj = self._get_object_parser(self.data)
    718         self.close()
    719         return obj

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in
_get_object_parser(self, json)
    737         obj = None
    738         if typ == "frame":
 --> 739             obj = FrameParser(json, **kwargs).parse()
    740
    741         if typ == "series" or obj is None:

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in parse(self)
    847
    848         else:
--> 849             self._parse_no_numpy()
    850
    851         if self.obj is None:

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in
_parse_no_numpy(self)
   1091         if orient == "columns":
   1092             self.obj = DataFrame(
-> 1093                 loads(json, precise_float=self.precise_float),
dtype=None
   1094             )
   1095         elif orient == "split":

ValueError: Expected object or value

and I can't understand why. Can somebody please help me? Thank's in advance.

[EDIT] the file is a .jsonl file, but yet I don't know how to import the dataset because I cannot use .read_json
      ,I have tried this:

openfile=open('Desktop\training_dataset.jsonl')
jsondata=json.load(openfile)
df=pd.DataFrame(jsondata)
openfile.close()
print(df)

but gives me the following error message:

OSError                                   Traceback (most recent call last)
<ipython-input-28-2422c1a9a77b> in <module>
----> 1 openfile=open('Desktop\training_dataset.jsonl')
      2 jsondata=json.load(openfile)
      3 df=pd.DataFrame(jsondata)
      4 openfile.close()
      5 print(df)

OSError: [Errno 22] Invalid argument: 'Desktop\training_dataset.jsonl'

[EDIT 2] by doing as suggested, so:

with open("\Desktop\training_dataset.jsonl") as datafile:
data = json.load(datafile)
dataframe = pd.DataFrame(data)

I again obtain another error message, which is:

OSError                                   Traceback (most recent call last)
<ipython-input-47-1365f26e6db5> in <module>
 ----> 1 with open("\Desktop\training_dataset.jsonl") as datafile:
      2     data = json.load(datafile)
      3 dataframe = pd.DataFrame(data)

OSError: [Errno 22] Invalid argument: '\\Desktop\training_dataset.jsonl'

but I don' understand, because my dataset is placed in my desktop.


texts:
Thanks for reading.

I'm currently reading Tom Mitchell's Machine Learning (I'm a beginner into ML), and I'm on chapter 4 about perceptrons. I'm really confused about this paragraph:

I understand the AND but I'm still not so sure how OR can be represented.

For example, if the inputs are:

$X_1 = +1$

$X_2 = -1$

In that case, using his logic for OR we would end up with the perceptron actually outputting -1, because:

$1*0.5  + -1*0.5 - .3 = -.3$

The function would give us a negative number. Is he wrong about the OR function? How would a single perceptron act as an OR?

Thank you!


texts: What are image-wise nearest neighbour classifier and pixel-wise nearest neighbour classifier?

texts:
I'm reading the paper "Few-Shot Semantic Segmentation with Prototype Learning", and on the figure 2, page 5, there is a image-wise nearest neighbour classifier but it doesn't explain what is.
I'm not English, and the problem is that I don't understand the meaning of "Image-wise" or "pixel-wise".
How a Image-wise (or pixel-wise) nearest neighbour classifier? Or in other words, what does it do?


texts: XGBoost: # rounds is equal to n_estimators?

texts:
I'm running a regression XGBoost model and trying to prevent over-fitting by watching the train and test error using this code:

eval_set = [(X_train, y_train), (X_test, y_test)]

xg_reg = xgb.XGBRegressor(booster='gbtree',
                          objective ='reg:squarederror',
                          max_depth = 6,
                          n_estimators = 100,
                          min_child_weight = 1,
                          learning_rate = 0.05,
                          seed = 1,early_stopping_rounds = 10)

xg_reg.fit(X_train,y_train,eval_metric="rmse", eval_set = eval_set, verbose = True)

This prints out as follows:

[93]    validation_0-rmse:0.233752      validation_1-rmse:0.373165
[94]    validation_0-rmse:0.2334        validation_1-rmse:0.37314
[95]    validation_0-rmse:0.232194      validation_1-rmse:0.372643
[96]    validation_0-rmse:0.231809      validation_1-rmse:0.372675
[97]    validation_0-rmse:0.231392      validation_1-rmse:0.372702
[98]    validation_0-rmse:0.230033      validation_1-rmse:0.372244
[99]    validation_0-rmse:0.228548      validation_1-rmse:0.372253

However, I've noticed the number of training rounds printed out and in the evals_results always equals the n_estimators.

In [92]: len(results['validation_0']['rmse'])
Out[92]: 100

If I change the number of trees to 600, the # of rounds goes up to 600, etc.  I was under the impression that what's being printed is the metric result from each round of training, which includes training all the trees at once.

What is going on here?  Is each layer of trees considered a separate training round?


texts: What issue is there, when training this network with gradient descent?

texts: Suppose we have the following fully connected network made of perceptrons with a sign function as the activation unit, what issue arises, when trying to train this network with gradient descent?


texts:
Hi i'd like to know a bit more about kNN-like approach implementations for classification problems, and specifically classification problems where we want to have a probability distribution as an output (to compute logloss like metrics for example)

In packages such as knn for R, a distance matrix is asked as an input. but thats highly inefficient for even medium sized data sets, as such matrix size is n^2 where n is the number of data.

but i can see an easy (empirical) approach where we hash all data into hypercubes, and return for the test set, the implied observed distribution of the train data in the corresponding hypercube it hashes into.

Of course, such an approach has many flaws - it will suffer the curse of dimensionality, but might it not work in there is a lot of data, but not many variables (2,3,4,5 ?) for example ?? - if the data density is heterogeneous, there would be some hypercubes with little data, and others a lot of data. one might then like to zoom on some areas which are overpopulated....

the advantage of hashing is that its computationally efficient and more like linear time/memory than quadratic time/memory

I don't think i'm discovering the moon, so what do you guys think about it ? if there already some algos/libraries basically doing that or much better, which are those ?


texts: Weighted loss functions vs weighted sampling?

texts:
For image classification tasks, is there a practical difference between using weighted loss functions vs. using weighted sampling? (I would appreciate theoretical arguments, experience or published papers, anything really.)

Some details:

By "weighted sampling", I mean attributing different sampling probabilities for each sample in the training set.

By "weighted loss functions", I mean weighting error terms differently depending on the sample considered.


texts: Why an increasing validation loss and validation accuracy signifies overfitting?

texts:
When I train a neural network, I observe an increasing validation loss, while at the same time, the validation accuracy is also increased.
I have read explanations related to the phenomenon, and it seems an increasing validation loss and validation accuracy signifies an overfitted model.
However, I have not really grokked the reasons why an increasing validation loss and validation accuracy signifies an overfitting.
Could you please give the explanations behind this phenomenon?


texts: how to improve searching index in dataframe

texts:
Given a pandas dataframe with a timestamp index, sorted.
I have a label and I need to find the closest index to that label.
Also, I need to find a smaller timestamp, so the search should be computed in the minor timestamps.
Here is my code:

import pandas as pd
import datetime

data = [i for i in range(100)]
dates = pd.date_range(start="01-01-2018", freq="min", periods=100)
dataframe = pd.DataFrame(data, dates)

label = "01-01-2018 00:10:01"
method = "pad"
tol = datetime.timedelta(seconds=60)
idx = dataframe.index.get_loc(key=label, method="pad", tolerance=tol)

print("Closest idx:"+str(idx))
print("Closest date:"+str(dataframe.index[idx]))

the searching is too slow. Is there a way to improve it?


texts: LightFM implementation for scala/java on spark

texts:
I am looking for hybrid recommendation libraries such as lightfm that I can use on Spark (with Scala).
Any alternative?
Or best would be for me to build a hybrid recommendation system on spark's mllib from scratch, combining the collaborative step and content filtering one.
Thanks a lot!


texts: Correcting for seasonality on multiple timelines

texts:
I have many time-series of economic data, broken down quarterly. There is always an element of seasonality in each timeline, but it can very wildly. See the below charts:

I'd like to correct for seasonality, but the normal factors to consider (like this) are different for each one.
I'm tempted to just do a rolling 4-quarter average, but I'm not sure how much nuance I would lose there.
How do I correct them all? Do I need to evaluate each one (over 4500) to figure out which way to correct for seasonality? Or will a simple approach suffice here?
Thanks in advance.


texts: Pytorch - Gradient distribution between functions

texts:
https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb

Hi I am trying to understand the NN with pytorch.
I have doubts in gradient calculations..

import torch.optim as optim

create your optimizer
optimizer = optim.SGD(net.parameters(), lr=0.01)

# in your training loop:
optimizer.zero_grad()   # zero the gradient buffers
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()    # Does the update

From the about code, I understood loss.backward() calculates the gradients.
I am not sure, how these info shared with optimizer to update the gradient.

Can anyone explain this..

Thanks in advance !


texts: NLP varying amount of features and BoW as feature concatenating to feedforward NN

texts:
I was looking at Google's Smart linkify machine learning models, as it closely relates to a personal project. And couldn't quite understand how the features are fed to the neural network.

It's about the following:

  Given a candidate entity span, we extract: Left context: five words
  before the entity, Entity start: first three words of the entity,
  Entity end: last three words of the entity (they can be duplicated
  with the previous feature if they overlap, or padded if there are not
  that many), Right context: five words after the entity, Entity
  content: bag of words inside the entity and Entity length: size of the
  entity in number of words. They are then concatenated together and fed
  as an input to the neural network.

I have 4 primary questions which I cant find a clear answer to:

The article specifies the features are concatenated. How does
a concatenation layer work internally? Does it concatenate all the
values in a single variable, in a very literal sense? how does that
work computationally?
How can a Bag of Words be a feature, when its a key-value pair? Or is it
also just all concatenated into one variable. Which again, how can
that work computationally?
The text specifies multiple words are used as a single feature; e.g. Left context: five words before the entity. Is this again concatenating the embedding / vectors?
Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many) does this mean a variable amount of features as input to the NN (or concatenation layer) or is this more intended as a configuration? Less context available so fewer amount of 'hard' coded input features?

Perhaps a simple keras model with some hardcoded input variables would help shape the answer.

Is there more material online to understand and recreate the entity recognition model?

EDIT:

The article also mentions the lack of available context or entities. E.g. the diagram shows 4 features for entity (2 for entity start, 2 for entity end). The article mentions duplication, but duplicating 3 times doesnt sound like a great idea. Would a convolution layer with a filter (1x3) work better?

And how would a Keras model look like then? Would it have two separate input layers? One input layer with 10 input features for the context + 1 feature for BoW + 1 feature for entity length. And another input layer with 4 input features followed by a convolution layer. And then both layers lead to a concatenate layer?


texts: How to put KerasClassifier, Hyperopt and Sklearn cross-validation together

texts:
I am performing a hyperparameter tuning optimization (hyperopt) tasks with sklearn on a Keras models. I am trying to optimize KerasClassifiers using the Sklearn cross-validation, Some code follows:

def create_model():
model = Sequential()
model.add(
    Dense(output_dim=params['units1'],
          input_dim=features_.shape[1],
          kernel_initializer="glorot_uniform"))
model.add(Activation(params['activation']))
model.add(Dropout(params['dropout1']))
model.add(BatchNormalization())
...
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

return model

Now what I want to do is to pass the Hyperopt params to KerasClassifier using the following way

def objective(params, n_folds=N_FOLDS):
"""Objective function for Hyperparameter Optimization"""

# Keep track of evals
global ITERATION

ITERATION += 1

clf = KerasClassifier(build_fn=create_model,**params)

start = timer()

# Perform n_folds cross validation
cv_results = cross_val_score(clf,
                             features_,
                             labels,
                             cv=5
                             ).mean()

run_time = timer() - start

# Loss must be minimized
loss = -cv_results

# Dictionary with information for evaluation
return {
    'loss': loss,
    'params': params,
    'iteration': ITERATION,
    'train_time': run_time,
    'status': STATUS_OK
}

I define the search space as:

space = {
'units1':
hp.choice('units1', [64, 128, 256, 512]),
'units2':
hp.choice('units2', [64, 128, 256, 512]),
'dropout1':
hp.choice('dropout1', [0.25, 0.5, 0.75]),
'dropout2':
hp.choice('dropout2', [0.25, 0.5, 0.75]),
'batch_size':
hp.choice('batch_size', [10, 20, 40, 60, 80, 100]),
'nb_epochs':
hp.choice('nb_epochs', [10, 50, 100]),
'optimizer':
opt_search_space,
'activation':
'relu'

}

Run optimization

best = fmin(fn = objective, space = space, algo = tpe.suggest,
            max_evals = MAX_EVALS, trials = bayes_trials, rstate = np.random.RandomState(50))

But it fails giving this error:

ValueError: activation is not a legal parameter

What's the right way to do it?


texts: What toolbox to use to create multi-output random forest(reggression) with custom spltting function at each node?

texts:
I am trying to implement "Real Time Head Pose Estimation fromConsumer Depth Cameras" by Fanelli et al.
I need to train a random forest(regression) with the following criterion

The predicted output is multivariate(yaw,pitch).
The input features is four dimensional float type.
The split function at each node is a custom function which needs to be computed at run-time using one of the features that has been randomly selected.
The custom function needs to evaluated using a separate user created code snippet which might then be used by the random forest code.

My question is what random forest software frame work would support custom split functions and random feature selection for evaluating the custom function.

I have checked random forest implementations of scikit-learn, opencv, weka to begin with, but found that they don't satisfy my constraints


texts: Regarding Class Balancing in Deep Neural Network

texts:
I was trying to do class balancing on the image semantic segmentation problem for some classes in the images are in the minority. The weight for each class is calculated as mentioned in this paper.

we weight each pixel by αc = median_freq / freq(c) where freq(c) is the number of pixels of class c divided by the total number of pixels in images where c is present, and median_freq is the median of these frequencies.

Then I weighted the cross entropy loss as follows, the size of the label is (img_col, img_row, num_class) for the labels are one-hot-labels:
def weighted_cce(coding_dist, true_dist, weights):
    # calculate weighted cross entropy loss
    # true_dist: ground truth, coding_dist: predicted
    coding_dist = T.clip(coding_dist, 10e-8, 1.0-10e-8)
    return -T.sum(weights * true_dist * T.log(coding_dist), axis=coding_dist.ndim-1)

What's strange is that instead of producing a more balanced output, the result is even more biased than without class balancing, namely the network now can only recognize the most dominant classes in the images.
Could anyone share some thoughts on this? Thanks in advance!


texts: Trying to use different image size to test my trained model

texts:
I have built my model using images which are all 512x384, I then exported the model through .pkl and am hosting it on Render, the UI is built on React App where the user will input their chosen image
I am aware that I could resize the images, either on the client side when the user inputs the image, on the server side using python, or I could change my model to accept different image sizes, could anyone advise what the simplest, most elegant solution would be for this?


texts: Logistic regression GD implementation in Python

texts:
I am implementing logistic regression in Python with the regularized loss function like this:

 But the gradient algorithm works bad. Read the bold text first, please! Just paste the code cell by cell

import numpy as np, scipy as sp, sklearn as sl
from scipy import special as ss
from sklearn.base import ClassifierMixin, BaseEstimator
from sklearn.datasets import make_classification
import theano.tensor as T

Here is the loss function: (scipy is to "clip" the logarithm's arg near 1)

def lossf(w, X, y, l1, l2):
     w.resize((w.shape[0],1))
     y.resize((y.shape[0],1))

     lossf1 = np.sum(ss.log1p(1 + ss.expm1(np.multiply(-y, np.dot(X, w)))))
     lossf2 = l2 * (np.dot(np.transpose(w), w))
     lossf3 = l1 * sum(abs(w))
     lossf = np.float(lossf1 + lossf2 + lossf3)
     return lossf

Here is the gradient function:(??PROBLEM HERE?? -see the end)

def gradf(w, X, y, l1, l2):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    gradw1 = l2 * 2 * w
    gradw2 = l1 * np.sign(w)
    gradw3 = np.multiply(-y,(2 + ss.expm1(np.multiply(-y, np.dot(X, w)))))
    gradw3 = gradw3 / (2 + (ss.expm1((np.multiply(-y, np.dot(X, w))))))
    gradw3 = np.sum(np.multiply(gradw3, X), axis=0)
    gradw3.resize(gradw3.shape[0],1)
    gradw = gradw1 + gradw2 + gradw3
    gradw.resize(gradw.shape[0],)
    return np.transpose(gradw)

Here is my LR class:

class LR(ClassifierMixin, BaseEstimator):
    def __init__(self, lr=0.0001, l1=0.1, l2=0.1, num_iter=100, verbose=0):
        self.l1 = l1
        self.l2 = l2
        self.w = None
        self.lr = lr
        self.verbose = verbose
        self.num_iter = num_iter

def fit(self, X, y):
    n, d = X.shape
    self.w = np.zeros(shape=(d,))
    for i in range(self.num_iter):
        g = gradf(self.w, X, y, self.l1, self.l2)
        g.resize((g.shape[0],1))
        self.w = self.w - g
        print "Loss: ", lossf(self.w, X, y, self.l1, self.l2)
    return self

def predict_proba(self, X):
    probs = 1/(2 + ss.expm1(np.dot(-X, self.w)))
    return probs

def predict(self, X):
    probs = self.predict_proba(X)
    probs = np.sign(2 * probs - 1)
    probs.resize((probs.shape[0],))
    return probs

Here are the tests:

X, y = make_classification(n_features=100, n_samples=100)
y = 2 * (y - 0.5)
clf = LR(lr=0.000001, l1=0.1, l2=0.1, num_iter=10, verbose=0)
clf = clf.fit(X, y)
yp = clf.predict(X)
yp.resize((100,1))
accuracy = int(sum(y == yp))/len(y)

This doesn't converge. But if I replace my gradw3 with theano:

gradw3 = get_gradw3(w,X,y)

where:

w,X,y = T.matrices("wXy")
logloss = T.sum(T.log1p(1 + T.expm1(-y* T.dot(X, w))))
get_gradw3 = theano.function([w,X,y],T.grad(logloss,w).reshape(w.shape))

it converges to 100% accuracy. That means, my gradw3 is implemented wrong, but I can't find a mistake.


texts: Daily access pattern of a real cloud storage

texts:
for a current research project I would need file access traces of real world cloud storages. Unfortunately I was not able to find any yet.

More precisely I need real world traces with at least the following information: filename, file size, access time, operation (put, post, get, delete). The kind of data (if the files are images, textdata or something else) does not matter. Furthermore, it is also ok if it is not a cloud storage, but for example an online image gallery, as long as all 4 operations (put, post, get, delete) are included its fine for me.

Any help would be appreciated.


texts:
I want to find the minimum value of the objective function if we set K equal to the number of samples.

I know the objective function is $J=\Sigma_{n=1}^{N}\Sigma_{k=1}^{K}r_{nk}||x_n-\mu_k||^2$
And we take the derivative to get $\mu_k=\frac{\Sigma_{n=1}^{N}r_{nk}x_n}{\Sigma_{n=1}^{N}r_{nk}}$

However, when plugging that back in and set K=N, it seems unable to be simplified:
$J=\Sigma_{n=1}^{N}\Sigma_{k=1}^{N}r_{nk}||x_n-\frac{\Sigma_{n=1}^{N}r_{nk}x_n}{\Sigma_{n=1}^{N}r_{nk}}||^2$

Any help?


texts: How do I convert a pandas dataframe to a 1d array?

texts:
I'm trying to create a contour map from two variables which store some temperature values and a third variable which is the time stamp.
I used this notebook as a tutorial

  https://plot.ly/pandas/contour-plots/

I'm not able to convert the pandas dataframe created, into a 1d array. And the kde_scipy doesn't work with a nd-array. I tried converting the dataframe into a 1d array using .as_matrix() but this is the error I am receiving.

Degrees of freedom <= 0 for slice

How can I convert this CSV file (with 3 columns of data) imported as a dataframe into individual columns of data? Or can I directly import each column of data into a 1d array and use it in the function kde_scipy?


texts: how to prevent machine crash while searching for hyper parameters of XGBoost with GridSearchCV

texts:
I am searching for best hyper parameters of XGBRegressor using GridSearchCV. Here is the code:
from xgboost import XGBRegressor
from sklearn.model_selection import GridSearchCV, KFold

param_grid_xgb_b1 = {
    'num_leaves': np.arange(20, 500, 5),
    'max_depth': np.arange(5, 50, 2),
    'reg_alpha': np.arange(0.01, 0.5, 0.01),
    'min_data_in_leaf': np.arange(50, 1_000, 10),
    'colsample_bytree': [0.65, 0.75, 0.85, 0.95, 1],
    'subsample': [0.65, 0.75, 0.85, 0.95, 1]
    }
kfold_xgb_b1 = KFold(n_splits=5, shuffle=True, random_state=42).split(X=X_train, y=y_train)
grid_search_xgb_b1 = GridSearchCV(estimator=XGBRegressor(learning_rate=0.01, n_estimators=20_000, random_state=0,
                                                         tree_method='gpu_hist', subsample_freq=5),
                                  param_grid=param_grid_xgb_b1,
                                  scoring=rmse_score, cv=kfold_xgb_b1)
grid_search_xgb_b1.fit(X=X_train, y=y_train, verbose=3)

I am using Google Colab with GPU. Machine is dual core CPU with 12.72GB of RAM. And GPU is Tesla T4 with CUDA Version 10.1.
How do I prevent crash and search for hyper parameters?
EDIT 1:
Crash type: RAM filling up;
size of data: 300,000 rows and 40 columns.


texts: Implementing custom TF/Keras loss function to ignore false negatives of a specific class during semantic segmentation

texts:
This is a cross post from StackOverflow but more succinct.
I have a semantic segmentation neural network which predicts on 6 classes. The first class, class 0, I wish to ignore false negatives for. That is, when the predicted value of a truly class 0 pixel is incorrect, I wish to ignore its contribution to my loss. To implement this behavior I am attempting to design an algorithm to alter the predicted tensor such that every pixel location which is truly class 0 is always class 0 (This way I can continue using whatever loss function I would like on the altered predicted value).
Based on my Stack Overflow question, this has put me on quite the journey. Essentially the entire hangup is working with immutable tensors within the custom loss function. I cannot implement the algorithm described because I cannot technically alter the predicted tensor. I have investigated the "workaround" of using .numpy() on the tensors (by setting run_eagerly=True in Model.compile(...)), but per this SO Answer this necessarily removes the tensor from the graph and results in no gradients being able to be calculated. I have also investigated tf.Variable'ing the predicted tensor so that I can mutate it, but again this also necessarily removes it from the graph and results in the same issue (under the hood it looks like _numpy() is called).
Thus my current best attempts (using .numpy())
def weighted_categorical_crossentropy_ignore(weights):
    weights = K.variable(weights)

    def loss(y_true, y_pred):
        yp = y_pred.numpy()
        yt = y_true.numpy()
        yp[np.nonzero(np.all(yt == [1, 0, 0, 0, 0, 0], axis=3))] = [1, 0, 0, 0, 0, 0]

        # Continue calculating weighted categorical crossentropy
        # -------------------------------------------------------
        # Scale predictions so that the class probs of each sample sum to 1
        yp /= K.sum(yp, axis=-1, keepdims=True)
        # Clip to prevent NaN's and Inf's
        yp = K.clip(yp, K.epsilon(), 1 - K.epsilon())
        loss = y_true * K.log(yp) * weights
        loss = -K.sum(loss, -1)
        return loss
    return loss

and using tf.Variable
def weighted_categorical_crossentropy_ignore(weights):
    weights = K.variable(weights)

    def loss(y_true, y_pred):
        # yp[np.nonzero(np.all(yt == [1, 0, 0, 0, 0, 0], axis=3))] = [1, 0, 0, 0, 0, 0]
        yp = K.variable(y_pred)
        yt = K.variable(y_true)
        #np.all
        x = K.all(yt == [1, 0, 0, 0, 0, 0], axis=3)
        #np.nonzero
        ne = tf.not_equal(x, tf.constant(False))
        y = tf.where(ne)
        # Perform the desired operation
        yp[y] = [1, 0, 0, 0, 0, 0]

        # Continue calculating weighted categorical crossentropy
        # -------------------------------------------------------
        # Scale predictions so that the class probs of each sample sum to 1
        #yp /= K.sum(yp, axis=-1, keepdims=True) # Cannot use \= on tf.var, must use var = var /
        yp = yp / K.sum(yp, axis=-1, keepdims=True)
        # Clip to prevent NaN's and Inf's
        yp = K.clip(yp, K.epsilon(), 1 - K.epsilon())
        loss = y_true * K.log(yp) * weights
        loss = -K.sum(loss, -1)
        return loss
    return loss

And now I am currently at a loss on how to continue. How can I implement the desired behavior in my loss function?


texts: Is it possible to use Memory Augmented Neural Network for regression?

texts:
Now I am working on stock price prediction using Machine Learning. I have tried different methods (ARMA, LSTM, GRU). Now I would like to try MANN (Memory Augmented Neural Network). I have tried to find examples of use in Github and Kaggle. I could not find an example of how it can be used for regression, only for NLP and processing images.
Can it be used for regression? For example, input:
Date       Open Low High Close Volume Target
2020-12-01 5    4   6    4     100    7
2020-12-02 5    4   8    7     200    6

Output: Next day close should be predicted.
How can I use MANN for a regression task like this?


texts: Tensorflow predicting same value for every row

texts:
I have a trained model. For single prediction I restore the last checkpoint and pass a single image for prediction but the result is the same for every row. Does anyone have a clue of what might be wrong?

This is the code for prediction:

And this is the output:


texts: Quasi-linearity in deep learning regression problems (sports betting)

texts:
I’m attempting to build a sports betting model that aims to predict final scores for games.  I’ve had some promising early results for US college football just by using linear regression to form team rankings where each team’s coefficient represents how much it would win or lose by against an average team.

With football specifically, but perhaps all major sports, I think outcomes are very dependent on individual matchups, i.e., the specific strengths and weaknesses of one team versus the strengths and weaknesses of another team as teams will try to exploit relative weaknesses of their opponents.

Here’s a hypothetical set of matchups between the offense of one football team, Team A, and the defense of another team, Team B.

Team A Offensive Ratings:
  Passing: +7
  Rushing: -3

Team B Defensive Ratings:
  Passing: -3
  Rushing: +7

What this means is that Team A’s passing offense contributes an average of 7 points more to its final score than an average team, and its rushing offense contributes 3 points less than an average team.  In total, assuming that passing and rushing are the only two ways to score, Team A scores 4 more points per game than a hypothetical average team.  This same thinking would apply to defenses (i.e., the defensive rating also measures how a team’s defense contributes to its average margin of victory or loss).  In practice, it’s not as simple as looking at the nominal statistical outputs for each team since different teams play different levels of competition, but that’s a somewhat separate topic.

With the example above, it first appears that Team A’s offense is even with Team B’s defense because both are 4 points better than average, but that’s probably an oversimplification.  Given that Team A has a relative advantage in passing and a relative disadvantage in rushing, we can expect Team A to pass more than it rushes and perhaps pass more than it usually does due to Team B’s relative weakness in pass defense and relative strength in rushing defense.  The fact that teams and coaches will be at least somewhat rational in their decision making should, in theory, lead to some degree of non-linearity in final scores based on specific strength profiles of each team.

I’m just getting started with data science and have to admit that I’m yet to code a neural network (although I’ve read a lot about them).  This is fundamentally a regression problem, but most of what I’ve seen indicates that using neural nets for (linear) regression is largely done for efficiency purposes for very large datasets, which is not the case here.

Seemingly, I can create whatever activation functions I want, even with different functions within the same layers as long as I can find their derivatives.  Generally, I think the activation functions should respect linearity regarding the net ratings difference for each stat component, but the network needs to account for any non-linearities that exist as result of interactions between stats.

I realize this is very open ended.  I could design a large number of ad hoc activation functions and see what works and what doesn’t, but I’m wondering if there’s a more generic way to approach the problem.

Any suggestions are appreciated.


texts:
When using similarity measures (eg. Resnik Information Content, Cosine Similarity, etc.) for any type of data, are there any standard similarity thresholds that are used, or does it all depend on the situation? A similarity threshold would be the value X in [0,1] such that all pairs with similarity score greater than X are "connected" while ones with similarity score below X are not.
Also, are low similarity thresholds (~0.15) acceptable when higher thresholds simply do not produce enough "connected" pairs and having a low similarity threshold still works well in practice?


texts: How to test the quality of a word embedding?

texts:
I have trained a word2vec model using GenSim 4.
The problem is that my corpus is quite small.
How can I test the quality of the word embeddings I have obtained?
Is there some standard measures to do that?


texts: Is there a tool/ library/ algorithm that can learn from the steps I perform in cleaning text and apply them to similar data?

texts:
I have a dataset of organization names that is quite messy. I used all the popular NER tools on it without much success(I assume it's because they lack context). I resolved to using OpenRefine but I reached a dead end with it's filters not picking up a lot of similar strings. I would like to use the data that I cleaned so far with OpenRefine for a (preferably supervised) machine learning algorithm that can afterwards continue the cleaning.
Are there any resources that could help with this?


texts: How can I transform a sequence into features

texts:
When Machine Learning libraries don't support categorical features those features can be one-hot encoded into a series of binary feature columns.  I have a feature that represents a sequence or permutation of values and I want to transform it into something scikit-learn or similar ML libraries can use.  What are the well known ways of doing this?
In my problem a physical system is damaged and I'd like to use ML to recommend the sequence in which the damages should be repaired.  Due to limited resources and limited repair crews and equipment only a certain number of components can be repaired at any one time.  I've already determined a rough importance of the various components.  Typically repairing the most important components first works well.  But in specific situations non-obvious repair orders work even better.  I have a dataset with a couple of million data points.  For each data point I have the set of components that were damaged and the order that the repairs were undertaken as well as a metric of how well the strategy worked.
The number of components that can be damaged is fixed and approximately 1600.
In a realistic scenario there would be less than 50 damaged sub components.
Say there were four components A,B,C,D
Assume B, A and C were damaged but D was not.
In an example dataset there might be two entries:
[A,C,B] = 11
[B,A,C] = 7
I want to transform the [A,C,B,D] part into something I can give to a regressor or categorizer.
Approaches I have thought of so far:

One column per component with the order the component was repaired.  If a component wasn't damaged then the column might have 0 or N/A

A_repaired_at
B_repaired_at
C_repaired_at
D_repaired_at
result

1
3
2
0
11

2
1
3
0
7

Instead of using the rank, use a normalized rank. Seems like approach #1 wouldn't work well when the number of damaged components changes. Being repaired third out of three damaged components means something was repaired last but being repaired third out of 50 damaged components means a component was repaired towards the front.

A_repaired_at
B_repaired_at
C_repaired_at
D_repaired_at
result

1/3
3/3
2/3
0
11

2/3
1/1
3/3
0
7

Use comes-before attributes directly.  #2 seems like it makes it possible for an ML library to compare the repair positions between rows - if that is how we'd like the ML to function maybe I should add those features directly.

A_before_B
A_before_C
A_before_D
B_before_C
B_before_D
C_before_D
result

True
True
na
False
na
na
11

False
True
na
True
na
na
7

Option #3 is nice because the columns are only {True,False,NA} but it has this nightmare inducing problem where my 1600 components become a million and some change feature columns.
Is there some other way to transform the sequence into useful ML features?


texts: Getting the earliest date (duplicates due to several call ids and agents)

texts:
I am trying to get the earliest call interaction, with only the agent that interacted at that call and the customer id. Using the following code, I still get duplicates:

select customer, agent, min(call)
from call_data;

I still get the following:


texts: Risk score from Neural Network classifier (more than 2 categories)

texts:
I am trying to use a Neural Network to perform multiclass classification. The classes represent Insurance Risk Level. The most risky level is Level 1, the least risk corresponds to Level 10. The labels came from Unsupervised clustering unannotated insurance data. The architecture is:

Input Layer 43 features

Dense, 1000 neurons, ReLu activation

Dense 500 neurons,  ReLu activation

Output 10 neurons, Softmax activation

In particular, for binary classification we can use single neuron as last layer. Then, we define the score to be the input value to this final layer. But for more than 2 classes I don't know what to do.

  My question: Is there a way to obtain a Risk score that is consistent, that is, the greater the Level is, the smaller the score is  ?  I tried to use logits, not normalized probabilites, etc. But not sure if there is a way to get a single consistent score from logits ? . Thanks for any hint on this.


texts: Number of units for first layer in Keras Sequential Model

texts:
I have a huge CSV structured dataset. I'm feeding that dataset to a Keras Sequential Model. My question is, can my Model have number of units greater than the number of input features?
At the moment, my features or csv columns are 118 and Model summary is as:

First layer of model has 256 units with elu activation.
Second layer 128 with elu activation
Third layer 64  with elu activation
Last layer 1 unit with segmoid activation

Now, I'm getting 100% accuracy on both training and testing data, which is surprising to me. For 50 features, the model was giving 94.25% accuracy on training and 95.68% accuracy on testing data. Accuracy can be even 99 or 99.XX% but 100% is something wrong I guess? If it's 100% on test data, it's probably not the overfitting..? Please guide me!


texts: Finding frequencies in a noisy, "uneven" dataset

texts:
I'm working on a problem where frequency analysis applies (decomposition of a signal into frequencies, that is), but it's noisy and the samples are unevenly spaced.

Specifically: given a list of items purchased at a bar/restaurant, try to estimate the number of guests on the check based on distinct "frequencies" of purchase. The logic is that if there are N guests on a check, then it's reasonable to see N frequencies of drinks being purchased, one person buying every 10 minutes, another every 15, etc. (Plenty of other properties of the check should be included, but here I'm focusing specifically on estimating distinct frequencies).

So more formally: given a noisy, unevenly spaced time series, find the smallest number of frequencies which reproduce the signal while minimizing the error (... for some sensible definition of how to minimize both the error and the number of frequencies simultaneously).

This is more a machine learning problem than signal processing. I realize it's also an open question, but can anyone point me in the right direction? Is there a particular method or algorithm that applies here?


texts: Find periodicity of a signal using python

texts:
I have a dataset that contains occurrences of the Kettle in a single-occupancy house for the duration of a month. In this dataset, ‘ts’ column indicates the unix-timestamp (this can be converted to date and
time) that the kettle was turned ON; and the ‘second’ column indicates the duration in seconds that the kettle was ON.
My requirement - Find routines in the data, evaluate their periodicities and variabilities. For instance, a routine could be “breakfast”, its periodicity might be every day at a certain time and variability might be a time window within the routine occurred.
What I've tried and I'm a bit unsure of - I plotted the autocorrelation of the virgin signal with different amounts of lag and found there was no significant correlation with any of its lagged versions. But on looking at the plot of this signal, I see there are peaks every day- morning time. Also, I've tried autocorrelation on the denoised(denoised using KNN) version and the under-sampled versions(retaining just first and last samples for each day of the month) and found that the denoised signal also isn't correlated with its lagged versions, but the under-sampled signal is.
I'm unsure first of all about how to find periodicity of a signal like this, and if the autocorrelation plot lets us make any conclusions about periodicity.
from statsmodels.tsa.stattools import acf
from statsmodels.graphics.tsaplots import plot_acf

# Compute the acf array
acf_array = acf(q4['seconds'])
print(acf_array)

# Plot the acf function
plot_acf(q4['seconds'],alpha=0.05)
plt.show()

More code showing what I've read can be found at - this link.


texts: How do feature selection on a sparse matrix?

texts:
Say I want to do features selection on a sparse matrix, i.e., 10,000 rows x 1500 features, but the matrix is mostly sparse. Let's say the features are all numeric and the target is binary and discrete.
What's the correct and efficient way to apply feature selection? Moreover, I'm interested in applying mutual information on it.


texts: Why the training loss increases, and predict everything as '1' or '0'

texts:
Those two pictures are from two similar experiments using same code.
I am fine-tuning a pretrained-Bert model to do a binary text classification task, the dataset is 50% positive vs 50% negative, so the classifier shouldn't classify everything as one class in the validation set as the picture shows.
I used AdamW optimizer with decreased learning rate.
I did the gradient clipping.
When I decrease the learning rate, it works fine, from 5e^-5 to  3e-5 or 2e-5.
What might be the problem here?


texts: Reflective padding as pure keras verision

texts:
I am using keras with the plaidML backend and need to implement reflective padding.
With a tensorflow backend that is simply tf.pad with mode set to REFLECT.

How can implement that functionality with K. functions or plaidml tile functions ?
Or is there an implementation somewhere i could use ?

With K. function simply slicing, inverting and concatenating the value back together would probably be possible, but all my tries in that direction where a mess and didn't really worked out.


texts: Extracting Useful features from large convolutional layers

texts:
I have been training a convolutional neural network on emotion detection. Now, I would like to extract features for my data to train an LSTM layer. In my case, the top convolutional layers in the network has the following dimensions: [None, 4, 4, 512] and [None, 4, 4, 1024]. Therefore, this will give a total of 8192 and 16384 dimensional vectors. This is too large to train an LSTM layer. Therefore, I would like to know what is the best possible way to reduce the dimensionality of this vector? In other words, should I apply global average pooling to the conv layer after obtaining the activation or any other dimensionality reduction technique? In this case, my features will be a vector of 512 or 1024 dimensions, which makes sense.

Any help is much appreciated!!


texts: How can positional encodings including a sine operation be linearly transformable for any offset?

texts:
In the paper "Attention is all you need" the authors add a positional encoding to each token in the sequence (section 3.5). The following encoding is chosen:

$ PE(pos, 2dim) = sin(pos / 10000 ^ {2dim/d_{model}} ) $

$ PE(pos, 2dim+1) = cos(pos / 10000 ^ {2dim/d_{model}} ) $

The text states that "for any fixed offset $k$, $PE(pos+k)$ can be represented as a linear function of $PE(pos)$". This did not seem obvious due to me due to the nonlinearity of the sine function. Other resources like Attention is all you need Explained mention this property but do not go deeper into it.

I decided to experiment with this by attempting to map a number of outputs from the $PE(pos)$ function to outputs with a given offset $k$.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

def PE_even(pos, dim):
    """ This corresponds to the 2dim state. """
    size = 10000
    d_Model = 512
    return np.sin(pos / (size ** (2 * dim / d_Model)))

positions = pd.Series(np.arange(0, 1000))

k = 10
a = positions.apply(lambda p: PE_even(p, 64))
b = positions.apply(lambda p: PE_even(p + k, 64))

X = a.values.reshape(-1, 1)
y = b.values
r = LinearRegression()
r.fit(X, y)
r.score(X, y)

However, no offset nor any range of numbers in these dimensions yields a suitable fit. As $k$ increases and the sine waves resulting from the $PE(pos)$ function get out of sync, the correlation of the transformation and the truth decreases. Even using simple neural networks with and without linearities does not yield a good fit.

Did I misapprehend the statement in the paper, or is my code or understanding of the underlying math here faulty?