texts: How to compute time-lagged correlation between two variables with many examples at each time t?

texts:
I have a dictionary of following form:
datetimes = {year : {name : (score1, score2)}} #there are 50+ names/year

So, essentially, I'm trying to get an aggregate picture of how score1 at year_n is correlated with score2 at year_n.


texts: Is it ok to use negatively correlated features in data modeling? Say I have features A and B that have a correlation coefficient of 0.2 and features C and D with -0.2 correlation coefficient, is it fine to use features C and D in the model, since they have a low negative correlation? Also, does this have different effects on a regression vs. classification problem?


texts: What is the difference between data-driven methods and machine learning?

texts: I was wondering (about a more semantic question), is there a difference between data-driven methods and machine learning? Or is it more correct to state that machine learning is a category of data-driven methods (and what then are other categories)?


texts: How can I go about building a model for large number of outputs?

texts:
I have previously worked on small-scale feedforward neural network problems.

But I have started working on a new project where the goal is to predict air quality in 25 locations throughout the country a day ahead. Now, I am quite well-versed with the air quality side of things.

My question:

In a problem like this, would I develop 25 independent models (which share the same input structure) or one model with 25 outputs.

I guess what I want to do - is there something like parallel neural networks? Or is this 25 different problems? I have mostly worked on physical models where the physics would be shared by all 25 locations. And the inputs would be different.

Would this be a data parallelism or a model parallelism problem?


texts: Retain similarity distances when using an autoencoder for dimensionality reduction

texts:
I am trying to reduce the dimensionality of topic vectors (300, 1) to a two dimensional space. This has been done with various methods (e.g. t-SNE and autoencoders). A published example of reducing topic vectors to 2-dim space can be found here.

To train a test autoencoder, I took the top 10k words from Google's word vector model and tried to encode/decode them.
My autoencoder (300, 100, 2) with tanh activation functions (due to the nature of the vector elements of the word2vec model) seems to learn the vectors quickly (the loss function quickly goes negative).

However, when I compare the closest words based on the 2-dim reduction the results from the autoencoder (cosine and euclidean distance of the word presentations) I notice that they don't match the similar words provided from the original w2v model (output of most_similar).

Most similar words based on the autoencoded representation (2-dim, calculated via cosine distance). Results look random and it seems that the original distances didn't get preserved.

>>> extract_from_embed('car')  # extract method for my own embedding
[(football', 0.91979183998553593), (u'\xa9', 0.9515135906035519), (u'Thank', 0.96150527440098321), (u'innings', 0.96893565858480013), (u'thank', 0.9699300787802696), (u'balls', 0.97004978663463826), (u'Admission', 0.97050270191042776), (u'thanked', 0.97098601610349322), (u'Announces', 0.97186348679591361), (u'drills', 0.97214077185079129)]

Word2Vec most similar words based on 300, 1 vectors

>>> word_vectors.most_similar(positive=['car'])  # std gensim method
[(u'vehicle', 0.7821096777915955),
 (u'cars', 0.7423831224441528),
 (u'SUV', 0.7160963416099548),
 (u'truck', 0.6735789775848389),
 (u'Car', 0.667760968208313),
 ...
 (u'automobile', 0.5838367938995361)]

Is there a way to "preserve" the relative distances between vectors when I used an autoencoder? I understand that the absolute distances differ between vectors from a 2 and 300 dim space. But can I preserve the relative distances between the vectors? Is this even possible?


texts: What is the difference between Hadoop and noSQL

texts:
I heard about many tools / frameworks for helping people to process their data (big data environment).

One is called Hadoop and the other is the noSQL concept. What is the difference in point of processing?

Are they complementary?


texts: ZeroR as performance baseline for binary classfication model?

texts:
It is known that ZeroR model is used predict the majority class in a given data set.
Having said that, is ZeroR a suitable performance baseline provided one has a balanced data set (50/50)?
If not, what would be a good baseline for a Naive Bayes classification model, used for binary classification (positive/negative)?


texts: Use liblinear on big data for semantic analysis

texts:
I use Libsvm to train data and predict classification on semantic analysis problem. But it has a performance issue on large-scale data, because semantic analysis concerns n-dimension problem.

Last year, Liblinear was release, and it can solve performance bottleneck.
But it cost too much memory. Is MapReduce the only way to solve semantic analysis problem on big data? Or are there any other methods that can improve memory bottleneck on Liblinear?


texts: Can AlphaGo Zero adapt to oponents skills/profile?

texts: I read the AlphaGo Zero paper and I didn't found nothing about it in there. But I would like to know if AlphaGo Zero can adapt to the way the oponent plays (oponent profile) or something like this. Thanks!!


texts: Detecting Different Distributions in data

texts:
Supposing I have a dataset that I assume that have instances generated by two different distributions, is it possible to separate these instances based on the underlying generating distribution?

Also, if I relax this hypothesis, and suppose I know one of the distributions beforehand, is this task possible? The second distribution could he chaotic, and also it is impossible to know it.

Finally, is there any result concerning the distance between the two distributions? Is there any theoretical proof of correctness, if there are algorithms for realizing that?

Thank you!


texts: Preprocessing: StandardScaler() Do we really need mean to be zero?

texts:
For instance, many elements used in the objective function of a
learning algorithm (such as the RBF kernel of Support Vector Machines
or the l1 and l2 regularizers of linear models) assume that all
features are centered around zero and have variance in the same order.

This is from the scikit-learn
Can someone please specify what elements they are referring when it is said the mean is expected to be zero.
I understand that the variance should be in similar range for the algorithms to give same significance to each feature. But is there anything also necessarily expecting zero mean in the data?
In other words, if I know variance of each feature is already in same range, can something still go wrong as it says some "elements" expect zero mean.


texts: binary decision tree hyperplane parallel to axis?

texts:
I'm reading the random forest paper.

  binary decision trees use a single feature at each nonterminal node. A test point is assigned to the left or right branch by its value of that feature. Geometrically this corresponds to assigning the point to one side of the hyperplane that is parallel to one axis of the feature space.

Can someone explain why the hyperplane is parallel to the axis?

Say the feature is represented by the $x$ axis, and that the decision is $x < 5$ vs $x >5$. To me it seems like the hyperplane should be vertical to the $x$ axis so that $x = 4$ is on one side and $x=6$ is on the other. What's wrong with my understanding? Thanks.


texts: How is a splitting point chosen for continuous variables in decision trees?

texts:
I have two questions related to decision trees:

If we have a continuous attribute, how do we choose the splitting value?

Example: Age=(20,29,50,40....)
Imagine that we have a continuous attribute $f$ that have values in $R$. How can I  write an algorithm that finds the split point $v$, in order that when we split $f$ by $v$, we have a minimum gain for $f>v$?


texts: How to add 'other' as one group to clustering algorithm inference pipeline

texts:
I have few clustering algorithms tuned having 5 cluster. I want 6th cluster if new data does not belong initial  5 cluster fall in 6th  cluster.
6th cluster [ say other category] consist of all data point which does not belong to 5 cluster.
P.S.:- initial whatever data is give is belong those 5 cluster. so say, kmean algorithms with number cluster as 5. during inference I wanted add 6th cluster so any which does belong given cluster can put this category depending on threshold distance. I have textual data. do let me which clustering algorithms i should go with dbscan, som etc..


texts:
I am looking for a design pattern that is relevant to a module that extracts features.
I want to define a certain number of features over my data points, and then according to the performance and the feature selection, I may want to remove some of them and add others, and also I may want to consider any subsets of them to test.

What is a good design pattern to do that? Did I miss something obvious? I am neither an engineer nor a developer, so I never study such things but I understand that it could help me a lot!

Thanks for any help,


texts: Problem of continuous training - Supervised learning

texts:
I am sure this is a most common problem, but would like to know by experts on how to tackle it. Note that, I mostly deal with textual data (NLP problems).
When a supervised learning model is created, say a text classifier, and it works well on seen data then we deploy the model in production (you can think of a chatbot also).
But in real time, when new type of data comes where the prediction fails, we find that a new word or new pattern is breaking the model. So we go ahead and retrain the model with new encountered data. This is where the continuous learning problem starts.
Can ML/NLP veterans please suggest some alternates to solve this labor work? Following approaches have been tried and the problems also listed:

We simply can't train with new data infinitely. As production systems should be self healing. We cant put the cost of a human admin constantly monitoring the project. Also, it is practically not possible to get huge domain data during model training phase.
Use of advanced embeddings, and SoTA models like BERT.   (Problem: The accuracy of these models is too hard to control)
Synthetic data generation/data augmentatoin.  (Problem : Does not work well in case of NLP problems. Refer: training-with-less-data )
Unsupervised classification  (Problem: Does not work well on closed domain problems, as most unsupervised models are either statistical which give a fair value of accuracy but not decent , or are trained on public domain data)
Reinforcement learning. (Problem: Real world NLP data is not labeled unlike a self driven car where the feedback is instant)


texts: How significant the variables like "id", "region code" etc are in the predictive modelling?

texts: I participated in one of the hackathon. And there the variables were like id, region, gender, age .etc.  It was a regression problem. I did scaling on the variables. But I am not sure what to do with the variables like, id and region code, numerical and ordinal variables respectively. Are they relevant and significant for modeling?


texts: Cross-fold validation done on whole dataset or training set?

texts:
I have a dataset of 77 samples with 302 features with two labels (0,1). I trained an SVM with gridsearch (cv=5) to perform binary classification.

In one run of my script, I do a test-train split, and then pass the training data into gridsearch. I am getting wide range of training accuracy (>70-90%) and test accuracy (40-75%). My question, is how do I evaluate my model if it appears that based on the train/since it seems like my data is getting different results based on the splitting.

What I've done is: run the entire scrip in a loop and iterate over 10 different splits and then plot the error bars on the ROC curve. Is this reasonable?


texts:
I'm working on a project for a class where I'm trying to create an algorithm that learns music and creates its own music.

I'm having trouble on how to set up the data for it to be inputted into the LSTM.

A single training example consists of a chord that is a vector of binary values based on what keys are pressed in MIDI form (indices 0-127), a value that denotes duration of the note, beat strength, numerator of the time signature, and denominator of the time signature, and the key signature represented by the number of flats

So one example might look like

$$\left[ \begin{array}{c} {0} \\ {1} \\{0} \\{1} \\{\vdots} \\ {1} \\ {0} \\ {4} \\ {3} \\ {4} \\ {4} \\ {-2}   \end{array} \right]$$

The result is a 132x1 vector

I was having trouble conceptualizing how to input this data type into an LSTM. Doing a linear output would not make that much sense, but I don't think I can directly one-hot this vector either.


texts: What would be the best way to map similar ngrams

texts:
I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same entity.

What would be the best way to implement this? I've been thinking and so far I've only come up with a brute-force approach checking for each synset of each word and trying to find a similar word or else add them as a new entity.

I am wondering whether there are any better methods of implementing this?

for 1-gram:

from nltk.corpus import wordnet as wn
from nltk.stem import WordNetLemmatizer
l = WordNetLemmatizer()

older = 'older'
elder = 'elder'
older_lemma = l.lemmatize(older, pos=wn.ADJ)
elder_lemma = l.lemmatize(elder, pos=wn.ADJ)

for syn in wn.synsets(older_lemma):
    if elder_lemma in syn.lemma_names():
        print(syn)

Ideally, I would want to extend it to n-grams and I'm searching for a better way to do this.

Edit : I'm not looking for vector based solutions.

What I'm thinking of is some sort of crude but fast similarity algorithm that can give me a crude representation of how close can two words/synsets possibly be. That way I could eliminate most of the absolutely dissimilar words saving time. I'm not sure if it exists


texts: Finding the equation for a multiple and nonlinear regression model?

texts:
Regarding nonlinear and multivariable regression, I am using R or Matlab. In the case where I have a regression with just two variables, I simply draw the graph Y with respect to X and look for the equation of the model that corresponds the most to the distribution of the point cloud.

However, in the case where Y is connected with several independent variables (in my case 12 variables $X_1, X_2,...,X_{12}$). How will I be able to define the equation of the regression model?


texts: Where's a good place to get feedback on my portfolio?

texts: I'm putting together my portfolio for freelance work in data science. I'm curious if there are communities on stackexchange or elsewhere that are helpful resources for receiving feedback.


texts: Any mini-batch implementation of Random Forest?

texts:
I'm using the scikit-learn's Random Forest to perform some classification task but run out of memory because of the amount of data.

Is there any mini-batch implementation of the Random Forest algorithm (or similar decision trees-based method)?


texts: Handling Numerical Categorical Column in ML models in Python

texts:
When I was exploring the titanic dataset to estimate the probability of a person of surviving using the Logistic Model, I realized there are two ways of handling numerical categorical variables :

Use them as is in the dataset ie don't convert them into dummy variables. See the PClass below:

Convert the numerical categorical variable into dummy variables as below:

I had assumed both these would bear the same result but the model results are different using both these approaches. So, I wanted to understand the logic working behind both these approaches and which one should be preferred?


texts: Clustering geo location coordinates (lat,long pairs)

texts:
What is the right approach and clustering algorithm for geolocation clustering?

I'm using the following code to cluster geolocation coordinates:

import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.vq import kmeans2, whiten

coordinates= np.array([
           [lat, long],
           [lat, long],
            ...
           [lat, long]
           ])
x, y = kmeans2(whiten(coordinates), 3, iter = 20)
plt.scatter(coordinates[:,0], coordinates[:,1], c=y);
plt.show()

Is it right to use K-means for geolocation clustering, as it uses Euclidean distance, and not Haversine formula as a distance function?


texts: fast processing: changing the column value in a geospatial environment

texts:
I have a table whose header looks like this: complaint_type, borough, street_name, incident_zip, latitude, longitude

1) I want to check if the "incident_zip" column of each row is in a specific list of zip codes and change the "borough" accordingly. There is a large amount of data and i cannot find any better code to do this. I am using python 3.6.
I want to change the borough where it is "unspecified". I used if statements along with replace but it is taking a lot of processing time (more than 3hrs) that I have to stop the kernel. There are five long lists of zip codes.

2) Is there any other way to update the "borough" column like latitude and longitude?


texts:
Question 1:

In the example of logistic regression, I often see the regularization constant and penalty methods being tuned by a grid search. However, it seems like there are a lot more options for tuning:

classifier_os.get_params()
gives:
{'C': 1.0,
 'class_weight': None,
 'dual': False,
 'fit_intercept': True,
...
and many more!

So my question is: Are these other parameters typically not worth tuning, or are they left out in examples for another reason?
For example, I changed to solver='liblinear' and got sub-par results compared to lbfgs. Is that a coincidence, or are the other parameters already at their best by default?

Question 2:

Does performing grid search on hyper-parameters guarantee improved performance when tested on the same data set?

I ask because my intuition was "yes", however I got slightly lower scores after tuning my regularization constant:

p_grid = {
    'C': np.logspace(-3, 3, 7)
}
clf = model_selection.GridSearchCV(classifier_os, p_grid, scoring='accuracy')
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)
metrics.classification_report(y_pred, y_test, output_dict=True)

Gives me the following scores (C=10):

accuracy :  0.8218181818181818
 macro avg:
     precision :  0.8210875331564986
     recall :  0.8213603058298822
     f1-score :  0.8212129655428624
     support :  275

As compared to before tuning (C=1):

accuracy :  0.8290909090909091
 macro avg:
     precision :  0.8287798408488063
     recall :  0.8285358354537744
     f1-score :  0.8286468069310212

The only thing that the tuning changed was to make the regularization constant C=10 instead of the default C=1


texts: Feature selection for tracking user activity within an application

texts:
I am developing a system that is intended to capture the "context" of user activity within an application; it is a framework that web applications can use to tag user activity based on requests made to the system.  It is hoped that this data can then power ML features such as context aware information retrieval.

I'm having trouble deciding on what features to select in addition to these user tags - the URL being requested, approximate time spent with any given resource, estimating the current "activity" within the system.

I am interested to know if there are good examples of this kind of technology or any prior research on the subject - a cursory search of the ACM DL revealed some related papers but nothing really spot-on.


texts: Is this random forest logical correct and correct implemented with R and gbm?

texts:
For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors.

I got a data set with 15 million entries and want to make a regression for a numerical target (time). The data structure is:

I have 7 categorical variables, 1 date and 4 numerical features. After data preparation I split the data into training and test data set.

Than I defined a gradient boosting machine model and search for the right parameters by try and error, research and more try and error. Is this approach correct as far?

#train and test are the prepared data frames
#RMSE is an implementation of the root mean squared error function
#gradient boosting machine training model
gbmTraining <- gbm(train$target~., data=train, distribution = "gaussian", verbose = TRUE,
                 n.trees = 500, #number of trees
                 cv.folds=5, #cross validation
                 n.cores=2, #number of processor cores
                 shrinkage = .03,       #learning rate
                 interaction.depth = 42, #number of notes
                 n.minobsinnode = 15)   #depth of tree
print(gbmTraining)
png(file = "Results/RelativeInfluence.png")
summary.gbm(gbmTraining, plot = TRUE)
dev.off()

print("best number of trees: ")
png(file = "Results/Convergence.png")
best_iter <- gbm.perf(gbmTraining, method="cv", plot.it=TRUE)
dev.off()
print(best_iter)

#make train prediction
fitTrain <- predict(gbmTraining, train, best_iter, type = "response", verbose = TRUE)

#training error
errorTrain <- RMSE(fitTrain, test$target)
errorTrain <- round(errorTrain, digits = 1)
errorDiffTrain <- fitTrain - test$target
print("meanTrainError(predicted, actual):")
print(mean(errorDiffTrain))

#write and save results
print("summary training")
print(head(data.frame("Actual" = train$target, "Predicted" = fitTrain)))
print(summary(data.frame("Actual" = train$target, "Predicted" = fitTrain)))
trainTable <- data.frame("Actual" = train$target, "Predicted" = fitTrain)

write.csv(trainTable, "Results/trainTable.csv", row.names = FALSE)

Here I want to know if I use the correct arguments if I calculate the errors. Do I calculate the errors between the fitted data and data from the test data set or do I use here the data from the training data set?

If I found good parameters and want to go to validation the implementation becomes shorter because I use the found parameters. I create a new model for the tests:

#gbm testing model
gbmTesting = gbm(test$target~., data=test, distribution = "gaussian", n.trees = 500,
                 n.cores=2, verbose = FALSE,
                 shrinkage = .03,
                 interaction.depth = 42,
                 n.minobsinnode = 15)

#make test prediction
fitTest <- predict(gbmTesting, data=test, n.trees = 500, type = "response", verbose = FALSE)

#test error
errorTest <- RMSE(fitTest, test$target)
errorTest <- round(errorTest, digits = 1)
errorDiffTest <- fitTest - test$target
print("meanTestError(predicted, actual):")
print(mean(errorDiffTest))

print("summary testing")
print(head(data.frame("Actual" = test$target, "Predicted" = fitTest)))
print(summary(data.frame("Actual" = test$target, "Predicted" = fitTest)))
testTable <- data.frame("Actual" = test$target, "Predicted" = fitTest)

write.csv(testTable, "Results/testTable.csv", row.names = FALSE)

Is this correct or do I fall into a mental tripping hazard?

p.s.
the stack exchange community is so huge I am not sure if this has to be in some of the other sites? If yes, which one?


texts: Class imbalance in one hot encoding for CNN

texts:
I am building a 2D Convolutional Neural Network for MFCC features for audio classification.

The issue I am facing is that there are 2 classes and huge imbalance between them. One class has 17687 samples while other has 67737 samples. I have done one hot encoding so I have two CNN outputs as [1,0] and [0,1].

From my research it seems that adding class_weights in model.fit only seems to work for  binary classification problems. Is there any way I can assign weights to one hot encoded results?


texts: Discrete Ordinal Classification with Probabilities

texts: If I have classes 1, 2, 3 and 4. But, I also need the probability for each of the other classes. I'm currently using XGBoost for one-vs-rest classification, but that means we're losing information since the classes are ordinal.


texts: Which are the strategies to counter the 80/20 dilema in Data Science projects?

texts:
Most of the time in Data Science projects is not spent in (performing) actual analytics but rather in other tasks, such as organizing data sources, collecting samples and preparing datasets, compiling and validating business rules in data, etc.This fact has been studied as the 80/20 dilemma in Data Science projects

In order to tackle this dilema, I would like to ask which are the strategies used to decrease the 80% of time spent in the other stages (organizing data sources, collecting samples and preparing datasets, compiling and validating business rules in data)


texts:
In the book Deep Learning and Neural Networks by Michael Nielsen, he used a naive approach to write the code which is a bigger considered to a code written in tensorflow.

I want to know which code is better and how it is better ?


texts: AI that maximizes the storage of rectangular parallelepipeds in a bigger parallelepiped

texts:
As you can see in the title, I'm trying to program an AI in Java that would help someone optimize his storage.

The user has to enter the size of his storage space (a box, a room, a warehouse etc...) and then enter the size of the items he has to store in this space. (note that everything must be a rectangular parallelepiped) And the AI should find the best position for each item such that the space is optimized.

Here is a list of what I started to do :

I asked the user to enter the size of the storage space (units are trivial here except for the computing cost of the AI later on I'm guessing), telling him that the values will be rounded down to the unit
I started by creating a 3-dimensional array of integers representing the storage space's volume, using the 3 values taken earlier. Filling it with 0s, where 0s would later represent free space and 1s occupied space.
Then, store in another multidimensional array the sizes of the items he has to store
And that's where the AI part should be starting. First thing the AI should do is check whether the addition of all the items' volumes doesn't surpass the storage space's volume. But then there are so many things to do and so many possibilities that I get lost in my thoughts and don't know where to start...

In conclusion, can anyone give me the proper terms of this problem in AI literature, as well as a link to an existing work of this kind ?
Thanks


texts: Google's election results data visualization

texts:
Can someone explain the data visualization of Google's live election results to me?

My understanding is that the states they've "called" have a check mark that is blue if a Democrat (Clinton) victory or red if a Republican (Trump) victory. The results additionally show the number of votes report thusfar as a count and as a percent. The states they consider too early to call based on the % of precincts reporting do not have a check mark.

However, in the case of Washington it has been "called" as a Democrat (Clinton) victory, but shows a higher count and percent for Trump (?).

View 1:

View 2:

Note: There were some smaller third party candidates cut off in the screenshot above, like 2 Socialist candidates.


texts:
For the XOR problem, 2 decision boundaries are needed to solve it using 2 inputs neurons, 2 hidden neurons, 1 output neuron. From the book "Neural Network Design" on page 84, the author says that "each neuron in the network divides the input space into two regions."

In the xor network, there are 3 neurons excluding the inputs. Therefore 3 decision boundaries should be drawn. But most books show only 2 decision boundaries for the xor problem. Can someone explain why is it so?


texts: How to measure accuracy of a route prediction

texts:
I developed a new route prediction algorithm and I am trying to find a metric that informs on how well a prediction was.

This metric is meant to be used offline, meaning that the goal is not to measure the quality of the prediction when it is need in real time. Instead, We are given a set $R=\{r_1,r_2,...r_{|R|}\}$ of routes that occurred in the past and for each $r_i\in R$ we take a small prefix of $r_i$ and provide it as input to the algorithm which in turn outputs a prediction route $p_i$.

Therefore, given the set $R=\{r_1,r_2,...r_{|R|}\}$ and the corresponding predictions set $P=\{p_1,p_2,...p_{|P|}\}$, I want to compare each pair of routes $(p_i,r_i)$ to determine how different $p_i$ is from $r_i$.

Can anyone point me in the right direction?

My first idea was to compute the area comprised between the two routes $p_i$ and $r_i$, but I really do not know how to interpret the output in terms of good or bad result.

Thanks for the attention


texts: What is the difference between Salient and Generic Object Detection?

texts:
There are two types of Detection methods in general. Generic and Salient.
Question: What is the exact difference in between the two?
Broad Difference:


texts: Is there an intuitive explanation why some neural networks have more than one fully connected layers?

texts:
I have searched online, but am still not satisfied with answers like this and this.

My intuition is that fully connected layers are completely linear. That means no matter how many FC layers are used, the expressiveness is always limited to linear combinations of previous layer. But mathematically, one FC layer should already be able to learn the weights to produce the exactly the same behavior. Then why do we need more? Did I miss something here?


texts: How do I use recency/time as a feature after I've a word2vec model

texts:
I have a word 2 vector model which works great but let's say I typed Eminem.

And it gives me:

  rap, music, revival and recovery.

How do I make another layer that shows revival first?


texts: Can I retrain my keras model on small dataset with some empty folders

texts:
I am working on a vehicle classification problem.

I trained my model with 605 labels(folders) and 300k photos.

After testing my trained model, I checked my wrong answers' training and validation photos. I realized some of my labels training or validation photos are wrong or not enough different from another.

I can prepare a small dataset that only contains these wrong learnt labels photos and keep other labels folders empty.

If i retrain my model, will I get better accuracy? Or should I train from scratch?


texts: Is PCA (by eigendecomposition) or SVD better in decorrelating the predictors of a machine learning model?

texts: Is there any reason to think that SVD is better than PCA (by eigendecomposition) in decorrelating the predictors of a machine learning model?


texts: Is the Apriori algorithm suitable for database tuples?

texts:
Problem description:

I need to use an association rule algorithm that lets me use database tuples and I think Apriori is a good option, but I am not sure.

Starting point:

I know the Apriori algorithm works in itemsets provided the items are independent one from each other, let's say:

T1: {Onions, Beer, Diapers}
T2: {Chili, Onions, Pizza}
T3: {Beer, Chili, Onions}
...

This would imply building a table of the form:

    -------------------//
    |  Onions |  Beer
|----------------------//
|T1 |    1    |   1
|T2 |    1    |   0
|----------------------//

Being the rows the transactions, the columns the items and a 0 or a 1 in the cell if the item is in the transaction or not.

Problem details:

I am using a beer database in which I have tuples of different kinds of beer, for example:

 id       name             type      country       liters   volume   container
------------------------------------------------------------------------------
 xxx  The Jolly Joker    Pale Ale   Netherlands    Regular   Mild      Can

...being the upper row the column names and the lower one a sample tuple.

I want to find, given a set of tuples - like a shopping cart of 7 beers -, find rules of association that can fit that cart, as it follows:

{Pale Ale, Netherlands} -> {Bottle}

I am doubting if the Apriori algorithm can mine anything on this kind of data structure: one possibility would be treating every single value (regardless the column name) as an item of an itemset and after running the algorithm, ignoring the senseless sets (for example, {Netherlands, Germany, Blonde}).

I don't know if that approach would compromise the algorithm's reliability, if there exists a better way of getting the rules I need or if I am just using the wrong algorithm and there's one I don't know that fits my specifications.


texts: How to implement LSTM using Doc2Vec vectors?

texts:
I would like to build a ANN for text classification, which has an LSTM layer, and using weights obtained via a Doc2Vec model trained before:

model_doc2vec = Sequential()
model_doc2vec.add(Embedding(voacabulary_dim, 100, input_length=longest_document, weights=[training_weights], trainable=False))
model_doc2vec.add(LSTM(units=10, dropout=0.25, recurrent_dropout=0.25, return_sequences=True))
model_doc2vec.add(Flatten())
model_doc2vec.add(Dense(3, activation='softmax'))
model_doc2vec.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

At the moment I am not able to get the weights in the Embedding() layer mentioned above. I would like to know which is the easiest way to get these weights.


texts:
A recent paper by He et al. (Deep Residual Learning for Image Recognition, Microsoft Research, 2015) claims that they use up to 4096 layers (not neurons!).

I am trying to understand the paper, but I stumble about the word "residual".

Could somebody please give me an explanation / definition what residual means in this case?

Examples

  We explicitly reformulate the layers as learning
  residual functions with reference to the layer inputs, instead
  of learning unreferenced functions.

  [...]

  Instead
  of hoping each few stacked layers directly fit a
  desired underlying mapping, we explicitly let these layers
  fit a residual mapping. Formally, denoting the desired
  underlying mapping as $\mathcal{H}(x)$, we let the stacked nonlinear
  layers fit another mapping of $\mathcal{F}(x) := \mathcal{H}(x)−x$. The original
  mapping is recast into $\mathcal{F}(x)+x$. We hypothesize that it
  is easier to optimize the residual mapping than to optimize
  the original, unreferenced mapping


texts: Need suggestions on customer segmentation

texts:
I have been tasked with performing customer segmentation for a Business to business use case based on customer purchase history. Can experts provide me inputs on how do I proceed with customer segmentation based on the following dataset
Dataset details which have been provided to me

Hierarchy 3,4,5 define the categories under which the product falls

Edit:
Also need inputs on how do i select features for my clustering algorithm?


texts: What is a YAML file and where is it used in a machine learning context?

texts: I am not entirely sure if this is on-topic here, so please let me know if it is not. I keep seeing the idea of YAML files pop up while reading machine learning literature. My question is, what exactly is a YAML file, and how does it relate to machine learning and data science projects?


texts: How to make an effective sampling from a database of text documents?

texts:
Problem: I want to know methods to perform an effective sampling from a database.
The size of the database is about 250K text documents and in this case each text document is related to some majors (Electrical Engineering, Medicine and so on). So far I have seen some simple techniques like Simple random sample and Stratified sampling; however, I don't think it's a good idea to apply them for the following reasons:

In the case of simple random sample, for instance, there are a few documents in the database that talk about majors like Naval Engineering or Arts. Therefore I think they are less likely to be sampled with this method but I would like to have some samples of every major as possible.
In the case of stratified sampling, most of the documents talk about more than one major so I cannot divide the database in subgroups because they will not be mutually exclusive, at least for the case in which each major is a subgroup.

Finally, I cannot use the whole database due to expensive computational cost processing. So I would really appreciate any suggestions on other sampling methods. Thanks for any help in advance.


texts: How to best read large dataset from disk

texts:
I want to solve a task using a ResNet in keras and tensorflow. My Dataset is big, and right now I'm considering my data loading options and trying to determine which one suits the task best.
About the Dataset:

x: arrays of 200x700 cells in range -1.0...1.0, I don't want to downsample them; they are currently saved as matlab or npz file
y: the label consists of two floats per x.
I have 1.2Million of these (x, y) which are currently saved in 1000 npz files, each with 1GB, totalling to 1TB of data.

Problem: I don't have 1TB RAM in my system, so I can't keep all the data in memory. Thus I need a suitable solution to read my data from disk while training my neural network.
Solutions that I found so far:

save these files as images and use keras dataset io "load_images_from_directory", downside: I need to save the images on disk which would probably take even more than 1TB. And what about the labels? Plus probably additional preprocessing to from range 0..1 to -1..1
tfrecords which feels like an overkill, since my dataset is not really a structured one but it's just (array, label)
hdf files which is also more for structured/hierarchical data.

Things that I also want to take into account:

Do I save my data as is, or do I need to save shuffled batches? But according to this I should also shuffle the mini-batches in each epoch new. This would mean that the order and filesizes (e.g. one file is one mini-batch) whith which I save my files is not important - the mini-batches should be shuffled anyways.
Later, I will most likely also need to transfer the whole project to pytorch, so a data storage which is supported by both (kears/tensorflow and pytorch) can save me some time later.
If I store each (x,y) sample as one small .bin file, this file is smaller than the block size of my disk, thus using more disk size than necessary.

So the question is: What are the pros and cons specific to my dataset/task, thus which dataloading should I use? Are there more options that I haven't discovered yet?


texts: A Question: is it possible for me to create an AI that learns to play games which I did not create?

texts:
Mainly, I have a question that I could not find an answer for anywhere, about an AI machine learning game.
Is it possible for me to create an AI which can learn a game that I did not myself create, without any source codes or APIs? And I'm talking about simple games such as chrome t-rex, snake, etc. I was told this is possible using image processing but I am still not so sure. I am referring to just open up a game, not a downloaded one, a one available at google, such as the mentioned above, run my code, and it will start playing and learning the game. Just creating 1 algorithm to learn multiple easy 2D google games, is such a thing possible with image processing?


texts: How can I have a sequential model inside Keras' functional model?

texts: I'm trying to build a architecture that takes in multiple inputs and has a smaller neural network inside a larger architecture. For me to keep track of of the smaller netowrk's loss, I need to compile the sub-network as a Sequential, but it still needs to be part of the functional model as a whole for the architecture. Is this possible in Keras?


texts: How to interpret the rec curve for a regression task?

texts:
I am using forest fire dataset and applied neural network model. I tried to generate REC curve, this is how it looks like. Pretty weird!!!

I have also applied XGBoost but the REC curve is almost parallel to X-axis.
I don't understand how to make sense out of it?
I also want to understand how to interpret REC curve in general?


texts: Do i need to handle missing values before EDA?

texts: I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to change the data). Can I work with the dataframe with a column with missing values during exploratory data analysis and only take the slides with no values missing when plotting something with this specific column?


texts: How are new neural network architectures 'discovered'

texts:
This might not be on topic here, but fingers crossed.

There are all manner of neural network architectures out there, everything from convolutional networks to deep recurrent networks (and even deep recurrent convnets). Some network arcitecture development I can kind of see the intuition behind; recurrent networks allow for feedback loop type things to be taken into account. Other ideas seem to come out of left field and change the field. Yolo is a good example of this - I understand how it works, I've implimented it on my own dataset. But how did someone come to the set up that allowed yolo to work.

Is it just creativity, trying random architectures and new ideas to see what sticks? Is there some methodology that I could apply to take some steps towards a new ground-breaking architecture? (I understand that I won't actually make the next new architecture, but it's interesting to understand the process)


texts: What does it mean to have 1 degree of freedom in ANOVA test?

texts:
So I used python to run multi-factorial ANOVA analysis on a data set. I first used a ols.fit() and then the anova_lm function. I realized for the variables I am analyzing their degree of freedom is 1. Does that mean only 1 value out of my data is extracted and used for calculation? Why is the residual df so high?

import pandas as pd
from statsmodels.multivariate.manova import MANOVA
import statsmodels.api as sm
from statsmodels.formula.api import ols
from statsmodels.stats.anova import anova_lm
from statsmodels.graphics.factorplots import interaction_plot
import matplotlib.pyplot as plt
from scipy import stats

#Some quick df transformation
#We want to analyze the candy sales numbers with respect to the flavors

formula = 'Sales ~ Mango+Raspberry+Chocolate+Dark_Chocolate+Ice_cream+Cherry'

model = ols(formula, df).fit()

aov_table = anova_lm(model_MS_ro, typ=2)

print(aov_table)

****ANOVA Results****

                      df    sum_sq   mean_sq          F    PR(>F)
Mango                1.0  0.008512  0.008512   2.325284  0.130999
Raspberry            1.0  0.006025  0.006025   1.645954  0.202998
Chocolate            1.0  0.049506  0.049506  13.524418  0.000412
Dark_Chocolate       1.0  0.007233  0.007233   1.976095  0.163447
Ice_cream            1.0  0.018032  0.018032   4.926093  0.029117
Cherry               1.0  0.024460  0.024460   6.682116  0.011444
Residual            85.0  0.311140  0.003660        NaN       NaN


texts:
I have just started Python and working on training models. The task that I have been assigned is to train a dataset named "austin_Weather"
Original Dataset
y attribute
After having done some manipulations (following the article), these are my new datasets for training and testing,
X Train
y Train
Now, I tried to apply Naive-Baysen algorithm on these two datasets, but I got this following error
ValueError: y should be a 1d array, got an array of shape (1045, 5) instead.

So, I tried searching it up online, and I found one solution that I should reshape these arrays, so I tried the following code,
X_train = X_train.reshape(-1, 1)
y_train = y_train.reshape(-1, 1)
X_test = X_test.reshape(-1, 1)
y_test = y_test.reshape(-1, 1)

But I got this error on this,
AttributeError: 'DataFrame' object has no attribute 'reshape'

I tried to fix it up by searching it up online, but I couldn't understand most of the things due to my lack of knowledge in Python syntax. If anyone can help resolve this and tell me what exactly is the problem here, I would be very thankful.


texts: AUC on ROC Curve near 1.0 for Multi-Class CNN but Precision/Recall are not perfect?

texts:
I am building a ROC Curve and calculating AUC for multi-class classification on the CIFAR-10 dataset using a CNN. My overall Accuracy is ~ 90% and my precision and recall are as follows:
              precision    recall  f1-score   support

    airplane       0.93      0.90      0.91      1000
  automobile       0.93      0.96      0.95      1000
        bird       0.88      0.87      0.87      1000
         cat       0.86      0.72      0.79      1000
        deer       0.88      0.91      0.89      1000
         dog       0.88      0.81      0.84      1000
        frog       0.83      0.97      0.89      1000
       horse       0.94      0.94      0.94      1000
        ship       0.95      0.93      0.94      1000
       truck       0.90      0.95      0.92      1000

    accuracy                           0.90     10000
   macro avg       0.90      0.90      0.90     10000
weighted avg       0.90      0.90      0.90     10000

The code where I calculate the ROC Curve and AUC is below:
def assess_model_from_pb(model_file_path: Path, xtest: np.ndarray, ytest: np.ndarray, save_plot_path: Path):

    class_labels = ['airplane','automobile','bird','cat','deer','dog','frog','horse','ship','truck']
    model = load_model(model_file_path) # load model from filepath
    feature_extractor = Model(inputs = model.inputs, outputs = model.get_layer('dense').output) # extract dense output layer (will be softmax probabilities)
    y_score = feature_extractor.predict(xtest, batch_size = 64) # one hot encoded softmax predictions
    ytest_binary = label_binarize(ytest, classes = [0,1,2,3,4,5,6,7,8,9]) # one hot encode the test data true labels
    n_classes = y_score.shape[2]

    fpr = dict()
    tpr = dict()
    roc_auc = dict()
    # compute fpr and tpr with roc_curve from the ytest true labels to the scores
    for i in range(n_classes):
        fpr[i], tpr[i], _ = roc_curve(ytest_binary[:, i], y_score[:, i])
        roc_auc[i] = auc(fpr[i], tpr[i])

    # plot each class  curve on single graph for multi-class one vs all classification
    colors = cycle(['blue', 'red', 'green', 'brown', 'purple', 'pink', 'orange', 'black', 'yellow', 'cyan'])
    for i, color, lbl in zip(range(n_classes), colors, class_labels):
        plt.plot(fpr[i], tpr[i], color = color, lw = 1.5,
        label = 'ROC Curve of class {0} (area = {1:0.3f})'.format(lbl, roc_auc[i]))
    plt.plot([0, 1], [0, 1], 'k--', lw = 1.5)
    plt.xlim([-0.05, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('ROC Curve for CIFAR-10 Multi-Class Data')
    plt.legend(loc = 'lower right', prop = {'size': 6})
    fullpath = save_plot_path.joinpath(save_plot_path.stem +'_roc_curve.png')
    plt.savefig(fullpath)
    plt.show()

I suppose I am just confused on how my AUC can be near 1 when my precision and recalls are not perfect. I understand that many thresholds are used to determine what is a positive class and a negative class. For example towards the beginning of the curve, if the threshold is very high (say around .99999), how is it that my tpr is near 1? Is it solely the fact that at that threshold I will only be giving positive classifications for the most absolutely high softmax probabilities?
Would just like a bit more explanation or intuition on the topic to make sure I am not doing something incorrectly.


texts:
Amazon's Alexa, Nuance's Mix and Facebook's Wit.ai all use a similar system to specify how to convert a text command into an intent - i.e. something a computer would understand. I'm not sure what the "official" name for this is but I call it "intent recognition". Basically a way to go from "please set my lights to 50% brightness" to lights.setBrightness(0.50).

The way they are specified is by having the developer provide a list of "sample utterances" which are associated with an intent, and optionally tagged with locations of "entities" (basically parameters). Here's an example from Wit.ai:

My question is: how do these systems work? Since they are all very similar I assume there is some seminal work that they all use. Does anyone know what it is?

Interestingly Houndify uses a different system that is more like regexes: ["please"] . ("activate" | "enable" | "switch on" | "turn on") . [("the" | "my")] . ("lights" | "lighting") . ["please"]. I assume that is integrated into the beam search of their voice recognition system, whereas Alexa, Wit.ai and Mix seem to have separate Speech->Text and Text->Intent systems.

Edit: I found a starting point - A Mechanism for Human - Robot Interaction through Informal Voice Commands. It uses something called Latent Semantic Analysis to compare utterances. I'm going to read up on that. At least it has given me a starting point in the citation network.

Edit 2: LSA is essentially comparing the words used (Bag of Words) in each paragraph of text. I don't see how it can work very well for this case as it totally loses the word order. Although maybe word order doesn't matter much for these kinds of commands.

Edit 3: Hidden Topic Markov Models look like they might be interesting.


texts: Intuition behind the RNN/LSTM hidden state?

texts: What's the intuition behind the hidden states of RNN/LSTM? Are they similar to the hidden states of HMM (Hidden Markov Model)?


texts: Image Classification using Single Class Dataset using Transfer Learning

texts:
I only have around 1000 images of vehicle. I need to train a model that can identify if the image is vehicle or not-vehicle. I do not have a dataset for not-vehicle, as it could be anything besides vehicle.

I guess the best method for this would be to apply transfer learning. I am trying to train data on a pre-trained VGG19 Model. But still, I am unaware on how to train a model with just vehicle images without any non-vehicle images. I am not being able to classify it.

I am new to ML Overall, Any solution based on practical implementation will be highly appreciated.


texts:
I know there are similar question on stats.SE, but I didn't find one that fulfills my request; please, before mark the question as a duplicate, ping me in the comment.

I run a neural network based on neuralnet to forecast SP500 index time series and I want to understand how I can interpret the plot posted below:

Particularly, I'm interested to understand what is the interpretation of the hidden layer weight and the input weight; could someone explain me how to interpret that number, please?

Any hint will be appreciated.


texts: Peformance evaluation of ranking algorithms

texts:
I have three questions:

How can we assess (or measure) the performance of the ranking algorithms?
Are there any specific measures, or performance metrics, for this?
More specifically, how can we compare the performance of AHP-based ranking and probabilistic ranking algorithms?


texts: How to specify output_shape parameter in Lambda layer in Keras

texts:
I don't understand how to specify the output_shape parameter in the Lambda layer in Keras/Tensorflow. The documentation says:

output_shape: Expected output shape from function. This argument can
be
inferred if not explicitly provided. Can be a tuple or function. If a
tuple, it only specifies the first dimension onward;
sample dimension is assumed either the same as the input: output_shape = (input_shape[0], ) + output_shape or, the input is None and
the sample dimension is also None: output_shape = (None, ) + output_shape

If we use a tuple how should I interpret these two expressions?
output_shape = (input_shape[0], ) + output_shape
and
output_shape = (None, ) + output_shape


texts: Merging dataframes in Pandas is taking a surprisingly long time

texts:
I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could have 45,000 rows and 20 columns, another has 100 rows and 900 columns), but they all have common columns of "SubjectID" and "Date", which I'm using to merge the dataframes. I read the dataframes into the list from CSV files, where I know the datatypes for each column.

However, when I try to merge even 10 of the dataframes, it takes around 7 hours, and for all 100, my kernel crashes. Reading all the files in takes about a minute. These aren't files that I would consider "big data" or even large files, like in the posts here, here, or here, so I'm surprised it's taking so long to merge them all. For all 100 dataframes, I'd expect the final dataframe size to be about 1.5 GB, but even that I think Pandas could handle. My code to merge the dataframes is

merged_df = reduce(lambda l,r: l.merge(r, on=['SubjectID', 'Date'], how='outer', suffixes=['_COPYL', '_COPYR']), df_list)

because I want columns to be added on at if they don't already exist for a subjectID and date (I take care of duplicates later).

EDIT

Some sample dataframes are produced by the following code

import numpy as np
import pandas as pd
from functools import reduce

df1 = pd.DataFrame({'SubjectID': ['A', 'A', 'A', 'B', 'B', 'C', 'A'], 'Date': ['2010-03-14', '2010-03-15', '2010-03-16', '2010-03-14','2010-05-15', '2010-03-14', '2010-03-14'], 'Var1': [np.nan, 1, 4, 7, 90, np.nan, 9], 'Var2': [np.nan, 0, 1, 1, 0, np.nan, 1]})
df2 = pd.DataFrame({'SubjectID': ['A', 'A', 'B', 'B', 'C', 'A'], 'Date': ['2010-03-14', '2010-03-15', '2010-03-14', '2010-05-15', '2010-03-14', '2010-03-14'], 'Var2': [ np.nan, 0, 1, 1, np.nan, 1], 'Var3': [0, 0, 1, np.nan, 0, 1]})

df3 = pd.DataFrame({'SubjectID': ['A', 'A', 'A', 'B', 'B', 'C'], 'Date':['2010-03-14', '2010-03-15', '2010-03-16', '2010-03-14', '2010-05-15', '2010-03-14'], 'Var3': [np.nan, 1, 0, np.nan, 0, 1]})

df1['Date'] = pd.to_datetime(df1['Date'])
df2['Date'] = pd.to_datetime(df2['Date'])
df3['Date'] = pd.to_datetime(df3['Date'])

My code to merge the dataframes is

df_list = [df1, df2, df3]
merged_df = reduce(lambda l,r: l.merge(r, on=['SubjectID', 'Date'], how='outer', suffixes=['_COPYL', '_COPYR']), df_list)
merged_df = merged_df.groupby([x.split('_COPY')[0] for x in merged_df.columns], 1).apply(lambda x: x.mode(1)[0])
merged_df['Date'] = pd.to_datetime(merged_df['Date'])

My expected output is

            Date    SubjectID   Var1    Var2    Var3
0     2010-03-14           A    NaN      NaN    0.0
1     2010-03-14           A    NaN      1.0    1.0
2     2010-03-14           A    9.0      1.0    0.0
3     2010-03-14           A    9.0      1.0    1.0
4     2010-03-15           A    1.0      0.0    0.0
5     2010-03-16           A    4.0      1.0    0.0
6     2010-03-14           B    7.0      1.0    1.0
7     2010-05-15           B    90.0     0.0    0.0
8     2010-03-14           C    NaN      NaN    0.0

So far, I've tried reducing the memory of each column datatype by downcasting, different ways to merge in Pandas and CSV files, converting to a sparse dataframe (my data does have a fair amount of NaN), using Dask, and I'm at the point where I'm considering creating a relational database using sqllite in Python. My versions are python-64 3.7.0, pandas 0.23.4, Ipython 6.5.0, using a Mac OSX Darwin with 32 GB memory. I would think my computer should have no trouble doing this, but it does. Is there really no better way or am I doing something wrong that I'm missing?


texts:
Sorry for the newbie question. I have 40 labels/subjects 20 diseased and 20 normals. All the 40 patients have 3d images. I want to run a classification task using 3D-CNN with multiple 3D MRI images. My question is: Is it possible to have multiple features(7 3d Images) per subject as the input. Can anyone help with an example? Or point me in the right direction? Thank you so much for your help.
Kevin


texts:
have a question about the type of model which I should use for a dataset I have.

I have use 2 data-sets for my project. After hypothesis testing, I me

Out of the 7 input variables, 6 of them are categorical and 1 is a date column. Now I have encoded the categorical columns using label encoding and converted them into numerical values. Now I’ve used a simple linear regression model on this dataset and achieved a normalized RMSE value of 0.11.

If I want to improve this accuracy, how do I go about doing it? How can I derive upon the kind of models that I can use considering the data set I have?

The data is mostly about forecast revenues for each product group. That's why I have those categorical columns which represent the hierarchy of the product groups.


texts: Weighted sum of word vectors for document similarity

texts:
I have trained a word2vec model on a corpus of documents. I then compute the term frequency (the same Tf in TfIDF) of each word in each document, multiply each words Tf by its corresponding word vector (this is the weighted part), and sum each of these weighted vectors together element wise to obtain a single vector for a document.

Is this method valid?

An example, to make it clearer. Take the document:

"The car drives on the road"

The Tf for each word would be {'the':0.3333, 'car':0.1666, 'drives':0.1666, 'on':0.1666, 'road':0.16666} (obtained by taking the word count of a word and dividing by the total number of words in the document). If we have a trained word2vec model, we can do the following

$$0.333*\begin{bmatrix}
    the_0 \\
    the_1 \\
    \vdots \\
    the_n
\end{bmatrix}
+ 0.1666*\begin{bmatrix}
    car_0 \\
    car_1 \\
    \vdots \\
    car_n
\end{bmatrix}
+ ...$$

where each of the column vectors is the word vector for that word. The final result is an $n\times 1$ dimensional vector representing the document.

$$
\begin{bmatrix}
    0.333*the_0 + 0.1666*car_0 + \dots \\
    0.333*the_1 + 0.1666*car_1 + \dots \\
    \vdots \\
    0.333*the_n + 0.1666*car_n + \dots
\end{bmatrix}
$$

I appreciate there are other methods such as doc2vec that aim to do very much the same, but in a much more sophisticated way. But is my method valid / is there anything blatantly wrong here?

I have tested this method and even used some document similarity metrics (yes, after normalizing the vectors of course), and yielded some good results for my industrial application. But I want to know more about the academics of this method.

The nice thing about this way, is that by using the word2vec vectors, similarity queries between documents yield very good results due to semantic similarities (euclidean closeness) between word vectors, even if different words are used across documents; this is something TfIDF cannot do as each word is treated differently.

Thanks in advance!


texts: Feeding 3 consecutive video frames to a CNN to track a tennis ball

texts:
I want to use CNN transfer learning to track a tennis ball from TV broadcasts of tennis matches.
I used VGG annotating tool annotation tool link (use version 1 of the tool for compatibility with matterport code) and have about 200 frames annotated with ball location and x,y coordinates given by the tool for the bounding circle.

Like this:

But, the ball is occluded by the bottom player's body or the net tape at times and at others practically invisible because it's moving too fast (other times, it is elliptical in the direction its moving).

A potential solution I saw used is below in an algorithm called Tracknet.
Tracknet:  Tracknet ball tracking I contacted the creators of it and was told that they would open source but it's been over a year so I want to try to mimic it.

Edit:  My Q and A via email with the Tracknet Team:

  No.

  We concatenate three images that are 9 slices in total and then input
  them together into a network but keep the rest of the network the
  same. In such a method, the calculation overhead is only on the first
  layer.

  If you would like to have a design like the figure in your previous
  email, we would suggest you applying an RNN or LSTM to track the ball.

  Best, Tsi-Ui

"9 slices in total"...does this mean RGB for each of 3 consecutive frames?

Also, I know from Karpathy Pong, how to feed difference frames but how would I feed 3 consecutive frames as done in Tracknet?

I know that 2 frames is not enough having manually gone through them and 3 seems to be the minimum required to have at least one visible tennis ball.

Furthermore, I was advised by Adrian Rosebrock of pyimage that I would need trajectory estimation and a high FPS camera so this is another avenue for investigation although Tracknet seemed to do it with neither of these features.

Edit:  I am reading the Deep Learning Book chapters on CNNs to learn more about how they process input info at a low level so I can figure out what concatenate 9 slices means.  Book chapter:  CNN chapter
Just as a thought, I was thinking of calculating a collection of difference frames at t - (t-1), t - (t-2),...,t- (t-n) which might help in approximating ball location.

Edit:  I just saw this video by Andrew Ng on outputting real numbers from a nn for object detection and localization:  coursera video
So, if that is possible, then I can output y =[x_0, y_0, x_1, y_1, x_2, y_2] which are the x, y coordinates of the center of the ball at t, t+1, t+2 and take the mean squared error between y-hat and y for a loss.
Note:  When feeding the network, I might have overlapping frames, ie. feed [t, t+1, t+2] and then feed [t+1, t+2, t+3] which has the intersection of t+1, t+2.  Also for frames for which the ball is too blurry, I am going to train a separate nn that gets inputs of ball location at t_0 and t+n concatenated and outputs ball location at t_j where j is sometime between 0 and n.  The use these outputs to annotate the "invisible ball" frames, then use this "complete" set of ground truth frames concatenated as the training set for the main nn tha tracks the ball in video.


texts: Character segmentation using deep learning

texts:
I'm developing a character segmentation algorithm for license plate OCR. My algorithm includes two steps: segmentation and recognition. There is almost no problem for recognition thanks to CNN. My problem is about segmentation. I know some state-of-the-art algorithms such as binarization + connected component labeling (or contour detection), MSER, vertical and horizontal projections. Aforementioned approaches are not strong. For example, in case of character breakage they fail. I, my self use adaptive local thresholding method and analyzing connected component for segmentation.
I need a deep learning approach for segmentation or any segmentation free OCR such as STN-OCR (the problem about STN-OCR is that I can not find any tutorial code for it, specially for for training such networks).
Is it convention using autoencoders or GAN for theresholding?"

A also provide some license plates as below:


texts:
Can I install Tensorflow in Anaconda without using Keras?
If I can what is the difference between using Keras with Tensorflow and only Tensorflow?

Thanks..


texts: Training LSTM with different sequence lengths in Keras functional api

texts:
I am trying to train an LSTM model using Keras functional API. My training data is of shape:

>>> data.shape()
(100000,variable_sequence_lengths,295)

where 100000 corresponds to the number of instances (the whole number of sequences) and 295 denotes the number of features in each element of a given sequence.

I am getting errors regarding the shape of the input data. How to define the shape of the data in the input layer and the subsequent one (LSTM) considering variable sequence lengths?

from keras.layers import Input, Dense, concatenate, LSTM
from keras.models import Model
import numpy as np

inputs = Input(shape=(x,y,z))
x=LSTM(128, return_sequences=True, input_shape=(a,b,c))(inputs)
....

What values should x, y, z and a, b, c take?


texts:
I have the following three sentences, extracted from a dataframe. I would like to check the similarity and create clusters based on their level of similarity.

Authors       Sentences
John Smith   Some people do not completely understand the risk of UV rays.
Jane Lower   People do not understand the risk of UV rays, wrote the journalist in the Herald.
Craig Avatan In Berlin, people do not know how dangerous can be for their health a long exposure to UV rays.

I would need to cluster them based on words and their sequences (like plagiarism).
I have tried to use k-means, but I have not completely understood how to create clusters to plot.
Something like this:

I have tried to use k-mean as follows:

def sent_tokenization (line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words_list]
    return words_lemmed

tfidf_vect = TfidfVectorizer(tokenizer= sent_tokenization)
tfidf = tfidf_vect.fit_transform(df['Sentences'])

kmeans = KMeans(n_clusters=2).fit(tfidf)

However I am not able to plot the results. What I am looking for is something that can be easily visualised. Specifically, I would need to plot in a scatter plot as in the example, which can show the name of authors based on their sentence similarity, like in plagiarism. I am trying to see which authors have written similar texts.

In my example, I should have the first two authors closers than the third one, as their sentences are very similar (in terms of words and structure).

Could you please give me advice on how to plot/cluster the above information?
If you need more information, feel free to ask.


texts:
I'm trying to apply SMOTE to a dataset that has time-constraints. I have information about users visiting a website. For some features, there are time constraints, e.g having the first visit and the last visit at the website, the first visit (timestamp) is always lower or equal than the last visit.
If I apply SMOTE(or SMOTENC for categorical), I end up having synthetic samples for which the last visit occurred before the first visit. This leads to a sample that cannot exist in the real-world, hence can affect negatively the performance of the model.
Is there a way to apply SMOTE and impose certain rules. Or in alternative, are there oversampling techniques that can deal with this problem?


texts:
I see from reading that the medical industry is using machine learning to apply to small data set and wanted to understand how this worked? As such I created a small data set for my wife's business.

I have a data set that will be around 100k units, how to I determine the best predictive or factor model to use when approaching this small'ish data set? I want to avoid overfitting etc.

Any common pitfalls that I should avoid and/or tricks I should use to get more out of this data?


texts:
Is there a way to check the result of A/B test to determine the better one.

Suppose,

Group A has 100 views and 10 clicks.

Group B has 10 views and 2 clicks.

How to chose which one produces more clicks ?


texts: Do I need to standardize my one hot encoded labels?

texts:
I'm trying to do a simple softmax regression where I have features  (2 columns) and a one hot encoded vector of labels (two categories: left = 1  and Right = 0). Do I need to standardize just the vector of features or also the vector of labels? when I do that all my zeros and ones transform in different numbers and also I don't know how to identify who is the Left or the Right category. I'm using mxnet and gluon.
Here is how I standardize:
labels = (labels - labels.mean()) / (labels.max() - labels.min())

labels before standardization:
[0. 1. 1. 1. 1. 1.

labels after standardization:
[-0.5633803   0.43661973  0.43661973  0.43661973  0.43661973  0.43661973 ...

How can I  after identify (with strings) if my prediction is actually giving me Left or Right?


texts: How to get similar visualization to R fviz_cluster function in Python?

texts:
It looks like R has some cool visualization function for clusters that gives output like this:

The input is 2D Points and labels for them.
How can i get same visualization in Python?


texts: Feature selection for time series prediction

texts:
I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables.

When calculating correlation between variables (e.g. Close price of Tesla vs Close price of Microsoft), would differentiating the curves give a more accurate (or correct) correlation index ? I'm finding values in the range 0.7-0.9 for non-differentiated variables, and lower values after differentiation.
Once I have a correlation matrix of all my variables, is there a way to figure out which ones would add information to the neural net and which ones would just add noise ?


texts: How deep should ones linear algebra knowledge be before starting data science?

texts: How important is linear algebra to being a data scientist? Are we talking college postgraduate level?


texts:
I have a very basic question on the optimization algotithm, when I'm adjusting weights and biases in a NN, should I:

Forward propagate and backpropagate to calculate gradient descent (DC) for each batch once and then repeat for iterations_number times.

or

Forward propagate and backpropagate to calculate gradient descent (DC) one batch for iterations_number times and then continue with the next batch.


texts: How to detect only the lane boundaries using Hough transform?

texts:
This is my code to detect the lane boundaries. But only one line(left most) is detected here. Is there a way to remove other objects from the picture (remove bushes and trees) and detect only the straight line?

import cv2
import numpy as np

img = cv2.imread('lane.jpeg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(gray,50,150,apertureSize = 3)

lines = cv2.HoughLines(edges,1,np.pi/180,200)
for rho,theta in lines[0]:
 a = np.cos(theta)
 b = np.sin(theta)
 x0 = a*rho
 y0 = b*rho
 x1 = int(x0 + 1000*(-b))
 y1 = int(y0 + 1000*(a))
 x2 = int(x0 - 1000*(-b))
 y2 = int(y0 - 1000*(a))

 cv2.line(img,(x1,y1),(x2,y2),(0,0,255),2)

#cv2.imwrite('20.jpg',img)
cv2.imshow('hough.jpg',img)
cv2.waitKey(0)
cv2.destroyAllWindows()


texts: Any difference between adding epochs and duplicating data for neural nets?

texts:
Let's say I am training a neural net (e.g. convolutional network or LSTM).

Generally, the longer the training (more epochs) leads to better accuracy, albeit at times at the expense of overfitting.

Another approach is to duplicate the data. Again, if I have 1000 examples, I could duplicate them 10 times and randomise them, and the neural net will learn better (but again, it may not be able to generalise well).

Is there any difference between duplicating data and adding epochs when training a neural net in terms of its learning?


texts: How to: Plot global mean precipitation data from the NDAA onto map regions?

texts:
Wondering how to/ if I can display amount of rainfall across regions (in env variable) on the map areas so that the points only pertain to separate languages and sized by rainsetPropn. Have used data from the NDAA climate prediction centre for precipitation data (averaged over 30 years). This is what I wish to plot on the map regions versus in dots. I have a variable and respective coordinates for regions: Africa, Australia, Papunesia, South America, North America, Eurasia.
Data available here
This is the current code for the image:

#install.packages("rnaturalearth")
#install.packages("rnaturalearthdata")

#library(rnaturalearth)
#library(rnaturalearthdata)

world <- ne_countries(scale = "medium", returnclass = "sf")

d$longitude <- if_else(d$longitude<180, d$longitude+0, d$longitude-360) #this line recalculates longitude values so they map onto the world correctly.

ggplot(data = world) +
  geom_sf(color = "black", fill = "black") +
  xlab("Longitude") + ylab("Latitude") + theme_bw() +
  ggtitle("Map of languages + env + rainsetPropn", subtitle = "XX languages") +
  geom_point(data = d, aes(x = longitude, y = latitude, colour = env, size = rainsetPropn), shape = 20, show.legend  = TRUE) +
  coord_sf(xlim = c(-180, 180), ylim = c(-90, 90), expand = FALSE) +  #this is where the coordinates can be restricted for greater clarity of a single region
  theme(panel.grid.major = element_blank(), panel.background = element_rect(fill = "aliceblue"))

So far, I have found this site: but not sure if helpful for this purpose (beginner here).


texts: What are suitable datasets for univariate time series forecasting with RNNs, LGBM, TBATS, SARIMA models (topic, frequency, sources)?

texts:
I am currently looking for a suitable dataset (univariate time series) for short-term forecasting using lag features or moving windows of lag features to employ models like LSTM, GRU, SARIMA, LGBM, TBATS and ETS. I have been able to successfully implement the models for a hourly traffic flow dataset using no external features, but I have tried 5 different datasets since then and the simple models (ETS, ARIMA) give nonsensical point forecasts so I don't really know how to find a second suitable dataset. What can i look out for when trying to find a useful dataset if I don't want to leave it up to chance?
So far I have also explored daily Air Quality AQI data, hourly water meter consumption data and monthly electricity demand data but the point forecasts of the simpler models were never useful.
Or can it just be the case that the data were simply too random to predict?
Any advice will be much appreciated.


texts: is there any rule to apply pca to the imbalance data?

texts:
Is there any rule to apply PCA to imbalanced data? (randomforest, xgboost)

I used multiclass imbalance data to pca
but the log-loss accuracy getting decrease
any theoritical background of this?


texts: Why does the overfitting decreases if we choose K to be large in K-nearest neighbors?

texts:
I am studying machine learning and I am focusing on  K-nearest neighbors . I have understood the algorithm, but I have still a doubt, which is on how to choose the K for the number of neighbors.
I have seen that choosing $K=1$ lads to a frafmented decision boundary, while if I choose a larger value of K, I obtain a smoother boundary:

I think that I have lear the reason of why this happens.
But I have also studied that if $K=1$ we do overfitting, whiile if we increase $K$ the overfitting decreases, but why?
So, why does the overfitting decreases if we choose K to be large in K-nearest neighbors?


texts: Why does storing the output of .count() result in all 1's or 0's?

texts:
I have a dataframe which I run loc on to find all values within the loc parameters. Whatever the loc returns gets stored in a variable. I try to use this variable to store it in another dataframe column but it only returns 1's and 0's for the whole column.
How would I get what is being printed to be properly appended to a new dataframe?
I think the .count() has something to do with it, meaning it is changing the value of s4 and s5 when applied to the new dataframe.
for name in pros_unique:
    s4 = df.loc[(df.pros_split == name) & (df.orc_4 == 1)].orc_4.count()
    s5 = df.loc[(df.pros_split == name) & (df.orc_5 == 1)].orc_4.count()

    tots_df['orc_4_totals'] = s4
    tots_df['orc_5_totals'] = s5
    # This prints out in the correct format
    print(name, s4, s5)

Output of print statement:
name1 77 22
name2 62 32
name3 43 29
name4 1 1
name5 61 24

This is the tots_df dataframe:

pros    orc_4_totals    orc_5_totals
0   name1     1        0
1   name2     1        0
2   name3     1        0
3   name4     1        0
4   name5     1        0
```


texts: Can Neural Networks be trained to smooth values / output the average?

texts:
Let's say we have a neural network with one input neuron and one output neuron. The training data $(x, f(x))$ is generated by a process

$$f(x) = ax + \mathcal{N}(b, c)$$

with $a, b, c \in \mathbb{R}^+$, e.g. something like

feature  | target
-----------------
0             0.0
0             1.0
0             1.5
0            -1.2
0            -0.9
...

I know that neural networks can deal pretty well with labeling errors in classification problems. Meaning if you have a large dataset and a couple of examples have the wrong label, they get basically ignored.

But for this kind of problem I'm not too sure. A first experiment indicates that they do smooth values.

Are there choices in architecture / training which help the smoothing / averaging / removal of noise?

What I tried

I created a network which can solve this kind of regression problem without noise. It gets a MSE of about 0.0005. When I add a bit of noise to the training set only, I get an MSE of 0.001:

#!/usr/bin/env python

# core modules
import random

# 3rd party modules
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import train_test_split
import numpy as np

def main(add_noise=True):
    # Get data
    xs, ys = create_data_points(10000)
    x_train, x_test, y_train, y_test = train_test_split(xs, ys, test_size=0.20)

    # Add noise to training data
    if add_noise:
        noise = np.random.normal(0, 0.1, len(x_train))
        x_train = x_train + noise

    # Create model
    model = create_model()
    model.compile(optimizer='rmsprop',
                  loss='mse',
                  metrics=['mse'])

    # Fit model to data.
    model.fit(x_train, y_train, epochs=10, batch_size=32, verbose=1)

    # Evaluate
    y_pred = model.predict(x_test, batch_size=100).flatten()
    print("MSE on test set:")
    print(((y_pred - y_test)**2).sum() / len(y_test))

def create_data_points(nb_points):
    xs = []
    ys = []
    for i in range(nb_points):
        x = random.random()
        xs.append(x)
        ys.append(2 * x)
    return np.array(xs), np.array(ys)

def create_model(input_dim=1, output_dim=1):
    model = Sequential()
    model.add(Dense(200, input_dim=input_dim, activation='relu'))
    model.add(Dense(200, input_dim=input_dim, activation='relu'))
    model.add(Dense(output_dim, activation='linear'))
    return model

if __name__ == '__main__':
    main()

Outliers

In an earlier version of this question I wrote "outlier" when I meant "label noise". For outliers, there is:

The Effects of Outliers Data on Neural Network Performance


texts: Balanced Train set to predict Imbalanced Prediction set

texts:
One of the methods to address a classification predictive analysis on an imbalanced set consist on undersample the majority class (others approaches consist on: undersample the majority class, synthesize new minority classes...).

So assume hereafter we use any of those solutions and then we train an algorithm with the new generated data set. Will this trained algorithm be useful to predict further data from this system which is in general imbalanced?

Or to make it more concrete, is it possible in general to train a model with a balanced training set so that we can effectively predict an imbalanced prediction set? Or both should be generarly either balanced or imbalanced?

Thanks!

Sergi


texts: How Do I Classify Dirty/Dusty Camera Images?

texts:
I would like to test (predict) whether camera images are dusty/dirty or not.; a classification task indeed.

The question is about finding the most appropriate Machine Learning approach, perhaps any suggestions for possible Deep Learning methods specifically tailored for this particular problem?


texts: Determining the ideal parameters for KeyGraph Algorithm

texts:
Is there a method to determine the ideal amount of Keywords, High Frequency Terms and High Key Terms for the Keygraph?

To determine the ideal amount of topics for the LDA you have Topic Coherence or Perplexity for example. Is there something similiar for KeyGraph?

Does someone here have experience with the KeyGraph and can help?


texts: Testing a tensorflow network: in_top_k() replacement for multilabel classification

texts:
I've created a neural network in tensorflow. This network is multilabel. Ergo: it tries to predict multiple output labels for one input set, in this case three. Currently I use this code to test how accurate my network is at predicting the three labels:

_, indices_1 = tf.nn.top_k(prediction, 3)
_, indices_2 = tf.nn.top_k(item_data, 3)
correct = tf.equal(indices_1, indices_2)
accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
percentage = accuracy.eval({champion_data:input_data,  item_data:output_data})

That code works fine. The problem is now that I'm trying to create code that tests if the top 3 items it finds in indices_1 are amongst the top 5 images in indices_2. I know tensorflow has an in_top_k() method, but as far as I know that doesn't accept multilabel. Currently I've been trying to compare them using a for loop:

_, indices_1 = tf.nn.top_k(prediction, 5)
_, indices_2 = tf.nn.top_k(item_data, 3)
indices_1 = tf.unpack(tf.transpose(indices_1, (1, 0)))
indices_2 = tf.unpack(tf.transpose(indices_2, (1, 0)))
correct = []
for element in indices_1:
    for element_2 in indices_2:
        if element == element_2:
            correct.append(True)
        else:
            correct.append(False)
accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
percentage = accuracy.eval({champion_data:input_data, item_data:output_data})

However, that doesn't work. Mainly because I don't have a clue what I'm doing when it comes to tf.unpack and tf.transpose.

If I leave the transpose out it gives the following error:

  ValueError: Cannot infer num from shape (?, 5)

If the transpose is in there the code runs but my accuracy is always 0.0.

So I have one of two questions:

1) Is there an easy replacement for in_top_k() that accepts multilabel classification that I can use instead of custom writing code?

2) If not 1: what am I doing wrong that results in me getting an accuracy of 0.0?


texts: Multi class Imbalanced datasets under-sampling imblearn

texts:
I have an imbalanced dataset. I am looking to under-sample. Even though, the oversampling process takes less time, the model training takes a lot of time. I have taken a look at imbalanced-learn website. There are several under sampling methods. I am looking at method that tries to undersample the classes with much as possible information intact. I tried .ClusterCentroids() methods and found it takes way too long to balance the classes.

I have tried other methods that have been mentioned in the website. However, even with sampling_strategy to equal values eg: sampling_strategy={0: 2000, 1: 2000, 2: 2000} The resulting dataset is not balanced. Such as in .CondensedNearestNeighbour() and .AllKNN() methods.
Would anyone be able to help me create a class balanced dataset using these methods.

Thanks

Michael


texts:
While working with 3 classes,my dataset contains different proportions of all classes. For example,

Class "0"  has  11098 cases.....Normal
class  "1" has  2369 cases....."Abnormal"
class  "2" has  3396 cases....."May be Normal or Abnormal"

I trained the model with similar proportions. I have two questions:

Is it necessary to balance all three classes?
What is the impact of class imbalance on classification performance?


texts: How to create a model to recognize matching label and ROI with OCR

texts:
I am trying to make a model in python using Tensorflow to process the Tesseract OCR to detecting and extracting particular ROI from Image. I want to recognize the particular fields and values from Invoice with our model. For example i want to extract the Totoal amount and itemized amount and prices which is in tabular format. I want to grab only those details with our model. I am able to grab the 4 ROI cordinates of label with its's value manual in OCR. I am able to achieve these so far:

Four coordinates with OCR output for specific ROI in image
My sample dataset is like in this format [231, 404, 352, 616, 'Total Amount'] where first four values are coordinates and the last one is a label.
I am doing manually with mouse selection with my custom made python script where i select the label and all its value in particular area, row/columns with values.

This is a sample of image:

Now i want to train and prepare a model so that when i test with other invoice image having different value my model can detect and extract only that particular area on image with label and all those values either in row or column.

Now my question is How can i achieve this, How to train a model to detect only those area having label with Total Amount and all its's value present in that area on invoice image. Thanks.


texts: How to export PCA to use in another program

texts:
I'm trying to write a random forest classifier for a very large dataset, as such as part of the pre-processing i have applied PCA to reduce from 643 features to 5 PC's. Is it possible to export these settings so I can

pca.transform(data)

in another program.

I have been able to do this to the the scaler using pickle but when i run the transfrom i get given the error:
Traceback (most recent call last):

  File "<ipython-input-35-68c9849c2acc>", line 1, in <module>
    pca.transform(data)

TypeError: transform() missing 1 required positional argument: 'X'


texts:
What are some examples of the Three V's of Big Data? The three V's stand for: volume, velocity, variety.

Reference:
Three V's of Big Data, provided by Norwegian University of Science and Technology.
https://www.ntnu.edu/ime/bigdata/what-is


texts: When using torch.mm function I get an underflow error such that I get an output matrix of NaNs that is passed to subsequent layers of my model. How can I avoid this underflow error? Is it possible to calculate a matrix multiplication in log-scale, for example?


texts:
I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired results yet.

I've read that doc2vec performs well only on large datasets. I want to know if increasing the length of each data sample, would compensate for the low number of samples, and help the model train better?

For example, if my sentences are originally "making coffee", "making tea", "playing with dogs", would changing them to "making coffee requires a cup of milk and some coffee powder", "making tea requires boiling water and some tea leaves" (supplement each document with more information) help in getting better results? Would the model understand the context better?


texts:
I used Mini-batch gradient descent to train the model, but i am unable to get the proper loss graph. The loss graph is always showed as a straight line. I know there is something wrong but would anyone be able to guide me
from sklearn import metrics

error = []

for epoch in range(epochs):
    for i in range(0,x_train.shape[0],minibatch_size):
        x_mini = x_train[i:i + minibatch_size-1,:]
        y_mini = y_train[i:i + minibatch_size-1,:]

        #feed forward
        #layer 1
        in1 = x_mini@w1 + b1
        out1 = sigmoid(in1)

        #layer 2
        in2 = out1@w2+b2
        out2 = sigmoid(in2)

    if epoch > 10:
        error.append(0.5*np.power((y_mini-out2),2).mean())

z=np.arange(len(error))
plt.plot(z,error,label="train",color='red')
plt.legend(loc='best')
plt.show())

I wanted the graph for loss before training and after but cant seen get a proper one


texts: I built a convolutional LSTM model for the classification of 4-image time series. I used n keras ConvLSTM layers, followed by a time-distributed flatten and a few dense layers, finalized by a dense softmax layer with m outputs for the classes. I train using k-fold cross validation with "leave one subject out", as it is processed sensor data from 10 subjects. Training works well and accuracies go up for both training and test set, as one would expect a typical learning curve. However, when looking at the confusion matrix of the predictions, I noticed that the model almost always predicts either 100% correct or 100% wrong for a certain class (see image attached). This happens across all 10 folds, always with different classes to fail. Is this a hint that the model is learning something different from the actual task?


texts: Is there a way to rank the Extracted Named Entities based on their importance/occurence in a document?

texts:
Looking for a way to rank the tens and hundreds of named entities present in any document in order of their importance/relevance in the context.

Any thoughts ?

Thanks in advance!


texts: estimating single or multiple model for Multiple Time Series Forecasting

texts:
I am a newbie in the ML field. So please, neglect or better correct, if I am wrong somewhere.
I am working on a requirement where details of loading time for each page/component will be given.

Now I need to come up with a time series model where I can predict the load time for all the component for next 10 Run.
My Analyses and questions:
How can I treat the data

Multivariate - I need to perform Granger causality test to determine if one time series will be useful to forecast another variable by investigating causality between two variables  and try to fit model for each series .But performing  will be a overhead.
Multiple Time series: can try using Facebook Prophet and forecast for each series .

can we have a single model to forecast all the series or any other approach that will be helpful??


texts: How to compare fast the value with the ones of neighbours in a 3D array

texts:
Question:

I want to compare every item in a 3D array with its first neighborhoods. It's really slow when I have a 500x500x500 (e.g. with only values of 0, 1, 2) ndarray. I post the principle lines here:

import numpy as np

# Create a list to stock all the neighbours' coordinations of the voxel wanted
def check_neighbor(array, x, y, z):
   #top
   t = array[x, y , z + 1]
   #down
   d = array[x, y , z - 1]
   #left
   l = array[x, y - 1 , z]
   #right
   r = array[x, y + 1 , z]
   #front
   f = array[x - 1, y , z]
   #back
   b = array[x + 1, y , z]
   return [t, d, l, r, f, b]

# Check the voxel with all its neighborhood
def compare_neighbor(array, Value2match, Value2Bmatched):
   for index in np.argwhere(array==Value2match)
      output[index] = 1 if Value2BMatched in checkneighbor(array, index[0],index[1], index[2]) else 0
   return output

# Main
array = np.random.randint(3, shape=(500, 500, 500))
output = compare_neighbor(array, 1, 2)

This code take me hours only for the n=1 neighbour! Is there an efficient way which can also check the 2, 3... nearest neighbours? Can somebody help me?

Solution 1:

Based on jayprich's answer and n1k31t4's comments, I fused the both function into one and replace the argwhere() by where(). The advantage of this code is that we don't iterate voxel by voxel but do it in a vectorized way:

import numpy as np
# Build a helper function to SHIFT(not roll) a 3Darray
def shift_helper(array, neib_value, shift=0, axis=0):
    #Roll the 3D array along axis with certain unity
    _array = np.roll(array == neib_value, shift=shift, axis=axis)

    # Cancel the last/first slice shifted to the first/last slice
    if axis == 0:
        if shift >= 0:
            _array[:1, :, :] = 0
        else:
            _array[-1:, :, :] = 0
        return _array
    elif axis == 1:
        if shift >= 0:
            _array[:, :1, :] = 0
        else:
            _array[:, -1:, :] = 0
        return _array
    elif axis == 2:
        if shift >= 0:
            _array[:, :, :1] = 0
        else:
            _array[:, :, -1:] = 0
        return _array

def compneib(array, that_value, neib_value):
    _array = np.zeros(array.shape)
    _array[np.where((array == that_value)
                    & (shift_helper(array, neib_value, shift=-1, axis=0)
                    | shift_helper(array, neib_value, shift=1, axis=0)
                    | shift_helper(array, neib_value, shift=-1, axis=1)
                    | shift_helper(array, neib_value, shift=1, axis=1)
                    | shift_helper(array, neib_value, shift=-1, axis=2)
                    | shift_helper(array, neib_value, shift=1, axis=2)
                    ))] = 1
    return _array

# Main
array = np.random.randint(3, shape=(500, 500, 500))
output = compneib(array, 1, 2)

Solution 2:

If there would still be a faster way.


texts: Why are convolutions still used in some Transformer networks for speech enhancement?

texts: So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still utilize convolutions. Is there any particular reason for this? Doesn’t that defeat the purpose of Transformers?


texts: When to include LDA axis on a random forest analysis

texts: I am using the R package abcrf and have the option of including LDA (linear discriminate axis) with my other statistics. If I include these I get a different answer to when I don't, so I want to know when I should add them?


texts: time series prediction training on multiple lags using tensorflow 2

texts:
I am just studying tensorflow 2. Here is where I learned time series training on multiple lags using LSTM:
https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/
In this example, time series should be turned into classification format to use LSTM. For example, if the original data looks like:
date   a1   a2   target
1      1    2      0.2
2      2    3      0.3
3      3    4      0.5
...

Then, for lag=1, the data should be turned to:
       time t-1               time t-1
date   a1   a2   target    a1   a2   target
1      1    2      0.2     2    3      0.3
2      2    3      0.3     3    4      0.5
...

So if there is multiple lag, then the above data frame will include a lot of columns, corresponding to multiple time points.
This seems complex for me. But I am new to LSTM and tensorflow, I do not know if there is some other way to do this?


texts: Plotting Weibull distribution on Wind Speed

texts:
I have a 720 hourly set of wind speed and wind direction data and I want to fit the Weibull Distribution on it. After searching for some time, I wrote the following code in Python to get my distribution, I will also share my image for clarification.

ws.hist(bins=np.arange(0, ws.max()), alpha=0.5, normed=True)
weibull_params = (sc.stats.exponweib.fit(ws,floc=0,f0=1))
x= ws
def weib(x,lamb,k):
   return (k / lamb) * (x / lamb)**(k-1) * np.exp(-(x/lamb)**k)
k_shape, lamb_scale = weibull_params[1], weibull_params[3]

plt.plot(x, weib(x, lamb_scale, k_shape), label='self-defined Weibull')

Does this seem alright? How do I make sense of this? I also found a package here but I don't understand how to implement it to my code. https://github.com/cqcn1991/Wind-Speed-Analysis

Thank you.


texts: Over-sampling: is my model over-fitting?

texts:
I would like to ask you some questions on how to consider (good or not) the following results:
OVER-SAMPLING
              precision    recall  f1-score   support

         0.0       1.00      0.85      0.92       873
         1.0       0.87      1.00      0.93       884

    accuracy                           0.92      1757
   macro avg       0.93      0.92      0.92      1757
weighted avg       0.93      0.92      0.92      1757

Confusion Matrix:
 [[742 131]
 [  2 882]]

I have a dataset with 3500 obs (3000 with class 0 and 500 with class 1). I would like to predict class 1 (target variable). Since it is a problem of imbalance classes, I had to consider re-sampling methods. The result shown above is from over-sampling. Do you think it over-fits and/or that it cannot be a good re-sampling method for my case?
I am looking at the f1-score column, since it is a text classification problem.


texts: Selecting most relevant word from lists of candidate words

texts:
Let's suppose I have 1000's of training examples where each consists of a bucket e.g. 'engineering' or 'management' and a list of tags e.g. ['software', 'python', 'product'] where a human has selected the most relevant tag for the use case e.g.'software'.

So our data is like:

bucket         tags                        best_tag
engineering    [fullstack, software]       software
engineering    [java, python, software]    software
management     [technical, product]        product

What kind of model or approach would suit taking a list of tags and predicting the best tag based on some kind of underlying latent hierarchy?


texts: Is the following a good method for determining learning rate in a neural network?

texts:
Start with a default Learning rate of 0.1

Run the gradient descents normally.

If the cost function ever gives a higher value than the previous iteration, divide the learning rate by 10.

Repeat steps 1-3 until the learning rate becomes $10^{-6}$.


texts: Custom Lambda layer Keras outputs predictions. I get 'An operation has `None` for gradient' error

texts:
I have a  Lambda layer that takes input from previous layer, makes some preprocessing. Output of the Lambda layer is a prediction, and keras.losses.mean_squared_error is used.

#model
inputs = Input(shape=(len_train_data_columns,))
dense1 = Dense(777, activation='relu')(inputs)
dropout1 = Dropout(0.4)(dense1)
softmax = Dense(3, activation='softmax')(dropout1)
predictions = Lambda(custom_layer,
            output_shape= custom_layer_output_shape,
            arguments= {'experiment_config': experiment_config},
               )(softmax)

model = Model(inputs = inputs, outputs = predictions)
model.compile(optimizer=adam(lr=0.0001), loss=keras.losses.mean_squared_error, metrics=keras.losses.mean_squared_error)

Output of Lambda is (num_rows, 1).

I get ValueError: An operation has None for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

Why is that? I use Keras loss...


texts: Feature selection by overfitting a small sample size

texts:
I am using a CNN based model to do sequence classification. Since training an entire dataset is very expensive, and I have a large set of features needed to try, its impossible for me to select features by full trainings.

Usually a good sanity check before training, is try to let model overfit a small set of training samples, to make sure the model is at least capable of remember a small sample size.

Borrowed from this idea, my question is, can I train a small subset of training data, and use its loss curve as a metric, to select best features? Each training is to test how quick would the training loss converges given a subset of selected features.