texts: spaCy - Text Preprocessing - Keeping "Pronouns" in text

texts:
I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text).

I have built a custom text classification model. I have below method that I run on all input text, before serving it to my model. (both in training and testing).

The method will remove stopwords, punctuations and lemmatize the text.

import spacy
from spacy.lang.en.stop_words import STOP_WORDS
import string

def normalize(text, lowercase, remove_stopwords, remove_punctuation):
    nlp = spacy.load("en_core_web_sm", disable=['parser', 'tagger', 'ner'])
    stops = spacy.lang.en.stop_words.STOP_WORDS

    if lowercase:
        text = text.lower()
    text = nlp(text)
    if remove_punctuation:
        text = [t for t in text if t.text not in string.punctuation]
    lemmatized = list()
    for word in text:
        lemma = word.lemma_.strip()
        if lemma:
            if not remove_stopwords or (remove_stopwords and lemma not in stops):
                lemmatized.append(lemma)

    return " ".join(lemmatized)

Consider below input string:

Input: You're very beautiful!

If I clean that text, using my method:

test_text = "You're very beautiful!"
test_text = normalize(test_text, lowercase=True, remove_stopwords=True, remove_punctuation=True)

It will return: -PRON- beautiful

Is this a "good" way to clean the text? I notice that -PRON- is kept and therefore also used when training the model (same goes for when testing the model, as I use the same normalize method).

Should the -PRON- also be removed from the text? And should I complete even more text preprocessing?


texts: AUC for basic models is higher than bagged models

texts:
Is correct to get a bit lower AUC with bagged algorithms than no bagged algorithms? The first figure shows the ROC with bagged algorithms and the second figure shows the ROC without bagging. The curves represents the mean function in a 10-fold cross validation.

As expected the mean accuracy of a 10-fold cross validation with bagged algorithms (1st Figure below) is higher than without bagging (2nd Figure below):

I thought that bagging algorithms performs better than without bagging. Am I wrong?


texts: What is "energy spectrum" in Latent Semantic Indexing (LSI)?

texts:
What is meant by energy spectrum in LSI(Latent Semantic Indexing)?

I am doing topic modeling with gensim's LsiModel, and part of the output per chunk is the following:

INFO : preparing a new chunk of documents
INFO : using 100 extra samples and 2 power iterations
INFO : 1st phase: constructing (100000, 600) action matrix
INFO : orthonormalizing (100000, 600) action matrix
INFO : 2nd phase: running dense svd on (600, 20000) matrix
INFO : computing the final decomposition
INFO : keeping 500 factors (discarding 6.560% of energy spectrum)
INFO : merging projections: (100000, 500) + (100000, 500)
INFO : keeping 500 factors (discarding 0.843% of energy spectrum)
INFO : processed documents up to #1400000

The above output is from 1,400,000 documents into the process, (out of aproxx. 3,500,000), and it appears to discard less and less for each chunk. The 2nd chunk was higher:

INFO : keeping 500 factors (discarding 6.556% of energy spectrum)
INFO : merging projections: (100000, 500) + (100000, 500)
INFO : keeping 500 factors (discarding 13.469% of energy spectrum)
INFO : processed documents up to #40000

I am not sure whether the discarding X % of energy spectrum is better with high or low numbers. Is the "energy" analogous to entropy? Does discarding mean it looses information, or does it mean the Singular Value Decomposition is getting better and better with more information? (Or none of the above)


texts: Human intuition behind SVD in case of recommendation system

texts:
I checked the SVD for recommendation engine thread but it does not answer my question. I struggled very hard to understand the SVD from a linear-algebra point of view. But in some cases I failed to connect the dots. So, I started to see all the application of SVD. Like movie recommendation system, Google page ranking system, etc.
Now in the case of movie recommendation system, what I had as a mental picture is...
The SVD is a technique that falls under collaborative filtering. And what the SVD does is factor a big data matrix into two smaller matrix. And as an input to the SVD we give an incomplete data matrix. And SVD gives us a probable complete data matrix. Here, in the case of a movie recommendation system we try to predict ratings of users. Incomplete input data matrix means some users didn't give ratings to certain movies. So the SVD will help to predict users' ratings. I still don't know how the SVD breaks down a large matrix to smaller pieces. I don't how the SVD determines the dimensions of the smaller matrices.
It would be helpful if anyone could judge my understanding. And I will very much appreciate any resources which can help me to understand the SVD from scratch to its application to Netflix recommendation systems. Also for the Google Page ranking system or for other applications.
I am looking forward to seeing an explanation more from human-intuition level and from a linear-algebra point of view: how does the SVD work deep down from the core?


texts: Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster

texts:
On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script:

dumbo start matrix2seqfile.py -input hdfs://hm1/user/trainf1.csv -output hdfs://hm1/user/train_hdfs5.mseq -numreducetasks 25 -hadoop $HADOOP_INSTALL

When I inspect my file system, I find that all the files produced are accumulated only in the hvs node.

Ideally, I'd like the files to get distributed throughout the cluster--my data is not getting balanced throughout the cluster. Can anyone advise me on how to fix this?


texts: Find irregular bounding shape of 3D particle distribution

texts:
I have some 3D particle data

e.g.

(x0,y0,z0)
(x1,y1,z1)
(x2,y2,z2)
...

I want to find the irregular bounding shape of the distribution. The image below shows an example distribution from three directions (x/y, y/z, x/z), as well as a spherical bounding box, which is a poor approximation to the true distribution.

For this particular use case, I wish to throw away particles near the edge of the distribution, since in my application these are subject to boundary effects. So, if I have a rough approximation of the bounding shape in 3D, I can scale this down and discard particles outside the re-scaled shape.

Solutions in interpreted languages (Python, R) preferable.


texts: Which algorithm would be suitable for clustering a billion datapoints?

texts:
I am running a K-means algorithm (using the sklearn implementation) on an aggregated dataset of ~350k datapoints on a 6 dimension hyper-plane (using 6 features).
I would like to do the same but in the "non-aggregated" version of my dataset, which is ~1b datapoints using the same 6 features
I know this is a very heavy task for K-means, the number of datapoints is just too big, even though the dimensions' size is pretty small.
Are there any suggestions of other algorithms that would help me on this task, apart from mini batch K-means ?


texts: Exploration vs exploitation tradeoff to find a price that maximizes revenue

texts:
Is there a practical strategy that can learn to price a product optimally? Right now I have the following arbitrary hill-climbing algorithm:

Run an experiment at starting price P and gather 500 data points (e.x. 20 buy and 480 not buy).
Run a t-test on what confidence level P yields higher revenue per visitor than P * 1.1 and P * 0.9. Then do a 3-way weighted coin-flip and the winner gets to run the next experiment.

There's many problems with this approach. For example, if price is at optimal, it can't price a product at a more optimal pricing e.x. P * 1.03. Another is that if at some price point P = K we happen to get really unlucky and get 1 buy of 500 data points, the algorithm won't converge fast.

The problem gets easy if we take lots of data points but that would reduce long term revenue. Is there a fast algorithm that can converge to the optimal price and then not do anymore exploration?


texts: Utilizing 1x1(x1) convolutions as a learned max pooling (3D)?

texts:
I have a semantic segmentation network that ingests 3D images (hyperspectral $(x, y, b)$) and predicts 2D images (semantic map $(x, y)$). This network takes the form of a classic UNet, though it utilizes 3D convolutions on the encoding side and 2D (de)convolutions on the decoding side.
Through the skip connections I have been utilizing a 3D max-pooling to collapse the hyperspectral band dimension, $b$, to $1$ so that I may keep the receptive field and structural information through the skip connections while still being able to perform them. That is, to actually perform them I need to concatenate my 3D tensor from the encoding side onto a 2D tensor in the decoding side.
Max-pooling "does the trick" though I have been hinted that I may be able to utilize 1x1x1 convolutions to perform an operation with the same result, though its parameters would be learned. Thus making it a "smart" pooling layer. The kernel essentially would represent a single small fully connected network that learns what contributions from each band are the most beneficial to the decoding process.
I have implemented this in my network, though, I am not sure if it is technically correct. Currently my network works with 5D tensors with the dimensions $(sample, x, y, band, filter)$. To perform this 1x1 convolution (which from the inception paper and other sources performs dimension reduction in the filter dimension) I necessarily had to reshape the tensor to $(s, x, y, f, b)$, perform the operation to achieve $(s, x, y, f, 1)$, and finally reshape again to achieve $(s, x, y, f)$.

Is this utilization of a 1x1 convolution in place of a pooling layer technically correct in the manner described? Is it proper to reshape my tensors in the network as described to perform this operation?


texts:
Concept drift means that the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways.

With reference to the classic house price prediction use case:

House prices change over time thus the model I use today could make no sense in the future.

What is the best approach to address concept drift?

Do we keep updating the input replacing older house prices of yesteryear?
Do we add an extra feature for Date of Sale - by including a temporal aspect as a feature with larger data sets?
Do we eventually change model hyperparameters during training to build a model that fits better the new data?


texts: Does anyone know if the rank:ndcg is available on AWS Sagemaker? I am currently trying to run a model, but it seems like it's not implemented. Am I using an older xgboost version? Kinda new to AWS Sagemaker, so if you have any tutorials/docs, shoot them over.


texts:
I'm a little bit new to machine learning.
I am using a neural network to classify images. There are two possible classes. I am using a Sigmoid activation at the last layer so the scores of images are between 0 to 1.
I expected the scores to be sometimes close to 0.5 when the neural net is not sure about the class of the image, but all scores are either 1.0000000e+00 (due to rounding I guess) or very close to zero (for exemple 2.68440009e-15). In general, is that a good or bad thing ? How can this behaviour be avoided?
In my use case I wanted to optimize for recall by setting a lower threshold but this has no impact because of what I described above.
More generally, how can I minimize the number of false negatives when in training the neural net only cares about my not ad-hoc loss ? I am ok with decreasing accuracy a little bit to increase recall.


texts: I have age, gender, height, weight and some other similar parameters of 15000 subjects. I also have one column showing if they had a medical condition (present in about 20% subjects). I now want to analyze the factors which are most contributing to the medical condition. The usual test for this is Logistic regression analysis. However, that assumes that the relationships are linear. What other machine learning tests can I apply for this purpose with this kind of data? Thanks for your insight.


texts:
I was looking for an implementation of audio\timeseries generation models.
I want a model that utilizes time series as well as a frequency map (such as log-Mel or STFT). My goal is to generate a time series that is as true as possible with respect to some dataset distribution. Is that task-specific? Is a model for generating speech, for example, different from a model that generates music? (I am assuming it is). If so, in what way are they different?
I was leaning towards GAN.

What other methods can I use?
What are the pros and cons?

B.T.W
links for implementations would be more than welcome.


texts:
What is the MAP@K metric for?
What are you measuring? And where does it make sense to use it?
Unfortunately, I can't find much about this on the Internet. Could someone help me with this? Thanks in advance.


texts:
I have a small dataset of 150 records with 25 features (too small to do train/test).
I'm using nested cv for both hyperparameter tuning and feature selection. 10cv in the outer loop, 5 cv in the inner loop.
Eventually i'm getting 10 sets of hyperparameters and 10 sets of selected features.
If i'm publishing my results, how eventually would i know which features should be selected for a model to be tried on an external data. (Currently i don't have another external data to test the model)
thank you


texts: How far or close would feature importance information from an ML model is from causal diagrams?

texts:
The title pretty much covers my question, but to elaborate it: given data (let's assume, for simplicity, it is good enough representation of the underlying distribution) for a binary classification problem (again, for simplicity, and to give a 'feel' of treatment and control groups), when we employ a machine learning model such as random forest, we eventually obtain feature importance from the trained model. The training has taken care of data imbalance using up or down sampling or some other method, as well as used proper samplings such as stratified during training and validations, to mimic randomized control trials. Let's also assume that we have all the confounds in the feature list, i.e., no other confounds left.
I know that an ML model would only hope to learn correlations but certainly not causality among features. How far or close would the feature importance plot would be from the actual causal structure? Sure, there won't be any causal arrows in the feature importance plot. Would a first guess on the causal arrows going from the most important feature to the least important feature be too far from reality? Genuinely trying to understand this issue rather than giving opinion here. If there is also some reference that discusses this, that would be helpful too.


texts: Can you have too uniform test data in a feedforward neural network?

texts:
I have been playing around trying to implement my own feedforward neural network. To try it out I decided on an easy example. 3 inputs, 3 output. When you send in (1,0,0) the expected output is (0,1,0). When you send in (0,1,0) the expected output is (0,0,1). And so on, you see the pattern.
When working with a test data size of one I get a perfect result. After 100 training sessions with the input of (1,0,0) i get (0.0192068, 0.980584, 0.0187576) and the more I train the better the result is.
However when I try with 2 sets of training data (1,0,0) and (0,1,0), I get the same result for trying both inputs after the training: (0.0189567, 0.499995, 0.500002) which isn't really what I wanted. The more I train the more even the result gets.
When I try with a 100 diffrent training examples (using an equal distribution of all 7 combinations: (1, 0 ,0), (0, 1 ,0), (0, 0 ,1), (1, 1 ,0), (0, 1 ,1), (1, 0 ,1), (1, 1 ,1)) the results get just more and more uniform : (0.420004, 0.429997, 0.430002).
So my question is this: Could this result be in response to a general weakness of feedforward neural networks, in other words, it can't handle training data that is too uniform? Or do you think this could be because of a math error that has snuck in somewhere? I am afraid I am new enough to the concept of feedforward neural networks I can't tell one from the other.
Ps.: For those who are curious about what code I am using I am following this code:
https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/src/network2.py
I am doing a port to C++ from that code example. I find porting code is a good way to learn because you really have to dig in deep into what every piece of the code is doing to get a clear understanding of how to port it. I also happen to be one of those weird ones that are more comfortable with C++ rather than Python DS.


texts: Update a pandas data frame column using Apply,Lambda and Group by Functions

texts:
I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value 'won' if the corresponding row's '% of Votes' is maximum when grouped by 'Constituency' Column otherwise the value should be 'loss'. I want to achieve the result by using a combination of apply, lambda, and group by, instead of using loops/iterations.

DataFrame : (df_andhrapradesh)

Code below works for a specific constituency in the data frame :

df_amalapuram=df_andhrapradesh[df_andhrapradesh['Constituency']=='Amalapuram']
df_amalapuram['Candidate Won']=df_amalapuram['% of Votes'].apply(lambda x:"Won" if x==df_amalapuram['% of Votes'].max() else "Loss")

Tried something like below to make it work for the entire data frame which has different constituencies but it failed:

df_andhrapradesh['Candidate Won']=df_andhrapradesh['% of Votes'].apply(lambda x:"Won" if x==df_andhrapradesh.groupby('Constituency')['% of Votes'].max() else "Loss")


texts: Trusting p-values when errors are not normally distributed

texts:
Suppose I have a multiple linear regression model and the errors are not distributed normally.Does the central limit theorem hold true in this case? should i trust the p values of the coefficients of variables assuming the residuals are not skewed?

Will really appreciate any help on this, thank you :)


texts: Confusion Matrix after XGBoost is showing positive as negative class

texts:
Please can you help me with confusion matrix. I've implemented the XGBoostClassifier. After fitting the model when I looked to the confusion matrix to view the performance on Test data. The confusion matrix was showing Positive class as Negative class. My positive class is Submitted and Negative class is Not Submitted.
My dataset is extremely imbalanced. Positive class is very small in size.
Not Submitted:1423948 count
Submitted:5775 count
I've attached the photo of confusion matrix
The reason why I'm asking is because the model which I've created is not performing well. I want to do a hyperparameter tuning. I've used ROC_AUC curve which gave around 99% score. But confusion matrix is showing something else.


texts: Every machine learing model i build, always predict wrongly almost the same samples. (Random forest, XGBoost, AdaBoost)

texts:
First of all, I'd like to apologize for any spelling or grammar mistakes.

I'm having a problem using R for a classification problem. My dataset contains ~300.000 genomic data, and the features are DNA-related features (number of dinucleotides, number of trinucleotides, the CG Content, and some more). In conclusion, I have a dataset of 300.000 rows and 84 columns (columns = features). The 84th feature is basically the classification variable (there are two classes: class 1 and class 2).

I split the dataset in train set and test set and fit the train set to create a model using different methods, like Random Forest, XGBoost, Adaboost, etc. and check the accuracy using the test set.

In every model, the accuracy of the prediction on the test set is always around 86-87%. What I noticed is that, in the test set, for EVERY MODEL, almost the same number of rows (data) is predicted right (the model predict its true class), and always almost the same data of rows (data) is predicted wrong (the model predict wrong class. For example its true class is 1 but the model predicts 2).

If I take the set of data that are predicted right for every model I've created (250.000 data from the 300.000), and build a new model out of them (splitting them again in a new train and test set), I have an accuracy of 98%.

That means that the other 50.000 data (from the 300.000), add an error at every model I'm trying to create.

But I can't just ignore them. I thought that it would be a way to create two differents models, but doesn't know how to figure this out.

Then I thought that there could be a way to find which variables from the two subsets have strong differences, but even if I figure this out, I don't know what's the way for example to scale the original dataset to prevent those differences from keeping the accuracy in 87% in every model (using the original dataset of the 300.000 rows).

In conclusion, I have a dataset that pretends like "two datasets". It consists of two subsets: the first one is always predicted right from any model (machine learning algorithm) I'm trying, and the second one is always predicted wrong. How could I build a strong model taking into account the differences?

I also took a look at the box plots of the features. The most important feature for every model is called "CG" (number of dinucleotide of cytosines and guanines in DNA). In the following image you can see the boxplot of CG feature separating the original dataset in two datasets (T = for the dataset that is predicted right in every model) and (F = for the dataset that is predicted wrong in every model). 1 stands for data of class 1 and 2 stands for data of class 2. So, T1 means data from the right predicted dataset of class 1, T2 means data from the right predicted dataset of classes 2, T means data from the right predicted dataset for both class 1 and 2, etc.

As I'm a new beginner in R and in machine learning, I hope you do understand my problem. Thanks sincerely.


texts: Is Gradient Descent central to every optimizer?

texts: I want to know whether Gradient descent is the main algorithm used in optimizers like Adam, Adagrad, RMSProp and several other optimizers.


texts: Using Transcoder Model for language to language conversion

texts:
I have a problem statement like Converting deprecated code into a modern version of the same language. I'm currently converting with a custom Rule-based engine. But the modern version of the language is so evolved that there are more than 100k modifications. So, I thought building an ML model for this purpose will solve the problem here.
Searching through the internet for any existing models for my problem statement, came across TransCoder.
My questions are :

Can I use this model or Is there any other efficient way of solving this problem?
Can I use the TransCoder repository code to train a model with my dataset or train the existing model to get high efficiency?
In Transcoder, they have used up to 600GB of open source projects for training. Can I attain that efficiency with low dataset count? Or I have to alter the code to achieve that?
Any suggestion to solve this problem, in general?

Thanks in advance.


texts:
While trying to emulate a ML model similar to the one described in this paper, I seemed to eventually get good clustering results on some sample data after a bit of tweaking. By "good" results, I mean that

Each observation was put in a cluster with high probability (as opposed to having being 50/50 between two clusters or something similar)
A high proportion of the observations were put in the correct cluster, indicating that the model actually did work.

For example, if we had observation $a$ that belonged to cluster $A$, and observation $b$ belonging to cluster $B$, then the model might output (0.99, 0.01) for observation $a$ (where the 0.99 indicates a high probability for $a$ belonging to $A$ and 0.01 represents a low probability of belonging to $B$) and (0.02, 0.98) for observation $b$. (These specific numbers are chose randomly, but generally the good results give probabilities close to 0 and 1.) However, every few times I train the model, I get a weird result; one that seems to still 'cluster' the data in a sense, but is technically wrong. A bad result would give something of the nature (0.99, 0.01) for observation $a$ (which is good), but then give something like (0.65, 0.35) for observation $b$ . In this way, when I look at the data I can tell that it has been "clustered" as the observations that belong to $A$ give different cluster probability distributions than the observations of $B$, but the model has not accomplished either of the two goals above when actually assigning clusters.

I would like to make my model more robust so that I get "good" results more often, but I don't know what I could do to do this. One thing that would definitely work to help avoid this problem would be to do the cluster training many times and take the average result; the problem with this is that each training takes somewhere on the order of hours, meaning that doing it multiple times might take on the order of days, which I want to avoid.

If there are any (hopefully quick) fixes I can do to avoid getting these odd results, I would love to hear any advice on the topic. I should also be able to answer questions if you need more information, but the paper I linked should have most of the pertinent info.


texts: LSTM for multiple time series regression with extremely large ranges

texts:
I have the following question for those which encountered the same dilemma as me:

My target is to develop a LSTM RNN for multi-step prediction for multiple time series representing daily sales of different products. The problem that I face is that the time series have very different ranges, some below 100 units per time observations, other more than 10000. Taking into account that I want to have only one model which should learn all the different time-series, I built a common array with all predictor sequences and another one for target sequences.

My latest approach was to scale in the range (0,1) only the predictor sequences in the same time, with the scope that the target sequences will be proportional with the predictor sequences. What I mean is that: higher input scaled predictors should conduct to higher predictions. Initially I tried to scale individually the input sequences and the target sequences, before I stacked them in the input array but I was not satisfied with the prediction accuracy. But after I changed the approach as described above (scaling all predictor sequences together) and left unscaled the target sequences, now my problem got bigger: the network does not converge anymore and the loss (mse) is not decreasing, althoutgh I am using a fairly large stacked LSTM network with 5 hidden layers and 10.000.000 parameters.

I would like to get an advise about how is best to handle regression for multiple time series which take values in very different ranges.

Thank you


texts: Confusion on result of K-Fold Cross Validation and Independent Test set

texts:
I am relatively new in Machine Learning. I am using Random Forest and SVM for a project. Where I did a 10-fold cross-validation on the train set. which gives the following K-Fold Average Score:

Random Forest: 0.8716 &
SVM: 0.8665

On the other hand, when I tested with the independent test set it gives the following accuracy:

Random Forest: 93.63%
SVM: 90.47%

I am confused is it ok? I mean can the test accuracy be greater than K-fold? Is it what underfitting called. Please help. TIA.


texts: After performing data Augmentation on tf.data.Dataset, should i MERGE it with original tf.data.Dataset?

texts: Kind of a silly question, but I read that data Augmentation can be used in order to solve problem of small datasets. In my case, I've got a dataset with 5 different classes and around 2k examples per class. After performing various image transformations, do I merge the augmented dataset with original dataset into a new, bigger (around 20k image) dataset, or did I misunderstood this procedure?


texts: Object Recognition for classification, is it being used in industry?

texts:
I'm wondering if e-commerce companies where products are offered by users, such as EBay, are using Object Recognition to ensure that an uploaded image corresponds to an specific type of object (clothing, shoes, glasses, etc) either to classify automatically or more importantly to filter undesired images (such as non related or even illegal types).

If so, which algorithms and/or open platforms could be use for doing so? From what I've looked it seems that HOG+Exemplar SVM might be one of the most accurate methods developed so far (http://www.cs.cmu.edu/~efros/exemplarsvm-iccv11.pdf), even having couple of public repo's with Matlab implementations (https://github.com/quantombone/exemplarsvm), but I'm still wondering if this is being used in industry.


texts: Naive Bayes Should generate prediction given missing features (scikit learn)

texts:
Seeing that Naive Bayes uses probability to make a prediction, and treats features as being conditionally independent of each other, then it makes sense that the model can still make a prediction given that there are some features missing in the test data.

I know that it is common practice to impute missing data, but why do this when Naïve Bayes should be able to make a prediction given that there are some features missing?

Can this be implemented in sci-kit learn? I tried a test set with less features, and got a ValueError as the shapes are not aligned.

So theoretically this is possible, but is it possible in scikit learn?


texts: Time series forecasting when one of the series is known

texts: I have a problem where there are two time series $\{x_t\}_{t \geq 1}$ and $\{z_t\}_{t \geq 1}$. These two time series are correlated for fixed time instant but uncorrelated with each other across time. We assume that time series $\{z_t\}_{t \geq 1}$ is completely known to us. Is there a way to incorporate this knowledge into an LSTM model for multistep prediction of $\{x_t\}_{t \geq 1}$. My main idea is that if e.g., I want to predict for $10$ time steps ahead (predict $x_{t+1}, \dots, x_{t+10}$), I can use a look-back window of size 10 on $x_{t-10}, \dots, x_{t-1}$ and on $z_{t+1}, \dots, z_{t+10}$, so a feature dimension $D=2$. My main concern is what would happen if I wanted to use a longer look-back window on $\{x_t\}_{t \geq 1}$ while still wanting to predict the next 10 values of $\{x_t\}_{t \geq 1}$. In this case the features of my input would require different look-back windows? Is this possible or is there a trick to bypass this?


texts: What is the common practice for NLP or text mining for non-English?

texts:
A lot of natural language processing tools are pre-trained with corpus in English. What if ones need to analyze, say, Dutch text? The blogs I find online are mostly saying traslating text into English as pro-processing. Is this the common practice? If not, then what? Also, does how similar a language is to English have an impact on the model performance?
For some also widely speaking languages (e.g French, Spanish), do people construct corpus in their own language and train models on it? Forgive my ignorance because I'm not able to read papers in many languages.


texts: Deep Learning in Spark Clusters vs on GPUs?

texts:
What would be a better way to perform deep learning on any given task,
either using Spark Clusters or by taking advantage of GPU's multiple cores.

Which would give less training time and be better way to go about?


texts: Architecture for Machine Learning resource management

texts:
The problem: We have several people that do a lot of ML work in our lab and have more people asking to use our equipment. We have several machines that have Ubuntu on them that people share. This causes a problem for resource contention. PersonA had all of their code/libraries on MachineA or a person is waiting on MachineB which has 4GPUs instead of 1GPU.

Looking for:
What I’m looking for is a way of creating a cluster with these machines that will allow people to utilize and share the resources. This would allow people to scale out and make better use of the resources that we already have (e.g. waiting for a specific machine that is already used). This could be used by interactively running code in their IDE or submitting jobs to the cluster like some type of scheduler (slurm, pbs). At this point the end user could look at this as a "pool" of resources. (e.g. we have 100GPUs, personA requests 4GPUs, we have 96GPUs available for someone else to request). Would also like to avoid the user from having to anything about the cluster. For example UserA simply requests 4GPUs, they don't know and don't care which machines it gets launched on.

They use a mix of Tenserflow and mxnet with keras and some theano.

Tensorflow/mxnet seem to have clustering option but don’t think this would handle the scheduling problem? or would it?

If you have successfully deployed some type of machine learning cluster with GPU nodes, I would really be interested in seeing your architect, tools, and software you used.

If this is out of scope for this forum please feel free to move this to another channel.


texts: Predict method of the perceptron algorithm

texts:
Can someone explain to me how the predict method of the perceptron algorithm works?

def predict(self, pattern):
    activation = np.dot(self.w, pattern) + self.b
    if activation > 0:
        return 1
    else:
        return -1

In this example b stands for bias.

def train(self, pattern, label, step = 0.01):
    out_label = self.predict(pattern)
    if out_label == label:
        return True
    self.w += step * (label - out_label) * pattern
    self.b += step * (label - out_label)
    return False

This is the train method. But as far as I understand in out_label there will only be either 1 or -1. How could this be possibly equal to label? If we would return activation I (think) I would understand it.


texts: Does auto.arima of the forecast package deal with seasonality and trend automatically

texts:
I'm reading some code involving auto.arima method from the forecast package in R. What I'm curious is whether there is a necessity for decomposing the time series data into seasonal, trend and stochastic compoents before passing to the auto.arima method, or is it automatically handled by the functionality of the method?
Thanks in advance!


texts:
Two of the widely known ways of saving a model’s weights/parameters.
torch.save(model.state_dict(), ‘weights_path_name.pth’)

It saves only the weights of the model
torch.save(model, ‘model_path_name.pth’)

It saves the entire model (the architecture as well as the weights)
I know those two ways to save the trained model, but I don't understand what is the difference between them when I use them for future inference. They seems like no  differences at all.


texts: Is ARIMA appropriate for time series prediction involving a mix of explanatory and independent variables?

texts:
I have a table with the following columns:

Date(Month,Year), Sold_Past_Month, Quantity_Available, Quantity_Shipping_In, Missed_Sales, Quantity_Needed

Quantity_Needed is the dependent variable that is numerical.

I want to predict Quantity_Needed and train a model using the columns mentioned above; however, I also want Quantity_Needed to train the model.

I have data from the past 5 years -> so I would have roughly 60 rows of data for one item.

Is it possible to use ARIMA for this?

If it is, what should I do next to build my model?


texts: I'm not getting the no of parameters concept in CNN

texts:
Hi guys i've attached two images of question please help me on solution.

Thank you


texts: Best neural network architecture for object detection without location

texts:
I want to build a model which is able to detect the presence of a specific object in the image without caring about the location (i.e. no bounding boxes, just 0/1 output). More specifically, the object is always the same and it is characterized by a specific pattern (e.g. a national flag or a bar code). The algorithm should be robust to scale, rotation and distortion.
What is the best architecture in this case? Is there something better than a generic CNN binary classifier?


texts:
Being relatively new to machine learning I am trying to use the simplest algorithm to classify mesh objects. I can have the mesh objects in any familiar file type such as STL, OBJ, PLY, FBX etc. I should have enough training data as well as validation data.

My question is where do I start? how is the mesh converted into some form of a shape that one of the common machine learning libraries (e.g. tensorflow, libsvm) can read?


texts:
I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the data, integrating, transforming, etc to actually get your data cleaned from any kind of duplicate, missing value, noise, etc. But I only know all of these theoretically.

The problem I have now is I have a very big set of data that I need to filter first before going on to the processing part. But I don't know where to start. Fyi, my dataset is very big that the usual spreadsheet programs like Ms Excel, Libre Office, WPS,etc can't open it. I have to use Linux terminal and commands to count the number of rows, columns etc.

What do I do in the preprocessing start? How do I 'clean' my data? I have been thinking to use Linux commands to do all of these, but I am also wondering how real data scientists clean their data. Do they do these manually or they already have some sort of software to help them? Because seriously I don't know where to start or to do. Every reference I found in the internet only explain things theoretically. ANd I need something more practical to help me understand.

What do I do with my dataset? Help please?


texts: Regression: Scatterplot with low R squared and high p-values

texts:
Based on three datasets, I have produced the scatterplot below in Python:

I am trying to fit a line on each dataset, but when I check the metrics this is what I get:

Set 1 (red):     R squared=0.002, p-value=0.651
Set 2 (purple):  R squared=0.008, p-value=0.378
Set 3 (blue):    R squared=0.001, p-value=0.714

My question: are such data sets impossible to fit? Is there any kind of data transformation I could apply, based on the scatterplot shape?

The goal is to predict from regression.


texts:
Let's say I have a model in production that predicts a certain customer behavior (for example, propensity to buy), and let's say that there is a business process underneath that takes some actions based on the model prediction (for example, offering a coupon to the high propensity customers).
When the model performance starts to drift, I'd like to retrain it. But how can I collect new historical data, since the customer behavior is indirectly influenced by the same model I want to retrain? How can I do this process without biasing the results?
One solution could be to take the aforementioned business actions on a fraction of customers, leaving out like 10%, and using this non-biased fraction to retrain the model. Any other suggestions?


texts: Why does my model, which reports high accuracy in training, does not perform well when run on the training set?

texts:
I trained a CNN based model on the Stanford Car Dataset and got 99% accuracy. I used test dataset to evaluate the model but it performed very poorly. I knew my model was overfitting and I tried to reduce that, but that's for later. The thing is that I got curious and just to see it predict something correctly, I ran the model on the "training set".

Now, this is where everything went weird. No matter how much the model is overfitting, it was supposed to give correct answers (atleast 99% of times) on training data. But it gave almost all wrong answers. I rerun the whole model, checked if there were some issues and I just can't seem to figure out what is wrong.

Using Image Data Generator

IMG_SIZE = 256
BATCH_SIZE = 16

datagen = ImageDataGenerator(
    rescale=1./255,
    validation_split=0.1)

train_generator = datagen.flow_from_dataframe(
    dataframe = df,
    directory = "/home/ashok/Downloads/cars_train",
    x_col = "filename",
    y_col = "class",
    class_mode = "categorical",
    target_size = (IMG_SIZE,IMG_SIZE),
    subset='training',
    batch_size = BATCH_SIZE)

valid_generator = datagen.flow_from_dataframe(
    dataframe = df,
    directory = "/home/ashok/Downloads/cars_train",
    x_col = "filename",
    y_col = "class",
    class_mode = "categorical",
    target_size = (IMG_SIZE,IMG_SIZE),
    subset='validation',
    batch_size = BATCH_SIZE)

Base Model

IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3)

base_model = tf.keras.applications.ResNet50(input_shape=IMG_SHAPE,
                                               include_top=False,
                                               weights='imagenet')
base_model.trainable = False

Adding to Base Model

model = keras.Sequential([
    base_model,
    keras.layers.MaxPool2D(2,2),
    keras.layers.Flatten(),
    keras.layers.Dense(512, activation = 'relu'),
    keras.layers.Dense(196, activation = 'softmax')
])

Compiling and Fitting

model.compile(Adam(lr=0.0001),
              loss="categorical_crossentropy",
              metrics=["accuracy"])

history = model.fit_generator(generator=train_generator,
                    validation_data=valid_generator,
                    epochs=10)

Last Epoch

Epoch 10/10
459/459 [==============================] - 111s 242ms/step - loss: 0.0468 - accuracy: 0.9958 - val_loss: 6.5822 - val_accuracy: 0.0061

Testing on Training Data

test_datagen = ImageDataGenerator(rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    directory="/home/ashok/Downloads/Train",
    target_size=(IMG_SIZE, IMG_SIZE),
    batch_size=BATCH_SIZE,
    class_mode=None,
    shuffle=False
)

test_generator.reset()

predicted_class_indices = np.argmax(pred,axis=1)
print(max(predicted_class_indices)) ## Also one interesting thing is that it doesnt even predict any class below 39 (min) or above 159(max) (there are total 196 classes).

Can anybody help me with these issues? Am I missing something important?

PS: Also I am curious as to why my Val_acc doesn't increase at all, even though I tried training with data augmentation too. it shouldn't be that bad in any case.


texts: How to make machine learning specifically for an individual in a group when we have the data on the group?

texts:
Lets specify the question with the help of the figure below: We know that one part of the behaviour (our target Y) will depend on common parameters (for the group). It is represented by the grey zone on the figure. And that one part is parameters specific to each individual. ( It is represented by the pink and blue zone )

The precise question is:

Knowing that we have data from the entire group. How to use all the data of the groups, to create a specific model for an individual of this group? The idea is to get a solid model/result because it is based on all the available data, but still specific to an individual.
I imagine an answer in the form of a short list of techniques to achieve that.

Let’s illustrate the question with an example:

We do a study of 100 people. Let’s name the people [0, 1, 2, 3, 4, 5 … 99]
For each person, we do a study of 30 days. Every day, we do 5 measurements of the emotion of the person (X). And then a measure of the quantify of calories in the evening meal (Y).

In this example, the goal, using machine learning, is knowing X on day 55 (after the tests), for person number 3, to be able to predict Y (on day 55) for a given person. I often came across this problem. With my experiments, tests and research, I see two possibilities:

The first option is to take all the samples. 100 people * 30
measurements (of 5 samples x and y) = 3000 points. We create a model
that connects x to y. Then we take the new x (for person number 3),
and ‘predict’ y. We then have an answer that takes into account all
the observations but is not the specificity of the person number 3.
The model and the answer are kind of an ‘average behavioural
response’ for the population. This model corresponds more to the grey
zone of the figure you see above.
The second option is to take only information from person 3 to
create the model. We have 30 points. We create a model that connects
x to y. The model is really specific to person 3. Since the model
uses fewer points, it is less accurate. We did not use our total
knowledge/data of the grey zone.

I have the deep intuition that there is a way to do better than these two options. I tried a lot but without satisfying success.

I tried, with random forest algorithms, to use all data at the same time and add an id in X vectors. This id represents the individual. It does not seem to have worked. The internet searches I have done on this topic give me unrelated results. Any help or keywords on this topic are welcome.

Is it useful or necessary that I share an example dataset?


texts:
Came across an interesting problem:

Given a paragraph describing how to do a process, need to break it down to various steps. Basically, need to determine for each sentence in the paragraph, if this is start of a new step or belongs to the previous step.

The paragraph here describes a particular process.

Example: How to install a software. So the first step in this case can be the system requirements. All the sentences regarding this need to be in the same step. The sentences thereafter stating how to download this would belong to the next step. The sentences thereafter stating how to install this would belong to the next step.

This is just breaking the paragraph into meaningful steps.

Any idea how to go about this, what algorithm/model to use?


texts: the difference meaning between latent and salient features

texts: I'm learning deep model. the words latent and salient features usually repeat.  I want to know the difference meaning between latent features and salient features


texts: Classification vs Regression Algorithms - Should exists algorithms only for Classification and/or Regression

texts:
Dummy question:

There exists algorithms that should only be used for Classification or Regression problems?

For example, should Random Forest should only be apply on Classification problems and Neural Networks for Regression problems?

Thank you for your time :)


texts: Why GA convergence curves continue as two parallel lines?

texts:
I'm working on a optimization problem and using GA algorithm (in MATLAB, ga function).

As you know MATLAB plots GA result with two curves, one for the best values and other to show the mean values and when this two curves touch each others it means algorithm has been converged.

On my cause these two curves don't get along and continue as two parallel lines until the end of max Generations number and finally a premature convergence,What makes it happen?

I have tested with other parameters value but same result. On pre-processing I used Missing value,Normalized and smoothing and tested data set belongs to UCI.

GA Parameters:

MutationFcn      : @mutationadaptfeasible: 0.03
CrossoverFcn     : @crossoverheuristic : 1.2
maxGenerations   : 200;
populationSize   : 180;
Chromosome length: 33
Test Instances   : 71


texts: I'm currently trying to learn one-shot learning using convolutional neural networks. According to this video, the CNN that I use should have been pre-trained on the MNIST. Why must the CNN be pre-trained??


texts:
I am a bit confused by reading A survey on object detection in remote sensing. They state that machine learning-based object detection consists of three essential parts - feature extraction, feature fusion + dimension reduction, and classifier training. Then, they list the feature extraction methods:

Histogram of Oriented Gradients,
Bag of Words,
Texture Features,
and more.

Later in the section, they list approaches to the classifier training, e.g.:

SVM,
AdaBoost,
k-nearest-neighbor,
neural networks.

This does not align with my understanding of things. My experience is mostly with CNNs. When I use convolutional neural networks for object detection, they consist of a feature extractor and a classifier, don't they? And the feature extractor are the convolutional (and/or other) layers. I've never come across an algorithm/technique that uses e.g. Bag of Words to extract features and then feeds this to a neural net. I can imagine e.g. Bag of Words and k-nearest-neighbors or SVM. But I thought that deep neural nets' main advantage is that we don't have to do the prior feature engineering.
Question: Does the article (the part I mention here) accurately describe the state of object detection machine learning models? Is the article just outdated (it's from 2016)? Or am I missing something


texts: In orange, how do you split a given table at a certain row id?

texts:
I was thinking of adding another column called "Test / Train" in which a 0 would indicate test and a 1 would indicate train data, and then as I'm collecting data end the train data collection phase and start entering each row with 1 in that column.  Then using select rows widget to select all rows with "Test / Train" = 0 or 1.  However, under PCA, the original columns go away so this wouldn't work!

How then, can I just select all rows with row id > k?  I need this because the past data is supposed to predict a smaller set of future data, so I want to be sure that it works and not just select the test set randomly from the training data (which is an option in Test & Score).


texts: Will Keras fit( ) function automatically shuffles the input dataset by default?

texts:
I am asking this model fit( ) function.

fit(x=array_x, y=array_y, batch_size=32, epochs=10)

The question is straightforward:

Whether fit( ) will automatically shuffle the input dataset? (array_x, array_y in this case)

I ask this question because I find my fit( ) and fit_generator( ) has different outputs, with same input. My generator feeds inputs to fit_generator( ) in order. I guess the reason is about shuffle.


texts:
I am working on a highly imbalanced dataset for a competition.

The training data shape is : (166573, 14)

train['outcome'].value_counts()

0    159730
1      6843

I am using XGBClassifier for building model and the only parameter I manually set is scale_pos_weight : 23.34 (0 value counts / 1 value counts)

and it's giving around 82% under AUC metric.

I guess I can get much accuracy if I hypertune all other parameters.

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,
       max_depth=3, min_child_weight=1, missing=None, n_estimators=100,
       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,
       reg_alpha=0, reg_lambda=1, scale_pos_weight=23.4, seed=None,
       silent=True, subsample=1)

I tried GridSearchCV but it's taking a lot of time to complete on my local machine and I am not able to get any result back.

clf = XGBClassifier()
grid = GridSearchCV(clf,
                    params, n_jobs=-1,
                    scoring="roc_auc",
                    cv=3)

grid.fit(X_train, y_train)
print("Best: %f using %s" % (grid.best_score_, grid.best_params_))

What others parameters should I target to tune considering higly imbalanced dataset and how to run it so that I can actually get some results back?


texts: Training model on a Balanced vs Imbalanced dataset?

texts:
Let's say that I have a 2-class classification problem where classes A & B have 10*N and N observations respectively.

I am pretty sure that the answer to my question depends on the specific classification problem and on the features of my dataset etc.. Still there are general analysis that can be done on my question.?

Something that I could clarify is that I am interested in having high recall in both classes ("macro-average recall"); not primarily in having the highest possible recall in the minority class as in imbalanced dataset classification problems such as spam detection, financial fraud detection or disease detection.

So for this, generally speaking, is it better to train my model on:

1) A: 10*N observations, B: N observations

2) A: 5*N observations, B: N observations

3) A: N observations, B: N observations

I am having an impression, that assuming we start with a balanced dataset then the more data you add to one class then the better the macro-average recall because of the new information added but after one point the dataset becomes so imbalanced that the performance of the model on the minority class probably deteriorates and hence the macro-average recall falls.

Am I right on this?


texts: What does this linear regression summary tells us?

texts:
I just started learning predictive modelling in R , however i do understand some terms below but i lack in making more interpretations ,  just want to know what a pro statistician or Data Scientist interpret from it . How does one look at it ?

> summary(model)

Call:
lm(formula = comp$Minutes ~ comp$Units)

Residuals:
    Min      1Q  Median      3Q     Max
-9.2318 -3.3415 -0.7143  4.7769  7.8033

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)    4.162      3.355    1.24    0.239
comp$Units    15.509      0.505   30.71 8.92e-13 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 5.392 on 12 degrees of freedom
Multiple R-squared:  0.9874,    Adjusted R-squared:  0.9864
F-statistic: 943.2 on 1 and 12 DF,  p-value: 8.916e-13


texts: Building a machine learning model based on a set of timestamped features to predict/classify a label/value?

texts:
I'm trying to apply machine learning to pharmaceutical manufacturing to predict whether batches of drug products manufactured are good or not. for the sake of relatability, let's use coffee brewing as an analogous process. Let's imagine that I'm trying to predict the acidity of the coffee that I've brewed.

The dataset that I have contains features such as temperature of water, stirring speed and pressure that are constantly measured (say..on a per second basis) over a variable amount of time (the first cup may be brewed in 5 minutes, the second in 10 etc).

What kind of preprocessing should I perform on such a multidimensional dataset? One stumbling block is that for each observation, the duration is different, which may complicate dimension reduction? Once preprocessed, is there any specific model that would suit the task at hand? I'm looking at something like a regression but alternatively, classifiers seem to be fine as well if I split the acidity(pH) into "<5.5" or ">5.5"?

I hope to get some general directions and if you can paste a few links to texts or examples that'll be good! Also, I'm more familiar with python and scikit learn, so if you can point me in the right section in the documentation that'll be great too!


texts: Best strategy to build Classifier for Mixed Data with class imbalance

texts:
I have a dataset which contains : 94 numeric features + 56 categorical features

I am trying to build a classifier to predict Target (disease/healthy).
2. Rows : 1812
3. Class imbalance ( Majority class(healthy : 1200))

Strategies i tried :

I tried SMOTE on training data( 80/20 split).
As number of features are 150 . I wanted to use feature selection . Filter methods (fs) dont work so well on mixed. So i used Wrapper fs algorithm( recursive feature elimination) on the full feature set. Used random forest for rfe().
I got a subset of 23 features from rfe (all numeric) . Wonder if rfe works with factor variables? or should i use dummy encoding first and then rfe?
I even tried hybrid feature selection alg. like Genetic algorithm but sensitivity is very low after randomforest (50 % for disease class)
I am trying to achieve a high sensitivity . I am concentrating on random forest as of now as my base classifier as it works well with mixed datatype.

Issues:
my best Cross validated accuracy is 70. Sensitivty : 50 .
How can i improve sensitivity for disease class?

Feature selection for mixed data ? I also tried PCA and FAMD for dimensionality reduction, but PC1 explains only 8 % variance. So thats out of the way.

Thank you for your time


texts: Getting Value Error while training a model for binary classification


texts: How can I normalize my output data for neural network?

texts:
I have a dataset consisting of 5 numerical variables, like the sample in the following image:

In this dataset the first four variables are the input and the GDP is the output. I am trying to build a neural network for regression of the GDP variable. For the input variables, I have used PCA to normalize the data, while for the GDP variable I have used a MinMax normalization algorithm. However, the accuracy of the results of the neural network is very low and I guess that the problem is on the normalization of my output variable. Any suggestions on how I can improve the accuracy of my neural network?


texts: Standard method to integrate tools coded in multiple languages in an analysis workflow

texts:
I am trying to stitch together multiple packages and tools from multiple languages (R, python, C etc.) in a single analysis workflow.

Is there any standard way to do it? Preferably (but not necessarily) in python.


texts: When to choose linear regression or Decision Tree or Random Forest regression?

texts: I am working on a project and I am having difficulty in deciding which algorithm to choose for regression. I want to know under what conditions should one choose a linear regression or Decision Tree regression or Random Forest regression? Are there any specific characteristics of the data that would make the decision to go towards a specific algorithm amongst the tree mentioned above? What are those characteristics that I should look in my dataset to make the decision? And are there some reasons that would make one choose a decision tree or random forest algorithm even if the same correctness can be achieved by linear regression?


texts: Pattern detection in sequence of users behaviour for clustering

texts:
I need to detect variable-size patterns in a set of fixed-size strings, where each string represents a sequence of activity of a  group of users.
As an example, i have a 10 character string or user 1 where each char represents an activity intensity: 'LLMLLMGLML' meaning that the first 2 samples where of a low intensity, the third a medium intensity and then the activity gets lower again.  Another example, for user2: 'LMMGMLLMLL'. Both users had a common pattern: 'LLMLL'
My intention is to classify users in terms of their overall activity, defining similarity based on the number of common patterns
Can you recommend an approach to this problem?


texts: How to generate data for future price predictions with Linear regression and Python

texts:
I have written the following Python code that makes predictions using linear regression model. However, I'll appreciate you pointing out what went wrong with how I generated data['X_forecast']. I'm sensing I might be wrong on the way I pulled out data['X_forecast']. Consider the following code which makes a one-day prediction:
import pandas_datareader as web
import numpy as np
from datetime import datetime, date, timedelta

start_date = '01/01/2004'
end_date = date.today()
ticker = 'TSLA'
data_source = 'yahoo'
days_into_the_future = 1
test_size = 0.25

df = web.DataReader(ticker, data_source, start_date, end_date)
data = {}
data['df'] = df.copy()
data['df']['target'] = data['df']['Adj Close'].shift(-days_into_the_future)
data['df'] = data['df'].drop(['Volume','Adj Close','Close'],1)
data['X_forecast'] = data['df'].iloc[-days_into_the_future:,:-1]
data['df'] = data['df'].dropna()
data['feature'] = data['df'].drop(['target'],1)
data['target'] = data['df'].iloc[:,-1]

X = np.array(data['feature'])
y = np.array(data['target'])
timefeatures = data['feature'].index

# split the dataset into training & testing sets by date
train_samples = int((1 - test_size) * len(X))
data["X_train"] = X[:train_samples]
data["y_train"] = y[:train_samples]
data["X_test"]  = X[train_samples:]
data["y_test"]  = y[train_samples:]
data["dates_train"] = timefeatures[:train_samples] # Extract the dates of train and test set dates
data["dates_test"] = timefeatures[train_samples:]

from sklearn.linear_model import LinearRegression
model = LinearRegression()
history = model.fit(data['X_train'],data['y_train'])

predictionlineReg = history.predict(data['X_test'])  # Linear regression
predictionlineRegfuture = history.predict(data['X_forecast'])

I know how to create features as technical indicators and perform features/training set scaling. However,  I am more concerned about the data I used to predict into the future. How would you have generated your own data['X_forecast'], or which way is ideal?


texts: Random Forrest Sklearn gives different accuracy for different target label encoding with same input features

texts: I'm using sklearn Random Forrest to train my model. With the same input features for the model I tried passing the target labels first with label_binarize to create one hot encodings of my target labels and second I tried using label_encoder to encode my target labels. In both cases I'm getting different accuracy score. Is there a specific reason why this is happening, as I'm just using a different method to encode the labels without changing any input features.


texts: Which tool should I use for combining this large dataset?

texts:
This is my first foray into data science and I've hit a snag before even getting to the analysis.

I have 40 CSV files; each one contains 2 columns - a time and value column.

I would like to consolidate this into one table by doing an outer join of the time column for all files, so that the final file would contain 1 time column and 40 value columns. I attempted this using pandas merge method, but my local machine ran out of memory before it could complete. I made sure there was nothing fundamentally wrong with the code by simply combining 8 of the 40 files and got the desired result.

At that point I decided to spin up a more powerful cloud compute instance on AWS; I chose one with absurdly high RAM so I'm working with 190 GB rather than 8. It got further, but got the same memory error around the 30th file. I should also mention that each file as quite a few rows - around 180K.

At that point I decided that I must be going about this the wrong way. I don't think pandas is meant as a tool for combining such large datasets. In my last job I used SQL pretty extensively and something like that seems more equipped. My next idea was to try to do it in AWS Athena, which is a SQL like service that can integrate with the csv files in S3. I assume there is no standard solution to this issue, but I just want to see if I'm way off base or going in the right direction.

Thanks!!


texts:
library(MASS)
library(plyr)
library(reshape)
library(ggplot2)
qplot(x = MPG.highway, data = Cars93, facets = ~Origin, geom = "histogram", fill = Origin, binwidth = 2)

I have this preceding code, and I had a question about the dataset. The question is does it look normally distributed I feel like saying that USA is normally distributed; meanwhile, the Non-USA bin is not normally distributed? I apologize in advance this is a data analytics question, which I was going to ask in Math StackExchange but it had R code.


texts:
I have a mission of classification with a lot of classes. I am comparing some ML algorithms for this case and I would like to try xgboost.
I am writing in python and I am trying to get the best 3 predictions using this algorithm but I couldn't find any method in xgboost API the fits what I want

Any recommendations ?


texts: Is it common to add noise to Time Series data before training a model

texts:
I once read about somebody who added noise to their time series before training a model. They didn't write why they did it though.

Is this common practice?

If it is, why do people do it ie. to prevent over-fitting?


texts: Deployment of tensorflow model which requires to get passed data for prediction normalized dependend on training statistics

texts:
The model trained in TensorFlow tutorial "Regression: predict fuel efficiency" can be stored with model.save('basic_regression.h5') and loaded/recovered with model = keras.models.load_model('./basic_regression.h5'). However the prediction

test_predictions = model.predict(normed_test_data).flatten()

requires normalized test data normed_test_data

def norm(x):
  return (x - train_stats['mean']) / train_stats['std']
normed_train_data = norm(train_dataset)
normed_test_data = norm(test_dataset)

which depends on training statistics

train_stats = train_dataset.describe()

which are available during training only.

In case I would like to deploy a model like this I'd have to deploy the training statistics as well. Until now I assumed that a trained model including weights, etc. and stored as .h5 file is sufficient to deploy a model into production. Have I missed something? Or is the tutorial incomplete/not suited as example for production ready models?


texts: How to fit an odd relationship with a function?

texts:
Let's say there is a function $f$ such that $y = f(x)$. However, if $f$ is a piecewise function such that:

$$y = \begin{cases} 0 \quad x \leq 0 \\ 1 \quad x >0\end{cases} $$

How do I fit $f$ in that case?

Many thanks, guys.


texts: How to identify sentiment of a given word from a sentence

texts:
Use case is like this:

Suppose I have a sentence(review data): The staffs were very kind and helpful. The room is ok for its price. There did not seem to be a heater in the room. So, a bit for our January trip.

Now if I want to know the sentiment of a word for example room then how should I proceed.

I have designed sentiment analysis for a given sentence using bag of words model, but determining the sentiment for the given word from the sentence (review data) is kind of new to me. What approach should I take here ?

Providing some approach or any link is enough.


texts: Doesn't over(/under)sampling an imbalanced dataset cause issues?

texts:
I'm reading a lot about how to use different metrics specifically for imbalanced datasets (e.g. two classes present, but 80% of the data is one class) and how to tackle the issue of imbalanced datasets.
One trick is to oversample, so to take more (or even duplicate some) data belonging to the underrepresented class. I've tried this and did achieve better results (before my models would easily just predict a single class for everything, achieving 80% accuracy lol).
However, I was wondering, will this model work well with real-life data? One of the 'laws' of data science/machine learning is that your training data has to have the same/similar attributes as your live data you're intending to use your model on. However, by oversampling, I create a dataset that's 50% one class and 50%  other, as opposed to the "natural", real-life-data having 80% of one class and 20% of the other.
So I guess the question in short is: Will oversampling my imbalanced dataset of 80/20 class distribution to 50/50 class distribution impact the usability of my model for real-life data? Why?


texts: Is it possible to use deep learning on small number of samples but each sample has a large amount data?

texts:
I have around 50 subjects and each subject has weeks worth of time series data.  The task is to identify whether a subject has as specific condition.  We have been running wavelet transformations in 30s windows across the time series and then summarizing across spectral band groupings to get a data matrix of subject x spectral band (50x6).  Although this method does reasonably well at detecting the condition, I'm nervous that this may wash away important temporal information.

Is it possible to use the wavelet transformations as images and feed this through a convolutional network?

To clarify, there are 50 subjects each with ~350 30s intervals of data.  The wavelets are run on each 30s interval producing 350 wavelet transformations per subject.  The wavelet transformations (previously) are summarized into power per spectral band, however, I would like to use each wavelet transformation as an image so each subject would now have ~350 images.


texts: Large Numpy.Array for Multi-label Image Classification (CelebA Dataset)

texts:
Task: Build CNN Model (preferably Keras or TensorFlow) to Predict Labels Associated to Each Image in CelebA Dataset (Multi-label Image Classification)

In past, for majority of multiclass/binary image classification problems, I used to feed images efficiently using ImageDataGenerator and .flow_from_directory in Keras after images are properly organized in a separate directory for each class. Therefore, I have never bothered converting images to numpy.arrary prior to feeding to the model, unless I had to and of course datasets were small so that I could do it easily in my local machine.

However, CelebA is a Multi-label Image Classification with each images having 40 labels (attributes like Smiling, Eyeglasses, Young etc.)  meaning that I can not organize them in subclasses as I used to do, so .flow_from_directory is off the table (as far as I know!). Still I've managed to convert the images to numpy.array by the following simple loop:

import numpy as np
import skimage.transform

images_path='../img_align_celeba/'

train_images=[]
from skimage import data

for filename in train['Images'].tolist()):
    tmp=np.array(skimage.transform.resize(io.imread(os.path.join(images_path,filename))/255., (64, 64)))
    train_images.append(tmp)

x_train=np.array(train_images)
del train_images

Well it was not an impossible task. CelebA dataset is large, well not super large compared to many other image datasets (>200K RGB images, totally 1.4GB in size, each image ~ 8 KB). YET surprisingly it takes the hell of the time to convert these images to numpy arrays and even stuck during the run of a small CNN model.

My computer specs: MacBook Pro (2015), Memory: 8GB, Harddisk: 128 GB.

Even with almost more than 4GM free memory, and 20 GB free hard-disk I could not manage this on my local machine.

UPDATE: It seems quite possible and way more efficient via .flow_from_directory method of ImageDataGenerator in Keras. While it is not that option for this multi label classification at hand, I simply made some dummy subclasses and it worked and the model runs much faster!!

To My Questions (finally!!):

Maybe I am not doing the image-to-numpy conversion efficiently? Please advice how I could construct the arrays in a more efficient approach!

UPDATE: I found a very similar question in stackoverflow a year ago, yet answers do not seem to offer any better alternative.

Maybe it is what it is and I only need better hardware to do it locally!?
How then Keras manage to do the conversion efficiently under the hood then?

At the end (as we speak), I sampled only 20% of the images to have at least a model prototype up and running, although accuracy is not impressive!!


texts: Curse of Dimensionality : How many dimensions is too many dimensions?

texts:
Say I have a dataset with 1000 columns and 3M rows.
I know that this is will definitely suffer from Curse of Dimensionality and that I need to reduce the number of dimensions. But to what extent am I supposed to reduce the dimensions by?

With each additional dimension the number of datapoints required such that the data is not too sparse increases exponentially according to my understanding.

So how do I know know for different number of columns what the golden number of data points is? Assuming that I have the capability to collect infinite amount of data but would still have a small cost associated for each datapoint, how much should I collect?

I am using these slides to understand the concept:
http://www.dataminingbook.info/pmwiki.php/Main/BookPathUploads?action=download&upname=slides-chap6.pdf


texts:
I have encountered the word oracle in the following context:

Given an $\alpha$-approximate oracle for stochastic optimization we
show how to implement an $\alpha$-approximate solution for robust
optimization under a necessary extension, and illustrate its
effectiveness in applications.

I saw this question, but it doesn't seem to have the same meaning. I was wondering what does oracle mean in this context.
Edits:
I found the following definition in this paper:

A $\rho$-approximate Bayesian optimization oracle is a function
$\mathcal{O}_{\rho}:(\Theta \rightarrow \mathbb{R}) \rightarrow
> \Theta$ for which: $$ f\left(\mathcal{O}_{\rho}(f)\right) \leq \inf
 _{\theta^{*} \in \Theta} f\left(\theta^{*}\right)+\rho $$ for any $f: \Theta \rightarrow \mathbb{R}$ that can be written as a nonnegative
linear combination of the objective and constraint functions $g_0,
g_1, \dots, g_m$.

I would appreciate if someone can shed some light on it.


texts: Ambiguity in Perceptron loss function (C. Bishop vs F. Rosenblatt)

texts:
Bishop's Perceptron loss

On one hand, it is stated in equation 4.54 of Chris Bishop's book (pattern recognition and machine learning) that the loss function of perceptron algorithm is given by:

$${E_p}(\mathbf{w}) =  - \sum\limits_{n \in M} {{\mathbf{w}^T}{\phi _n}} {t_n}$$

where $M$ denotes the set of all misclassified data points.

  Original Perceptron loss

On the other hand, The loss function used in the original perceptron paper written by Frank Rosenblatt is given by (wikipedia):

$${1 \over s}\sum\limits_{j = 1}^s {\left| {{d_j} - {y_j}\left( t \right)} \right|} $$
which when translated to the notation of Bishop's book is given by:

$${1 \over N}\sum\limits_{n = 1}^N {\left| {{t_n} - {\mathbf{w}^T}{\phi _n}} \right|} $$
where $N$ denotes the set of all data points.

  My question

My question is that why Bishop's version of Perceptron loss is different from the original paper? Considering Bishop's book as a highly recognized book in machine learning field, can we call it Bishop's Perceptron?!

  Scikit-learn's implementation

By the way, it seems that Scikit-learn uses Bishop's version of Perceptron loss (Scikit-learn documentation). It is apparent from the following formula and figure:

-np.minimum(xx, 0)

which for one sample reduces to:
$$ - \min \left( {0,{\mathbf{w}^T}{\phi _n}{t_n}} \right)$$


texts: Getting GitHub repository information by different criteria

texts:
New to the Data Science forum, and first poster here!

This may be kind of a specific question (hopefully not too much so), but one I'd imagine others might be interested in.

I'm looking for a way to basically query GitHub with something like this:

Give me a collection of all of the public repositories that have more than 10 stars, at
least two forks, and more than three committers.

The result could take any viable form: a JSON data dump, a URL to the web page, etc. It more than likely will consist of information from 10,000 repos or something large.

Is this sort of thing possible using the API or some other pre-built way, or am I going to have to build out my own custom solution where I try to scrape every page? If so, how feasible is this and how might I approach it?


texts: Difference between machine learning and artificial intelligence

texts: Is there any difference between machine learning and artificial intelligence? Or do these terms refer to the same thing?


texts: Relating ROC curves with class statistics


texts:
I trained a word2vec model using Gensim library which is of type .bin
Q1: can we convert this trained model in bin format to pickle?
Q2: would it speed up the execution time?


texts: Reason: Average word vector embedding encodes word content and word order effectively

texts:
I was going through a research paper: FINE-GRAINED ANALYSIS OF SENTENCE
EMBEDDINGS USING AUXILIARY PREDICTION TASKS

The key take away was Comparison of Encoder decoder and average word sentence embedding validated for accuracy for sentence embedding on 3 basic language characteristics- sentence length, word content, and word order.

I found it surprising, that an averaged word embedding for a sentence is better at predicting presence of a word in a sentence than an Encoder Decoder. Also, how is it that, increasing embedding size deteriorates it's performance.

Same question goes for word ordering, how is average word embedding able to do that? The experiments are able to explain, what would happen if prediction is based on permutation of words, but the explanation doesn't feel intuitive to me. How is simple avg word embedding able to contain information like word order when taking it's average kinda nullifies the order info


texts: How to restrict the columns to be passed to final classifier in PMML Pipeline

texts:
I am working on building XGBoost PMML using SKLearn and SKLearn2PMML.
I am having some numerical,somecategorical and datetime columns from which i am creating new feature inside the pipeline. When i am trying to train the model, it gets failed as the original categorical features also gets passed to the final classfier by default. Is there any way to restrict the features by specifying the feature names ?


texts:
In Machine Learning, is the definition of the Model just the algorithm that was selected for the problem domain, or is the Model the algorithm and the training data?

Thanks.


texts: Do records with the same key in two RDDs repartitioned by key reside in the same node in spark?

texts:
I have two RDDs named "data" and "model", they are repartitioned by key described as below :

Does the tuple records with the same key reside in the same node in my cluster ?

Should it save IO cost in shuffle operation, such as "data.cogroup(model)" , if it comes true ?


texts: Method of hyperparameter tuning for regression tree ensembles in Matlab

texts:
What regression tree ensemble methods and what parameters does Matlab actually consider in hyperparameter tuning? The documentation appears to be sparse, to say the least.
See https://se.mathworks.com/help/stats/fitrensemble.html and the example "Optimize Regression Ensemble" therein. It says "You can find hyperparameters that minimize five-fold cross-validation loss by using automatic hyperparameter optimization."
But what is the search space here?
The output in that example only displays Bag and LSBoost as methods. Does it neglect random forests, i.e. subset sampling instead of bootstrapping the input space? Or is bagging here an umbrella term that covers also random forests?
Furthermore, the output in the above example only displays NumLearnCycles (tree count), LearnRate (for boosting) and MinLeafSize (obvious). How about the CART decision tree algorithm hyperparameters? Are they included? The LSBoost algorithm appears to involve optimization of these hyperparameters (see https://se.mathworks.com/matlabcentral/answers/302103-what-is-the-algorithm-behind-lsboost-from-fitensemble-function), but, again, documentation is .. meh.


texts: Regression dataset with categorical features

texts:
I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties:

Be a tabular dataset (no images).
Have at least 20k rows, and ideally around 100k.
Have some categorical variables with many levels (at least a variable with 100 levels or more).
Ideally, the target should have long tails.

Does anyone any public dataset with these properties? I have found the stack overflow developer survey to work for me, but I'd like to have some more datasets with such structure.


texts:
I'm new to datascience. I was wondering how one should treat an extremely dominant feature.

For example, one of the features is "on"/"off", and when it's "off", none of the other features matter and the output will just always be 0. So should I drop all rows where it's "off" in my train/test data sets? I feel like I would get a better fit that way.

If I delete those rows, I'm concerned about how I would handle those rows in the test set. For example, I'd have to write code to loop through the data and put a 0 in the prediction column for those rows, as well as make sure everything else lines up. (This is all Kaggle related, so the training set is several columns of features and a y_column, whereas the test set doesn't have the y_column and we're supposed to predict it.)

I'm using Python and Scikit Learn's random forest, if that matters.


texts: PyTorch neural network parameters and tensor shapes

texts:
I am following this tutorial: https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html

it's quite nicely done, however I do not understand/see where you can know the expected image input size for the small network they have defined. They say that the images must be of size 32x32.

I know there is a formula (given in stanford CS231n ) that states that ((N-K)/strideLen) +1 = , where N is the (padded) input size and K is the 2D convolution kernel size (e.g. 3x3).

But that does not seem to help me deduce what I want.

How can I know the expect input size ? I would be happy if someone whats to go in more details for the example of this simple PyTorch conv nn.

(Please note that for sizes significanlty different that 32x32 the net outputs an error message. Also, I don't understand why it accepts inputs like 30x30 and 31x31, e.g. this is accepted:

input = torch.randn(1, 1, 31, 31)
out = net(input)
print(out)

The code is:

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

It is not explained where from the shape (16 * 6 * 6, 120)  comes from, for the linear layers, e.g. why 120 ?


texts: Different learning rates for each dimension

texts:
I have been thinking about why normalization and scaling are done for each feature in the basic context of gradient descent.

One thing that got me wondering is that we use a pre-defined set of learning rates for each of our dimensions. Now I believe if we don't normalize them we would need to keep our learning rate small due to different scales of the features, but that got me thinking, can't we have different learning rates (suited to each feature dimension scale) to compensate for not doing the normalization?


texts: User profiling with Mahout from categorized user behavior

texts:
I'm trying to cluster and classify users with Mahout. At the moment I am at the planning phase, my mind is completely mixed with ideas, and since I'm relatively new to the area I'm stuck at the data formatting.

Let's say we have two data table (big enough). In the first table there are users and their actions. Every user has at least one action and they can have too many actions, too. About 10000 different user_actions and millions of records are in the table.

user        - user_action
u1          - a
u2          - b
u3          - a
u1          - c
u2          - c
u2          - c
u1          - b
u4          - f
u4          - e
u1          - e
u1          - d
u5          - d

In the other table, there're action categories.  Every action may have none or multiple categories. There are 60 categories.

user_action - category
a           - cat1
b           - cat2
c           - cat1
d           - NULL
e           - cat1, cat3
f           - cat4

I'm going to try to build a user classification model with Mahout but I've no idea what I should do. What type of user vectors should I create? Or do I really need user vectors?

I think I need to create something like;

u1 (a, c, b, e, d)
u2 (b, c, c)
u3 (a)
u4 (f, e)
u5 ()

Problem in here, some users performed more than 100000 actions (some of them are same actions)

So; this is more useful, I think;

u1 (cat1, cat1, cat2, cat1, cat3)
u2 (cat2, cat1, cat1)
u3 (cat1)
u4 (cat4, cat1, cat3)
u5 ()

The things I also worry about are

How should I weight categories for users? For example u1 has at least three action that related with cat1, while u3 has only 1. These one should be different?
How can I decrease the difference between active users and passive ones? Like u1 has too many actions and so categories, u3 has only 1.

Any guidance are welcome.


texts: Observing a set of features while training is same as seeing it one time. How?

texts:
If we use the same training example to train multiple times, it won't bring any change to desired theta values. Can someone provide intuition behind it or some theoretical explanation to back this?

To me, it looks like using the same training example in training could help. Given a training example, weights try to converge but they don't converge to the optimum level because of small learning rate, etc. So, when the model sees the training example again, it may help the weights to converge towards the minima. Thoughts/Intuitions?


texts: Word2Vector multicontext CBOW model with Adam optimization

texts:
In cbow multiword context word2vec model there are two weights matrixes
$$ W,W^{'}$$
Where $W$ is $I$ -> $H$ weight matrix, and $W^{'}$ is $H$ -> $U$ weight matrix and output is just softmax applied to $U
$.
I use update rules which were derived and explained in this paper. I wonder, if I update the $W^{'}$ first, should I use old values of it to calculate $EH$ which is used to update $W$? Or should I first update the output weight matrix and then calculate $EH$ to update input matrix? Also - can I use Adam optimizer for two of these gradients? First use one pair of $m,v$ to update the output weights, and then use other pair of $m,v$ for input weights? Is it appropriate use of such optimization for this particular case?


texts: Intuitive explanation of Convolutional layers in Deep CNN

texts:
I am studying a bit about deep learning and CNNs. I understand the math. However, I am a bit stuck on a basic concept related to filters.
In Convolutional layers, the number of filters corresponding to the depth. For instance, if I have 128 filters, I'll apply each of these individually to the image. Since all these filters are being applied to the image, would this not make it Fully connected? (image is connected to each filter).
Can I have filters of different sizes in a single layer?


texts:
Hello folks,
            I am trying to plot three variables in a single XY plot. I am having a problem setting the x-axis limit which is for time. I tried setting limits using x_limit but getting error. I will provide the code I am using and also the fig I got. If anyone can help me in this regard it will be much appreciated.

   import matplotlib.pyplot as plt
   from   netCDF4 import *
   import netCDF4
   import numpy as np
   from datetime import datetime
   import pandas as pd
   import matplotlib.dates as mdates
   myFmt = mdates.DateFormatter('%b %d')
   df = pd.read_excel('SUCS_1982_2020.xlsx',sheet_name='TEST')
   time = df.Date
   temp = df.air_tmp
   prs = df.Pressure
   fig = plt.figure(figsize=(8,4))
   ax = fig.add_subplot(111)
   ax.set_facecolor("#f1f1f1")
   #ax = plt.axes([1, 1, 0.7, 0.7])
   lns1 =ax.plot(time, sst,'-' "r",label='SST')
   lns2 =ax.plot(time, temp,'-' "g",label='AIR')
   ax2 = ax.twinx()
   lns3 =ax2.plot(time, prs,'-' "b",label='PRESSURE')
   ax.set_ylim(26,29)
   ax2.set_ylim(920,1010)
   legend=fig.legend(loc ="upper right",bbox_to_anchor=
                       (.21,.8),bbox_transform=ax.transAxes,frameon=False)
   legend.get_frame().set_facecolor('none')
   ##########################################
   import matplotlib.dates as mdates
   myFmt = mdates.DateFormatter('%b %d')
   ax.xaxis.set_major_locator(mdates.DayLocator(interval=1))
   ax.xaxis.set_major_formatter(myFmt)
   #plt.grid(True)
   ax.grid()
   ax.set_xlabel("DAY")
   ax.set_ylabel(r"SST/AIR TEMPERATURE($^\circ$C)")
   ax2.set_ylabel("PRESSURE(hpa)")
   #ax.set_xlim(0,37)


texts: How to find the values that never matched in a full join

texts:
Hello I have done this full join query:

SELECT
  F.date AS Campaign_Date,
  F.Store AS AS Campaign_Store,
  F.IMPRESSIONS,
  F.CLICKS,
  F.SPEND,
  C.Date AS visitors_Date,
  C.ANNEE_MOIS,
  C.ANNEE,
  C.MOIS,
  C.Store AS visitors_Store,
  C.FREQUENTATION_Store
FROM
  `project.campaigns`  F
FULL JOIN
  `proiject.visitors` C
ON (
  C.Store= F.Store
  AND C.Date =F.date
  )

And now, I want to retireve that values of column FRQ_Store that never matched any of those Campaign_Store ! In other words, how can I retrieve all the value of two columns that never matched in a full join query ?

Thank you!


texts: How mean and deviation come out with MNIST dataset?

texts:
I am a novice at the data science, and I notice some repository state the mean value and deviation in MNIST dataset are 0.1307 and 0.3081.

I cannot imagine how these two numbers come from. Based on my understanding, the MNIST dataset has 60,000 pics and each of them has (28 * 28 = 784) features. How do I convert this feature vectors to get the mean and deviation?

Especially, this should classify by the label, right? For example, the number 0 should have its mean and deviation. For number 1 should also have its mean and deviation.


texts: Can we consider high correlation to be a good predictor?

texts:
The problem of predicting the daily number of COVID-19 cases is indeed challenging and many (external) factors should be taken into account to come up with a reasonable predictor. However, we have studied Twitter for a specific country (not English) for the period Mar-Nov 2020, and found out that the volume of daily tweets related to symptom X is highly correlated with the number of confirmed cases in that country (pearson correlation 0.84 with p-value 0.00031).
In the field of data science, would this suffice, at least partially, to say daily tweets of X is a good predictor for the number of COVID-19 cases?


texts: Is standardization needed before using scikit-learn SVM?

texts: I am using the SVM function provided by scikit-learn. I would like to know whether I need to perform standardization before fitting the model. As I know, LibSVM tends to require pre-processing the data. I am not sure whether scikit-learn automatically normalizes the data instead of expecting us to handle it ourselves.


texts: How to build in symmetry of inputs into a Deep Neural Network?

texts:
I have a Deep Neural Network that takes $n$ inputs $X = [X_1, \ldots, X_n]^T$ and gives $n$ ouputs $Y = [Y_1, \ldots, Y_n]^T$. Normally, I can just do a standard deep neural network with a few fully connected hidden layers. However, I want to build into the network the fact that there should be permutational symmetry. That is to say, if $\pi(\cdot)$ is a particular permutation, then $\pi(X)$ should output $\pi(Y)$. So each input should be treated symmetrically in some sense.

A simple example would be I get $n$ IID sensor readings, and I want to build an auto-encoder for these sensor readings.

Is there some way of building this symmetry into the neural network, perhaps through some parameter sharing or a special architecture?


texts:
I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID.

For each ID, I have a number of words and some metadata.

Let's say my data have three metadata/criteria that need to be balanced (well... kind of, I want to be able to choose the percentage for some criteria values) :

The name of the speaker, each speaker should have approximately the same number of words.
The status of the speaker (active if the speaker is still in activity or inactive if not), where I want approximately 70 % active speakers and 30 % inactive.
The gender of the speaker, where I want approximately 50 % male and 50 % female.

Example : ID12345 with 10,000 words. Metadata : John, male and active

If I balance one of the criteria, then another, I will surely unbalance the previously balanced one.
Moreover, if I balance using only one criterion at a time, I could throw away data that could be useful to the balance of the following criteria, which mean I will end up with less data that I could have had optimally.

Is there an algorithm that could balance the data using all criteria with their percentage target, and maximise the number of words remaining ?

I'm looking for a general algorithm, with a variable number of criteria.

EDIT : I use C++ and MongoDB (but I'll accept other languages/tools if it can help me, as well as algorithms).


texts: Perform clustering from a similarity matrix

texts:
I have a list of songs for each of which I have extracted a feature vector. I calculated a similarity score between each vector and stored this in a similarity matrix. I would like to cluster the songs based on this similarity matrix to attempt to identify clusters or sort of genres.
I have used the networkx package to create a force-directed graph from the similarity matrix, using the spring layout. Then I used KMeans clustering on the position of the nodes from that graph and this resulted in clusters that made sense. However, I'm not sure that this is the correct approach as it is fundamentally linked with the positions given by the spring layout.
I have also attempted to run Spectral Clustering on the similarity matrix, however, it is too slow.
Is using the positions derived from a graph generated from the similarity matrix, and then using the spring layout piped into KMeans to extract clusters, fundamentally flawed? If so, what could be a potential alternative way to cluster elements given a similarity matrix?


texts: I installed and configured CUDA and cuDNN, but still can't run my program with my GPU

texts:
The reason I ask this question in the Data Science part is mostly Data Scientists are facing with Hardware Acceleration problems.
I installed CUDA and cuDNN, also made a lot of configuration to recognize my GPU and finally did it. I can confirm it with this code below
import tensorflow
import keras

from tensorflow.python.client import device_lib

print(device_lib.list_local_devices())

and get an output as:
Created TensorFlow device (/device:GPU:0 with 2787 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1)
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 6475366838755771989
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 2922866279
locality {
  bus_id: 1
  links {
  }
}
incarnation: 17383900152389168062
physical_device_desc: "device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1"
]

Also I tried some other confirmation codes as code belows:
import tensorflow as tf

print("NUM GPUs available: ", len(tf.config.experimental.list_physical_devices("GPU")))

and got a result
NUM GPUs available: 1

Lastly I tried to confirm this code below, if my tensorflow recognizes my GPU:
import tensorflow as tf
Result = tf.test.is_gpu_available(
    cuda_only=False,
    min_cuda_compute_capability=None
)
print(Result)

and output is:
True

But, when I run my ML code to train the dataset, I checked the System Properties and realized when I run the code CPU has been using and my GPU is still around using 2-3% but CPU is around 40%. Also, I want to mention about all the files about dynamic libraries while executing the tensorflow are successfully opened and I have only (I) Ignore sign on the outputs.
I want to enable my GPU as a hardware accelerator on my editor. Couldn't find any resource for exactly this problem.
I tried to specify my hardware accelerator with
with tf.device("/GPU:0):

but still my processor was working on it.
I guess I should change one options on the editor/system OR I should write extra code to run it on my editor.
I am using Visual Studio Code as an editor.
Thanks in advance.