texts: How to setup a home-laptop cluster to 'practice' elasticsearch, hadoop, mesos and spark

texts:
I'd like to have elasticsearch, hadoop, mesos and spark on my home laptop (it's an 8GB with an intel i7 CPU) for practicing the setup of such frameworks and eventually to test their functions.

I'm not interested in performances, but only to practice their settigs/functionalities.

I wonder which is the easiest solution to make them work on a cluster of a few nodes (can 3 nodes be enough?). I hope it's possible to create a cluster of nodes which can be used for all the frameworks.

I've seen some tutorials on creating clusters which involve for example virtual machines or dockers.

Particularly, docker-swarm project seems interesting for my task: is this a good way to start? Or there already-built solutions for those frameworks which don't require the manual creation of a cluster?


texts:
We are trying to run J48 on a classified data set. Our class attribute has two possible values ( 0,1) when running J48 the tree terminates at the very first node and doesnt process any further.

Instead of considering (0- false) as the starting point of J48. How can we consider running J48 by selecting (1-true) as the starting point of the tree?

Any suggestion will be greatly appreciated.


texts: Pearson correlation coefficient - is correlation estimator acceptable?

texts: As far as I know when it comes to theory, we use Pearson correlation when we want to check the correlation between two variables, which are both continuous or discrete. For a mixed case it's not so easy to use it to compute correlation  coefficient. On the other hand, we have Pearson correlation estimators, where we can calculate mixed case without any problems (based on samples). Does the Pearson correlation coefficient give deceptive results in this case ?


texts: Subsampling the “right” amout of data to train an ML model

texts:
I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process to find the 'right' amount of data to retain from the dataset.
Of course I can create a script which automatically tried different thresholds but I was wondering if there is any principled way of doing this. It seems strange that nobody ever tried to create a proper solution since this seems to be a very common problem.
Some additional criteria:

I am using sub sampling on a stream of data so it would be better to find something that works in this setting
I would prefer to avoid training the classifier more than once since it takes some time
I appreciate theoretically justified approaches

Any suggestion or reference?


texts: How to know if a model is overfitting or underfitting by looking at graph

texts:
Just recently got my hands on tensorboard, but can you tell me what features should I look for in the graph (Accuracy and Validation Accuracy)
And please do enlighten me about the concept of underfitting as well.


texts: Main options on how to deal with imbalanced data

texts:
As far as I can tell, broadly speaking, there are three ways of dealing with binary imbalanced datasets:

Option 1:

Create k-fold Cross-Validation samples randomly (or even better create k-fold samples using Stratified k-fold: https://scikit-learn.org/0.16/modules/generated/sklearn.cross_validation.StratifiedKFold.html ).
For each fold apply a resampling technique (upsampling, downsampling or a combination of both) separately on the "training" and "test" sets.
Use a "traditional" metric for evaluation: for instance the AUC of the ROC curve (TP Rate vs FP Rate).

Option 2:

Create k-fold Cross-Validation samples randomly (or even better create k-fold samples using Stratified k-fold).
Do not apply any resampling technique.
Use an "alternative" metric for evaluation: for instance the AUC of the Precision-Recall curve or something like the F-score (the harmonic mean of Precision and Recall).

Option 3:

Use something like XGBoost and tune the scale_pos_weight ( https://xgboost.readthedocs.io/en/latest/tutorials/param_tuning.html ).
Create k-fold Cross-Validation samples randomly (or even better create k-fold samples using Stratified k-fold).
Use a "traditional" metric for evaluation: for instance the AUC of the ROC curve (TP Rate vs FP Rate).

My main question is if I correctly interpret what the options are. Is there any conceptual mistake in what I'm saying? Is it appropriate to use Stratified k-fold in the three cases when dealing with imbalance? Is it not necessary to apply any resampling when using XGBoost and tuning scale_pos_weight? When some resampling is applied (Options 1 and 3) does it make sense to use a "traditional" metric and does not make sense to use an "alternative" metric? In general, the resampling has to be applied separately on training and test sets? Etc.

Also, it would be nice if you have any good reference to SMOTE and ROSE, regarding how they work, how to apply them and how to use them with python.


texts: In what stage should you plot validation and learning curve

texts:
So lets assume I have a binary classification problem and I started out with a logistic regression model. I quickly evaluate the models accuracy (lets assume we don't have a skewed dataset). After this I start to implement other models to compare, I begin with checking the accuracy score and plot a ROC curve to evaluate AOC score.
Lets say that I carry on with my logistic regression model. I wan't to fine-tune it and use a grid search algorithm and from there carry on choosing the best hyperparamters w.r.t the accuracy score.
When do I actually plot the learning and validation curve? I pressume that I should plot it next to see how a larger set of paramters affect the score and then to see if my model is underfitting or overfitting.
But shouldn't I plot the learning curve much earlier for example in the first step where I choose to start with a logistic reg. model?
To sum it up: In what stage is it appropiate to plot a learning curve?


texts: Build algorithm for Price Prediction/Classification

texts:
lets say I have historical data for prices and some additional information like article, location and maybe text like "higher price in this area due to less competition". Important is, that prices are made by experts who exactly know the town and insider infos like competitors prices and so on. A lot of data rows but only a few features, like:

123/london road/item x/text abc
456/new york   /item y/null

Is something like this little information useful?
Ideas that comes to my mind:
-Regression to predict price (additional info: how prices rise the last years)
-Clustering (Maybe you find price cluster due to the locations)
-Use the magic of neuronal networks predict prices with this technic and use the additional input for sales people and let the neuronal network learn also from gathering this new data.

But I am not sure if a use case with such few features makes sense, maybe someone else had the same Problems and maybe find a way to add additional features I did not think due to similar use cases.


texts: How to compute the size of a significant sample for manual validation in classification?

texts:
I am currently working on a product classification business case.

I have over 2 millions products from one source to classify in 23 categories. The input is the name of the product(ex:'Nike Air Max'). These products are already labeled. Whereas I can split my dataset into training and test set to compute the accuracy (which is 90%), I would like to predict categories of a whole new dataset, coming from an other source.

Once it will be done, I need to check products one by one to know if my classifier works well on this new dataset because it is not labeled and I really need to know the accuracy of prediction for this new dataset.

How can I know, the size of the sample coming from the new dataset (which has about 500k products) I need to check to be sure that my accuracy will have a small error margin?


texts: implementing forward and backward of a Linear model

texts:
I'm implementing the code of this abstraction.
The forward is easy and looks like that:

I don't understand the backward path and how it fit's the abstraction in the first image:

Why is db defined as multiplication of ones of x's shape and dout ?
Why is dw defined as multiplication of ones of x.T and dout ?
Why both of them are accumulated. i.e it is used += and not = ?
Why is dw defined as multiplication of ones of dout and w.T ?


texts: Regression Trees - Splitting and decision rules

texts:
I understand that a regression tree is built by splitting a node, such that the MSE for the label/output variable is minimized in each of the two resulting nodes.
I have two questions about this:

1.)
Is the search for the optimal split even dependent from the input variables?  The MSE could be minimized through exhaustive search for two subsets which  minimize the MSEs of the label in each of the subsets. No knowledge about input variables is needed for this.
If this is the case, how are the decision rules for upcoming instances, for which an output should be predicted, set?
How is it decided which feature to split at what point in order to obtain the split into the 2 subsets?

2.)
Or does the algorithm run through all possible splits (split at each value of each feature once) and then chose the one with the minimal MSEs?
This way the decision rule would be clear.

Thanks in advance


texts:
On the y-axis you've got RMSE and on the x-axis you've got the number of epochs. Then in blue, the validation error, in red the training error.

What do you think is the optimal number of epochs before the model starts overfitting? I would go with 74 epochs.


texts:
I am trying to build a simple linear classifier. I have two classes A and B each with two features [x, y] and hence a 2d dataset. Now, I need to find the equation of the line that separates the two classes.

y = $\vec w \vec x - b$

I know exactly what $\vec w$ is. $\vec w$ is vector that is perpendicular to the line that separates the two classes, and b is the bias that shifts the line along $\vec w$. When b is 0, line is exactly at the origin.

I created the algorithm exactly as mentioned in this link.
https://en.wikipedia.org/wiki/Perceptron#Learning_algorithm

But, in some cases it doesn't work properly. I've understood intuitively what the algorithm does. At each step, it takes a misclassified point, and tries to make correct classification by rotating the current line segment, and shifting it so that point is more closer to correct class.

I need an algorithm that helps me find $ \vec w $ and $ b $.

Here's my algorithm in ES6.

const learn = (target, learning_rate = .00001) => {
const { featureVector } = target;
// shift towards object
// const nm = math.norm(lineConfig.normal);
// lineConfig.normal = math.divide(lineConfig.normal, nm);
// lineConfig.bias /= nm;
const observedFeatureVector = featureVector.concat(1);
const wc = lineConfig.normal.concat(-lineConfig.bias);
const boundaryValue = math.dot(wc, observedFeatureVector);
const curOutput = boundaryValue >= 0 ? 1 : -1;
// lineConfig.bias += learning_rate * -target.cls;
// [w -b] [x, 1]
const dv = math.multiply(observedFeatureVector, learning_rate * (target.cls - curOutput));
const w = math.add(wc, dv);
lineConfig.bias = -w.pop();
lineConfig.normal = w;

};

I transform my input 2 dimensional data set into 3d by adding 1 [x 1] is our input vector. Then, all we need to do is to find $ \vec w $ only since bias is automatically adjusted. For every misclassification, I add the training point vector to the $ \vec w $ so that classification moves closer to the goal.

My training set is this:

[200, 200, 1]
[227, 347, 1]
[399, 237, 1]
[358, 344, 1]
[265, 263, 1]
[60, 20, -1]
[26, 100, -1]
[126, 95, -1]
[145, 58, -1]
[51, 141, -1]
[49, 186, -1]


texts: What criteria can be used to decide whether to develop a decision tree or one or more rules using oneR or jRip(contained in RWeka). The similarities between these two algorithms (decision tree and rule) is very high making it a little confusing.


texts: Keras: ValueError: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2

texts:
I am using Keras functional API to write an LSTM model but It throwing an error can somebody please help below is the code for the model the output shape is 65. I am using Keras 2.2.4 and TensorFlow 1.15
event_num = 65
droprate = 0.3
vector_size = 572

def DNN():
    train_input=Input(shape=(vector_size*2,),name='Inputlayer')
    train_in=LSTM(512)(train_input)
    # train_in=BatchNormalization()(train_in)
    train_in=Dropout(droprate)(train_in)
    train_in=LSTM(256)(train_in)
    # train_in=BatchNormalization()(train_in)
    train_in=Dropout(droprate)(train_in)
    train_in=Flatten()(train_in)
    train_in=Dense(event_num)(train_in)
    out=Activation('softmax')(train_in)
    model=Model(input=train_input,output=out)
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    return model


texts: Does cross_validate in scikit-learn automatically fits and train the model?

texts:
# from the titanic dataset
X = df.drop(columns="survived")
y = df.survived
scoring = ['accuracy','precision','roc_auc','f1',]
from sklearn.model_selection import cross_validate
from sklearn.linear_model import (LogisticRegression)
def model_LR(): #logstic Regression
    index = ["kfold-1","kfold-2","kfold-3","kfold-4","kfold-5"]
    s = model_selection.cross_validate(LogisticRegression(), X, y, scoring = scoring, cv = 5 )
    s = pd.DataFrame(data = s, index = index)
    display (s)
    print ("The mean scores for the above:\n", s.mean())

model_LR()

# OUTPUT :
    fit_time    score_time  test_accuracy   test_precision  test_roc_auc    test_f1
kfold-1 0.003998    0.006969    0.774809    0.711340    0.823673    0.700508
kfold-2 0.003990    0.005005    0.820611    0.778947    0.856481    0.758974
kfold-3 0.003003    0.003989    0.774809    0.715789    0.796667    0.697436
kfold-4 0.002992    0.003992    0.767176    0.709677    0.841852    0.683938
kfold-5 0.001994    0.003989    0.819923    0.819277    0.877081    0.743169
The mean scores for the above:
 fit_time          0.003195
score_time        0.004789
test_accuracy     0.791466
test_precision    0.747006
test_roc_auc      0.839151
test_f1           0.716805
dtype: float64

My Question:
In my code above, by calling on the cross_validate with LogisticsRegression, I get scores (such as roc_auc) as if the model has been fitted and trained.
I did not use any fit or train function.
Does that mean that cross_validate does that automatically?
Thx.


texts: What's the right way to setup an image classifier by multiple params?

texts:
I'm very new to the data science and machine learning, so apologies for my ignorance.
What I'm trying to understand is how to setup an image classifier system (maybe based on CNN) which will classify my image by multiple params. Most of the examples I found are about classifying by single class, i.e. "cat", "dog", "horse", etc, but what I'd like to have is, for example, {"red", "dog", "tongue"}. Is there a simple way to do it? The best option would be to have a ready setup, so I can just change their test dataset with mine and see the right formatting. Thanks!

UPDATE:

Also please help me understand if it's a complicated task for an experienced machine learning engineer? What'd be the timing and cost given a dataset?


texts: System administration skills for data engineering

texts:
As a data scientist student, I will do an internship with a lot of data engineering tasks.

It would be good if I had some initiation skills in sys-admins.

This leads me to my questions:

Why should I have skills in sys-admins?
What introduction course would you suggest to me? (I know that their data are on AWS)

Thanks a lot !!


texts: Looking for similar items in a large data set

texts:
I have a large database of people and I want to show a small number of people who are similar to each person in the database. So if one of the people was Wolfgang Mozart I would want to show Beethoven, Haydn, Schubert etc based upon they are "similar" in that they are roughly contemporary with the same occupation (i.e. Composer). For each person, I have a short occupation which usually contains the nationality (e.g. Austrian composer, American actor, British politician) and dates of birth/death.

Is there a recognised algorithm or technique to do something like this? I don't have a training set as I haven't manually produced any examples of what I want to see, so I don't think machine learning is appropriate.

If I had to write something right now I'd probably just do a string comparison and order so the closest contemporaries (i.e. dates of death) come first.

But is there a better way?


texts: Guys I am trying to create a application to create training data

texts:
import os
import cv2
import matplotlib.pyplot as plt
from colorama import Fore
import random
import numpy as np
import pickle

training_data = []
def create_training_data():
path = os.getcwd()
path_dataset = os.path.join(path,'dataset')
categories = ['a','b']
for category in categories:
    path_categories = os.path.join(path_dataset,category)
    index = categories.index(category)
    for img in os.listdir(path_categories):
        try:
            path = os.path.join(path_categories,img)
            img_array = cv2.imread(path,cv2.IMREAD_GRAYSCALE)
            training_data.append([img_array,index])
        except Exception as e:
            print(Fore.RED+path)
    print(Fore.GREEN+f"imported all data from cateogry {category}")

#For every object in the training dataset
#training_data[element][data type*]
#0 - Stores the values of all images
#1 -  stores the values of all category

create_training_data()

print(training_data[1201][1])
plt.imshow(training_data[1201][0],cmap="gray")
rows,columns = training_data[0][0].shape
print(f"rows-{rows} cols-{columns}")
plt.show()
random.shuffle(training_data)
X = []
Y = []

for feature,label in training_data:
X.append(feature)
Y.append(label)

X = np.array(X).reshape(-1,rows,columns,1)

pickle_out = open(X.pickle,'wb')
pickle.dump(X,pickle_out)
pickle_out.close()

pickle_out = open(Y.pickle,'wb')
pickle.dump(Y,pickle_out)
pickle_out.close()

After I run this script I am getting an error:

Traceback (most recent call last):
  File "trainingdata.py", line 55, in
    pickle_out = open(X.pickle,'wb')
AttributeError: 'numpy.ndarray' object has no attribute 'pickle'


texts: How to do imbalanced classification in deep learning (tensorflow, RNN)?

texts:
I am trying to do binary classification of News Articles (Sports/Non-Sports) using recurrent neural net in tensorflow. The training data is highly skewed [Sports:Non-Sports::1:9].

I am using cross-entropy as my cost function, which treats both classes equally.

What are the ways by which user can penalise one class? Or is there any other cost function suitable for this purpose?


texts:
I came through this questions and I failed to find the right answer for it.

How can Clustering (Unsupervised Learning) be used to improve the accuracy of Linear Regression model (Supervised Learning)?

a- Creating different models for different cluster groups.
b- Creating an input feature for cluster ids as an ordinal variable.
c- Creating an input feature for cluster centroids as a continuous variable.
d- Creating an input feature for cluster size as a continuous variable.


texts: What is the best approach to extract keys/values from documents?

texts:
I am thinking of training a model to automatically extract information from more or less structured documents like invoices.
Here are the main challenges regarding this task:

In fact, even though invoices are often called "structured" documents, there is a lot of variance in their layouts depending on the field, company, and other factors, which makes it almost impossible to achieve great results with pattern matching.

When relying on text, the most straightforward solution to get textual information from documents is using an OCR engine like Tesseract or EasyOCR (or, probably, commercial solutions from Google, Amazon, Microsoft). Unfortunately, more often than not, outputs from OCR libraries are rather dirty, which can be explained by the quality of scans.

Some fields can appear in a document multiple times. This is especially true for line items

Some values might belong to several classes (entities) at the same time. For example, extracting addresses, company names is definitely not a one-to-one mapping in that several companies are mentioned with the corresponding addresses and other types of information.

That said, what could you recommend as a possible approach to this problem? I understand that given (1)-(4) there is no way I should expect a perfect solution, but at least something that would at least make sense to try.
My current thoughts:

This problem has a lot of similarity with traditional NER-like tasks, but those are most often trained on sentenced where there's a lot of semantical value and not just key/value pairs.

I am almost 100% certain that without some rules, even some hardcoded ones, it will be extremely difficult to get something useful. One such idea is to use a pre-trained NER model, extract standard entities like MONEY, DATE, COMPANY, etc, and then just look in the neighborhood of these entities (in terms of coordinates, I mean) to check whether phrases like Total amount or Invoice date are nearby.

Probably, it makes sense to use both bounding boxes and text values for labeling and further model building. On the other hand, it would be quite challenging to do that labeling because some entities might change their location from document to document, span over several lines, etc.

What should I read/try to have a better understanding of how to approach this problem?
Thanks in advance!


texts: Separate business emails from personal emails

texts: I am working on a classification problem where I would like to separate business emails from personal emails to analyse their behaviours separately. I am thinking about using regex but after looking at the emails, I realised that the email addresses vary significantly. Although I could manually spot some emails with obvious business names, such as xxx@carsales.com.au, it is hard to filter these emails systematically. Are there some useful techniques I could use? Thx


texts:
I am attempting a classification project. I have split my data, 20000ish, into training and test sets. On the training sets I run a selection of classifieds including knn random forests and gbm . These give me about 20-30% accuracy at best. For each sample I generate probabities of each class and make a new model

Knn proba 1, knn proba 2.... Random forests proba 1 etc

On this I then run a random forest classifier which gives me a 90ish% accuracy against the test set.

Fantastic!... But when I use the model against new data the accuracy is very poor.

In part it feels like a case of overfitting but surely the test set should be poor as well

Why might the test data so good but new data so poor... What have I done wrong?

Thanks
Chris


texts:
I'm using LIME to explain my random forest model. Everything is working great. However, I don't quite understand the image that is generated. Taking the example from the Readme:

How can it predict poisonous with 1.00 but still have a gill-size=broad which has 0.13 probability? Or do I misunderstand the table?

I already asked this question in the github project, but didn't get any answer.


texts: How does Tensorflow compute gradients of reduce_min operation?

texts:
With a non-differentiable operation, such as a minimization, how does Tensorflow compute the gradients? Some kind of soft-minimum approximation?
If so, can I retrieve the analytical computation for a specific gradient?


texts: Dataset with disproportionately more of a single label than any other

texts: I'm using the data analysis software Orange to analyze rows of data with labels $\{H, T,L\}$. $T$ is the neutral state of the system I'm trying to model so data is almost always labeled with $T$.  This leads to an extremely high $97$% classification accuracy, that is untrue.  Almost no datapoint with an $H, L$ actually gets labeled with an $H, L$ respectively by the neural net.  The neural net is doing what's logical: flatten everything to $T$ since that occurs most of the time.  How can I repair this situation?


texts:
I am working on a dataset that has a dependent variable that is binary, but it contains 98% of 0's and 2% of 1's. I am trying to use Logistic regression to predict purchase of a product. But because of the huge number of 0's, the model is not predicting well and getting a large number of false positive result.

Kindly suggest how can I approach this.


texts: How to combine different kernels for Gaussian process in GPyTorch?

texts:
I am trying to learn gaussian process by using GPyTorch to fit a Gaussian Process Regression model. However, I can't figure out a way to combine different kernels as shown in sklearn implementation of gaussian process. I am using GPyTorch as it is more flexible and have lot more kernels that one can play with compared to scikit-learn.
Any help, particularly with code snippet would be very useful.

Thank you.


texts: How to repeat classification experiment with different set of data by loop

texts:
I want to repeat my experiment pipeline to validate my feature extraction. My goal is with different set of data which randomly picked as train test and validation, I want to check if the result are consistent with same classifier method (ex: SVM).
Let say i have list of ECG signal record which has id label for two class (class a = [records 1,2,3,4,5,6,7,8,9,10] and class b [records 11,12,13,14,15,16,17,18,19,20]) which i have set for train and valid and testing proportion of each record is 50%:20%:30% and both a and b are randomly shuffled per my experiment test. For example:
Train = a[1,3,5,10,9],b[11,20,19,17,16]
Valid = a[2,4] b [14,15]
Test = a[6,7,8] b[12,13,18]
For each row in class a, beside record_name, it has time_id (30 to 1). The goal is i want to loop through the time_id to validate and test the feature i used. But in class b, since it is a normal ECG signal, it is only one row per record, so the same class b rows will be used if the time is changed in test or validation process.
This is the method to validate and test (I use classification_result from sklearn):
t = 30
for tid in range (t):
    a_data = a.loc[a["MINUTE"]==t]
    test = [a_data,b_data]
    test_data = concat(test).values
    Y_true = test_data.pop("CLASS").values
    Y_pred = model.predict(test_data)
    print(classification_result(Y_pred,Y_true)
    tid = tid+1

Then as usual i do training validation and testing from the set of data
to make it easier, i try to loop my process which look like this:
n_exp = 100
for k in range(n_exp):
     a_train,a_valid,a_test = <logic to shuffle list of class A>
     b_train,b_valid,b_test = <logic to shuffle list of class A>
     a_set_train= A.loc[A['NAME']isin(a_train)]
     ...
     b_set_test= B.loc[B['NAME']isin(b_test)]
     train_set = [a_set_train,b_set_train]
     train = concat(train_set)
     model = SVM()
     model.fit(train)
     test_method(validation_data)
     test_method(test_data)

But when i try to run these sequences, it always give the same result for each experiment even with different train valid and test records. Is there any proper way to repeat these experiments using loop, since if I hardcoded or rerun the program will consume more time.


texts: Is parsing files an application of machine learning?

texts:
I presently receive files from a device in a semi-csv format. I have a written a simple recursive descent parser for getting information out of these files. Every time the device updates firmware, I have a new version of the parser for the changes the update brings.

Down the road, we will be taking data from other devices, which means another parser and more updates to firmware. I'm wondering if I could define a basic structure of "this is the data I need" and use a neural network to get the parsed data without having to write a parser for each new file type that comes in.

Is this a pipe dream or is it a valid application of machine learning? I'm much more of a software engineer than I am a data scientist, but I'm starting to dip my toes into the machine learning realm.

Thanks in advance.


texts: Feature Selection for K Nearest Neighbour and Decision Trees

texts:
I have 2 digits numbers and 9 features.

I must pick 2 features, so decided to plot the features against each other to see whether I can get any insight on the best features to train my algorithm.

The plot colours indicates the two digits.

Algorithms I considered to use were, K-Nearest Neighbour and Decision Tree.
I am very new to machine learning, I chose these two algorithms simply because I have come across them.

Feature matrix of f1 to f9 against f1 to f9

Decision Tree decision boundary

I have a few questions:

Will choosing the feature x against feature y with the least amount of overlap will help achieve the optimal decision boundary?

When I look at features should I initially consider linear data separation.
Then work my way up to use an algorithm that can deal with non-linear separated feature points?

What important visual properties should I look out for when choosing optimal features for training?

How can I visualise the tree in sklearn python?

Thanks.


texts: Studying machine learning algorithms: depth of understanding vs. number of algorithms

texts:
Recently I was introduced to the field of Data Science (its been 6 months approx), and Ii started the journey with Machine Learning Course by Andrew Ng and post that started working on the Data Science Specialization by JHU.

On practical application front, I have been working on building a predictive model that would predict attrition. So far I have used glm, bayesglm, rf in an effort to learn and apply these methods, but I find a lot of gap in my understanding of these algorithms.

My basic dilemma is:

Whether I should focus more on learning the intricacies of a few algorithms or should I use the approach of knowing a lot of them as and when and as much as required?

Please guide me in the right direction, maybe by suggesting books or articles or anything that you think would help.

I would be grateful if you would reply with an idea of guiding someone who has just started his career in the field of Data Science, and wants to be a person who solves practical issues for the business world.

I would read (as many as possible) resources (books,articles) suggested in this post and would provide a personal feed back on the pros and cons of the same so as to make this a helpful post for people who come across a similar question in future,and i think it would be great if people suggesting these books can do the same.


texts:
What are the correct and common ways to normalize image for CNN?
I used to work with text and it was pretty straightforward. Removing stop words, clean text from noise, tokenization, stemming etc. There is no problem with length we can easly add padding. I am not sure how to treat images.
Grey scale is much better, less data as input.
Should I rotate each image for random angle? How to pick right size of image in my traing dataset? What about adding additional filters?

(my goal is to classify 2 classes with tensorflow)


texts: Principle behind seq2seq model's example in keras?

texts:
I am referring to seq2seq model's example code in keras (https://github.com/fchollet/keras/blob/master/examples/addition_rnn.py). The model is :

model = Sequential()

model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))

model.add(RepeatVector(DIGITS + 1))
for _ in range(LAYERS):
    model.add(RNN(HIDDEN_SIZE, return_sequences=True))

model.add(TimeDistributed(Dense(len(chars))))
model.add(Activation('softmax'))

In this model we are passing the encoded input vector from encoder's last state to each time step in the decoder.

Now we are not passing any other input to the decoder except the encoded input vector, but in all seq2seq models we pass output sequence also (time delayed) with the encoded input.

How is this a valid seq2seq model? To my surprise it works well. How does this works?


texts: How to predict numerical value "y"? if I have datasets having different number of parameters

texts:
I have three excel sheets for training (all numerical values).

1st excel sheets have parameters a, b, c and output as y
2nd excel sheet have parameters a, b, c, d, e and output as y
3rd excel sheet have parameters a, b, c, d and output as y

I have to predict value y if values of only a, b, c are given.

Should I apply linear regression?


texts: Label spreading for classification/clustering problems

texts:
I have a question regarding label propagation and label spreading semi-supervised algorithms. I am working on building a look-alike model to identify marketing personas. Using supervised learning algorithms is getting quite complicated as it takes a lot of time to run. And using unsupervised learning is quite complicated as well, as we need to specify "k". I am unsure of how to automate this process for new datasets. I need a middle ground machine learning algorithm suggestion that has lower time complexity and is reliable for new datasets.
So my question is: Can I use a graph-based semi-supervised machine learning technique to classify my dataset into personas? Specifically, label propagation and/or label spreading?
Please let me know your thoughts and any other suggestions that you may have. Thanks in advance!


texts:
I want to prepare the following chart. Say, I have three datasets of different lengths, for example:

A = [c,d,e,g,h,o,m,n,o,x,y,z]
B = [d,e,f,g,h,i,j,k,l]
C = [a,b,c,d,e]

Any dataset might contain items already there in other 2 datasets. Now I want to show the size of each dataset as a circle. More items, bigger circle. But all three circles also overlap each other depending on how much items are common. As demonstrated below.

What this chart is actually called? And what library can I use which might have this kind of chart available?


texts: CNN always predicts either 0 or 1 for binary classification

texts:
I am using a Kaggle dataset on stress characteristics, derived from ECG signals, and I would like to train a CNN to recognize stress/non-stress situations.

I have built a model in Keras:

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(U1, (3, 1), activation = 'relu', input_shape = (num_features, 1, 1)),
    tf.keras.layers.Conv2D(U2, (3, 1), activation = 'relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(U3, activation = 'relu'),
    tf.keras.layers.Dense(1,  activation = 'sigmoid')
])

where U1, U2 and U3 are the parameters I have been changing to find the right combination to ensure the best performance.

What I did is, specifically:

divide the samples in training and test set (I don't have a validation set as the number of available samples is small);
normalize both training and validation set by dividing them by the max value found in the samples;
train the network with various combinations of U1, U2 and U3 to find the ones ensuring the best performance.

The training is done as follows:

model.compile(loss = 'binary_crossentropy' , optimizer = 'adam' , metrics = ['accuracy'])
history = model.fit(x_train, y_train,
          epochs = 50,
          batch_size = 160,
          validation_data = (x_test, y_test))
score = model.evaluate(x_test, y_test, batch_size=128)

The network performs really well on both the training and test set, achieving an accuracy of 99.3% on test set (and 99.5% on training set).

However, when applied on real data (by taking one's ECG, computing the features and normalizing them by the same normalization value used on training and test set above), the network is always predicting:

a label of 0.0 for "normal" ECGs;
a label of 1.0 for noisy ECGs (which are taken as stressed ECGs).

It bugs me that no other labels rather than 0.0 or 1.0 are never returned. It is true that sometimes the network predicts labels of exactly 0.0 or exactly 1.0 also for samples in the training and validation set, but never seeing a label different from 0.0 and 1.0 in real-world data sounds strange.

Is this a problem of the network? Or a problem of the dataset? Maybe could this be related to the fact that the real-world ECG data I am using are not extracted from the same distribution of the Kaggle dataset? I see that, for instance, real-world data have profound differences in values with respect to the data in the Kaggle dataset (so much difference that even after normalization values are not really normalized), but I don't know if this is a valid reason that justifies the problem I see.


texts: how to get prediction from trained random forest model?

texts:
i have a dataset with two columns user posts (posts) and personality type (type) , i need personality type according to posts using this dataset so i used random forest regression for prediction
here is my code:-

df = pd.read_csv('personality_types.csv')

count_vectorizer = CountVectorizer(decode_error='ignore')
X = count_vectorizer.fit_transform(df['posts'])
y = df['type'].values

Xtrain, Xtest, Ytrain, Ytest = train_test_split(X, y, test_size=0.33)

random_forest = RandomForestClassifier(n_estimators=100)
random_forest.fit(Xtrain, Ytrain)
Y_prediction = random_forest.predict(Xtest)

accuracy:

random_forest.score(Xtrain, Ytrain)
acc_random_forest = round(random_forest.score(Xtrain, Ytrain) * 100, 2)
print(round(acc_random_forest,2,), "%")

100%

now i want to get prediction from a custom text how can i achive that ?
how can i get personality type of a post separately using this model.


texts: Tensorflow: How to load/restore a FULL model (.pb, .pbtxt, .graph, .ckpt, variables, events)?

texts:
I recently switched from PyTorch to TF (1 and 2) and I am trying to acquire a good workflow with it.
The simple things I want to do are the following:

Load a full pretrained object detection model from TF1 zoo or TF2 zoo
Use model.summary() to inspect the network architecture of the loaded model.
Do inference with a pretrained loaded model.
Modify (e.g. reshape, drop, add) the layers and weights of the loaded model.
Retraining the modified loaded model.

I know that TF has the concept of graph and weights as opposed to PyTorch which just have models encompassing everything.
Nevertheless I could not find an easy and best way to load a pretraind model and the internet is full of different answers for different tf versions.
I am really confused because to achieve the points above I have so many different files available when I download a pretrained model from TF1 zoo (or TF2 zoo).
Take this one for instance, the first in the list of the TF1 zoo.
I have the saved_model folder with the saved_model.pb and the variables (empty) folder, the frozen_inference_graph.pb the model.ckpt files, the pipeline.config and in some cases an event file.
Are all these different files really necessary to encode graph structure and weights?Am I missing something or this is just more complicated than necessary?
In addition, the file/folder structure is different iF you download a model from TF2 zoo (see image below)

What I tried
import tensorflow as tf #(v2.4)
def load_pretrained_model(self, saved_model_sub_folder,
                          mode):
    # 1. this only load an AutoTrackable object that can be use for inference but no graph
    if mode == '.pb':
        model_dir = str(TRAINED_MODEL_DIR) + saved_model_sub_folder
        model_dir = pathlib.Path(model_dir) / "saved_model"
        model = tf.saved_model.load(str(model_dir), None, '.')
        detection_model = model.signatures['serving_default']

    # 2. this returns None
    elif mode == '.graph':
        def load_graph(frozen_graph_filename):
            with tf.compat.v1.gfile.GFile(frozen_graph_filename, "rb") as f:
                graph_def = tf.compat.v1.GraphDef()
                graph_def.ParseFromString(f.read())
                return graph_def
        detection_model = tf.compat.v1.import_graph_def(load_graph(frozen_graph_filename))
    else:
        detection_model = None

    return detection_model

Tl;dr
Can someone answer some of the points (1 to 5) above regarding how to load a complete (graph, weights, everything..) customizable tensorflow1 or tensorflow2 model in python3?


texts: How do I choose the optimal parameters for reliefF

texts:
For feature selection I use reliefF provided by matlab.
The reliefF function offers a parameter k to influence its ouput, additionaly for my specific task I can also vary a window length l on which the features are computed. ( I am classifying sensor signals )
My question is how do I choose the optimal parameters k and l ?

Especially how should I score the different parameter configurations ?
Can I use the weights outputed by reliefF ( average weights of all features ) ? I dont't want to use classification error based on the selected features, since then an other question arises : Which classifier should be used, and how are its hyperparameters selected ?


texts: GAN discriminator converging to one output

texts:
I am running a GAN on the MNIST dataset. As the GAN continues to train, the quality of the generator seems to get worse, and it even seems as if it is starting to converge to just one value.

Epoch 7:

Epoch 21:

Why is this happening? I am running the code from this repo here:
https://github.com/bstriner/keras-adversarial/blob/master/examples/example_gan.py


texts: Sentences language translation with neural network, with a simple layer structure (if possible sequential)

texts:
Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure:

  "the cat sat on the mat" -> [Seq2Seq model] -> "le chat etait assis sur le tapis"

Example: A ten-minute introduction to sequence-to-sequence learning in Keras, Python for NLP: Neural Machine Translation with Seq2Seq in Keras

I noticed that in all these examples the structure of the neural network is not done by using a Sequential structure with consecutive layers, but rather a more complex structure like this:

Question: are there successful attempts of doing sentence language translation with classic Sequential layers?

i.e.:

Input layer: word-tokenized sentence in english, zero padded: "the cat sat on the mat"
=> x = [2, 112, 198, 22, 2, 302, 0, 0, 0, 0, 0, 0, 0, 0, ...]
Output layer: word-tokenized sentence in french, zero padded: "le chat etait assis sur le tapis"
=> y = [2, 182, 17, 166, 21, 2, 302, 0, 0, 0, 0, 0, 0, 0, 0, ...]

What would you use as layers? I think we could start with:

model = Sequential()                                   # in  shape: (None, 200)
model.add(Embedding(max_words, 32, input_length=200))  # out shape: (None, 200, 32)
model.add(LSTM(100))                                   # out shape: (None, 100)
.... what here? ...

but then how to have a second Embedding for the output language and reverse it? from a 200x32 embedding (floats) to an integer list like this [2, 182, 17, 166, 21, 2, 302, 0, 0, 0, 0, 0, 0, 0, 0, ...]?

Also, how to measure the loss in this situation, mean squared error?

More generally, what is the simplest structure you can think of (even if it does not give the best results), working for language translation? (it's ok even if not sequential)


texts:
I have a hard time understanding the LSTM model performance as I summarize my model as follow:

X_train.shape
(120, 7, 11)
y_train.shape
(120,)
X_test.shape
(16, 7, 11)
y_test.shape
 (16,)

model = keras.Sequential()
model.add(keras.layers.LSTM(100, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences = True))
model.add(keras.layers.Dropout(rate = 0.2))
model.add(keras.layers.LSTM(20))
model.add(keras.layers.Dropout(rate = 0.2))
model.add(keras.layers.Dense(1))
model.compile(loss='mean_squared_error', optimizer=keras.optimizers.Adam(0.001), metrics = ['mae'])

history = model.fit(
X_train, y_train,
epochs=60,
batch_size=5,
verbose= 0,
validation_split = 0.1,
shuffle=False
)

Based on the below plots, both MSE and MAE decrease in the training process and their corresponding values are near zero.

However the prediction is not precise enough as I realize:

y_pred = model.predict(X_test)
model.evaluate(X_test,y_test)
[0.04673878103494644, 0.15574690699577332]

So my question is, how does really my model perform? I mean how can interpret its performance,since both MSE and MAE seem to be low but the prediction values are not quite convincing.


texts: How to explain a stable NDCG@K in extreme multilabel recommender model

texts:
I am working in a multilabel recommender project and I try to evaluate it as a ranking problem.

I calculate recall@k and precision@k which both looks quite well. Recall increases and Precision decreases as I try higher K values, which is expected.

However, the NDCG@K increases up to a certain K and after that it stays the same. How can we explain such a behaviour?


texts: Overcoming "too many indices for array" error when loading file into Orange

texts:
I am attempting to load the Stack Overflow 2017 developer survey results into Orange. The dataset has ~51.4K rows with 156 features. It is comma separated and is about 91MB in size.

When I attempt to load this file into Orange, I get the following error:

Cannot parse dataset H:/survey_results_public.csv: too many indices for array

How can I overcome this error?


texts: Getting constant accuracies for training and validation sets despite their losses are changing during CNN training?

texts:
As the title clearly describes the issue I've been experiencing during the training of my CNN model, the accuracies of training and validation sets are constant despite the losses of them are changing. I have included the detail regarding the model and its training setup below. What may cause this issue?
Here is the data that was used by training (X_train & y_train), validation, and test sets (X_test and y_test):
df = pd.read_csv(CSV_PATH, sep=',', header=None)
print(f'Shape of all data: {df.shape}')

y = df.iloc[:, -1].values
X = df.iloc[:, :-1].values

encoder = LabelEncoder()
encoder.fit(y)
encoded_Y = encoder.transform(y)
dummy_y = to_categorical(encoded_Y)

X_train, X_test, y_train, y_test = train_test_split(X, dummy_y, test_size=0.3, random_state=RANDOM_STATE)

X_train = X_train.reshape((X_train.shape[0], X_train.shape[1], 1))
X_test = X_test.reshape((X_test.shape[0], X_test.shape[1], 1))

Here are the shapes of training and test sets:
Shape of X_train: (1322, 10800, 1)
Shape of Y_train: (1322, 3)
Shape of X_test: (567, 10800, 1)
Shape of y_test: (567, 3)

Here is my CNN model:
# Model hyper-parameters
activation_fn = 'relu'
n_lr = 1e-4
weight_decay = 1e-4
batch_size = 64
num_epochs = 200*10*10
num_classes = 3
n_dropout = 0.6
n_momentum = 0.5
n_kernel = 5
n_reg = 1e-5

# the sequential model
model = Sequential()
model.add(Conv1D(128, n_kernel, input_shape=(10800, 1)))
model.add(BatchNormalization())
model.add(Activation(activation_fn))
model.add(MaxPooling1D(pool_size=2, strides=2))

model.add(Dropout(n_dropout))

model.add(Conv1D(256, n_kernel))
model.add(BatchNormalization())
model.add(Activation(activation_fn))
model.add(MaxPooling1D(pool_size=2, strides=2))

model.add(Dropout(n_dropout))

model.add(GlobalAveragePooling1D()) # have tried model.add(Flatten()) as well

model.add(Dense(256, activation=activation_fn))
model.add(Dropout(n_dropout))
model.add(Dense(64, activation=activation_fn))
model.add(Dropout(n_dropout))
model.add(Dense(num_classes, activation='softmax'))

adam = Adam(lr=n_lr, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=weight_decay)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['acc'])

Here is how I have evaluated the model:
Y_pred = model.predict(X_test, verbose=0)
y_pred = np.argmax(Y_pred, axis=1)
y_test_int = np.argmax(y_test, axis=1)

And, my model always predicts the same class of three classes during the model evaluation as you can see from the classification result below (via classification_result(y_test_int, y_pred) function):
          precision    recall  f1-score   support

  normal      0.743     1.000     0.852       421
     apb      0.000     0.000     0.000        45
     pvc      0.000     0.000     0.000       101

The model was trained using the EarlyStopping callback of Keras. So, the training has continued for 4,173 epochs. Here is the obtained losses during the training for training and validation sets:

Here is the obtained accuracies during the training for training and validation sets:

The model was implemented using Keras, and hosted on Google Colab. Please feel free to ask any further information regarding my model.


texts:
Getting a error when trying to import tensorflow library for tensorflow gpu help

>>> import tensorflow as tf
Traceback (most recent call last):
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: DLL load failed: The specified module could not be found.

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.


texts: how to extract substrings from a dataframe column

texts:
I have a dataset as below:

 col1

{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,jjjhjhd :70

I want to add 2 more cols such as col_50 & col 59 as below:

 col2                                        col3
 ghgjlj45llj,ljhjlhlhj,clause                jhjhhjxjhj65@,jjjhjhd

I tried for col 2 like below:

   def find_between( s, first, last ):
      try:
        start = s.index( first ) + len( first )
        end = s.index( last, start )
        return s[start:end]
     except ValueError:
        return ""
  df['col2']=find_between( df1['col1'].str, "50k:", ":" )
  df['col3']=find_between( df1['col1'].str, "59:", ":" )

Such that it extracts all characters between 50k: & untill next ':' appears, however not getting desired result. Can anyone please help?


texts: Julia - Identity matrix - eye() alternative

texts:
I'm working with what I guess is an older textbook that is using an older version of Julia as they are using the eye() function to create an identity matrix, which appears to not exist in the version I am currently using. What alternative is there for eye()?

Is there a


texts: Is there any detailed materials that can help explain how to set up HMM on R Studio?


texts: How to ignore vectors of zeros (i.e. paddings) in Keras?

texts:
I'm implementing a LSTM model with Keras. My dataset is composed by words and each word is an 837 long vector. I grouped the words in groups of 20 and to do this I padded them: initially I had groups of words of variable length and the maximum group length that I found was 20, this is why I padded all groups to 20.

For example, a group of 5 words is:

[[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837]]

where xi is the i-th feature of the vector.

To pad this group to a length of 20, I added 15 vectors composed by 837 feature with value equal to zeros:

[[0.......0],
............
............
[0........0]]

So, at the end, my group is of the form:

[[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[0...........0],
..............
..............
[0...........0]]

How could I ignore the vectors of zeros during training?


texts: How can I send the detected signs from YOLO to the classifier?

texts:
I have a YOLO model to detect the location of traffic signs and I have another model that determines what the sign is.

How can I send the detected signs from YOLO to the classifier then write the correct label above the bounding box, like the image below?


texts: How could PageRank be used to rank paragraphs in relevance to keywords?

texts:
I have a data set with keywords describing paragraphs in a car manual and the actual paragraphs. I want to rank those paragraphs by that keyword using PageRank algorithm.

How would I rank these paragraphs in relevance to the keywords?

The keywords are given an ID number while the paragraphs in the manual are texts. Let's say there is a keyword with ID number 89 and I have about 15 paragraphs that the keyword describes. How would the paragraphs be ranked in relevance to the keyword? Can I even use PageRank for this problem?

I understand for ranking websites, backlinks are used to sort the websites into ranks but for paragraphs of texts, how would I link and rank them?


texts: Attention model with seq2seq over sequence

texts:
On the official tensorflow page there is one exmple of a decoder (https://www.tensorflow.org/tutorials/text/nmt_with_attention#next_steps):

class Decoder(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):
    super(Decoder, self).__init__()
    self.batch_sz = batch_sz
    self.dec_units = dec_units
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(self.dec_units,
                                   return_sequences=True,
                                   return_state=True,
                                   recurrent_initializer='glorot_uniform')
    self.fc = tf.keras.layers.Dense(vocab_size)

    # used for attention
    self.attention = BahdanauAttention(self.dec_units)

  def call(self, x, hidden, enc_output):
    # enc_output shape == (batch_size, max_length, hidden_size)
    context_vector, attention_weights = self.attention(hidden, enc_output)

    # x shape after passing through embedding == (batch_size, 1, embedding_dim)
    x = self.embedding(x)

    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)

    # passing the concatenated vector to the GRU
    output, state = self.gru(x)

    # output shape == (batch_size * 1, hidden_size)
    output = tf.reshape(output, (-1, output.shape[2]))

    # output shape == (batch_size, vocab)
    x = self.fc(output)

    return x, state, attention_weights

The output here is of shape (batch_size, vocab). I would like to have an output with shape (batch_size, max_length, vocab), where max_length is the output sequence length. I suppose there can be something done with the TimeDistributed layer but I tried several things and nothing realy worked out. Is there any work around to obtain this?

One way would be:

self.attention = layers.TimeDristributed(BahdanauAttention(self.dec_units))

but since BahdanauAttention has two inputs I do not know how to fix it, since timedistributed layer does not work with multiple inputs easily.


texts: How to fix similarity matrix in pandas returning all NaNs?

texts:
I've been trying to create a similarity matrix in pandas from with a matrix multiplication operation on a document-term count matrix with 2264 rows and 20475 columns.
The calculation completes in IPython but inspection shows the results all come back as NaN.
I've also tried doing the same job in numpy, tried converting the original matrix to_sparse and even re-casting the values as integers, but still no joy.
Can anyone suggest the best approach to tackle the problem?
EDIT: Here's my code thus far:
path = "../../reuters.db"
%pylab inline
import pandas as pd
import numpy as np
import pandas.io.sql as psql
import sqlite3 as lite
con = lite.connect(path)
with con:
    sql = "SELECT * FROM Frequency"
    df = psql.frame_query(sql, con)
    print df.shape
df = df.rename(columns={"term":"term_id", "count":"count_id"})
pivoted = df.pivot('docid', 'term_id', 'count_id')
pivoted.to_sparse()
similarity_matrix = pivoted.dot(pivoted.T)


texts:
I am new to Data Science and have a problem with categorical variables.
My data set has 2 columns of strings - Departure City and Arrival City. It looks like this:

ID  | DepCity| ArrCity |Price|Time|SomeColumn   |ColToPredict|
1   |London  | Berlin  | 300 | 95 | 220         | 4          |
2   |Dublin  | Nice    | 420 |115 | 59          | 1          |
3   |Milan   | Brussels| 150 |108 | 154         | 3          |
4   |Paris   | Rome    | 160 |120 | 200         | 4          |
250 |Madrid  | Oslo    | 290 |300 | 110         | 2          |

So there are a lot of categorical variables in both columns and these columns are important (their values depend on other columns).

I use Python and sklearn. And it's not possible just to eliminate them as suggested in some tutorials.

I know there is a way to deal with categorical variables by creating new columns with zeros and ones for each variable. But I'm not sure that it's my case, because I have about 30 unique cities in each column. What will be the best way to convert categorical variables into numerical?


texts:
I have read in multiple articles that R-square always increases with the number of features, even though a feature may not be of any significance.
The formula for R-square is
$$1 - \frac{\sum(y-\hat{y})^2}{\sum(y-\bar{y}^2)}$$
If the denominator is constant that means R-square is dependent upon only numerator, so basically on $\hat{y}$.
$$\hat{y} = a +b_1x_1 +b_2x_2 \ldots$$
Now if I have a new feature which is really not important shouldn't the beta coefficient of that feature be zero? And if it is really zero how will it really impact R-square?


texts: Issue with IPython/Jupyter on Spark (Unrecognized alias)

texts:
I am working on setting up a set of VMs to experiment with Spark before I spend go out and spend money on building up a cluster with some hardware. Quick note: I am an academic with a background in applied machine learning and work quit a bit in data science. I use the tools for computing, rarely would I need to set them up.

I've created 3 VMs (1 master, 2 slaves) and installed Spark successfully. Everything appears to be working as it should. My problem lies in creating a Jupyter server that can be connected to from a browser not running on a machine on the cluster.

I've installed Jupyter notebook successfully... and it runs. I've added a new IPython profile connecting to a remote server with Spark.

now the problem

The command

$ ipython --profile=pyspark

runs fine and it connects to the spark cluster. However,

$ ipython notebook --profile=pyspark
[<stuff is here>] Unrecognized alias: "profile=pyspark", it will probably have no effect.

defaults to the default profile not the pyspark profile.

My notebook config for pyspark has:

c = get_config()
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8880
c.NotebookApp.server_extensions.append('ipyparallel.nbextension')
c.NotebookApp.password = u'some password is here'


texts: How to pick the most optimal set of association rules?

texts:
I'm currently running association rules on a dataset that has about 22000 rows, but there is no obvious threshold that the support value must have. So, I ran the rules three times with the following support and confidence values:

Support - 0.15, Confidence - 0.8 | largest Lift value was 1.8
Support - 0.08, Confidence - 0.8 | largest Lift value was 10
Support - 0.005, Confidence - 0.75 | largest Lift value was 190

Which one would be the most optimal?


texts: Why is activation needed at all in neural network?

texts:
I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know that by definition the activation function transforms the sum of w*x+b to a number between some lower and upper limit.

In the video Risto Siilasmaa explains that in the training process there are the following steps:

Start with random weights.
Calculate the outcome (sum of w*x+b) - we know what it should be because we know what image we gave to the system.
Calculate the error.
Nudge all weights to reduce the error.

But what about the activation? Where to place it in the previous list? Before the error calculation?
And what would happen if we omitted it altogether? Just calculate the outcome and error and nudge the weights? Is it because the error calculation doesn't work well when the value of outcome isn't transformed between some lower and upper limit?


texts: I am facing a weird problem in my on going project and thought if someone here could help me out with this. Actually I have large data set. I have to perform a regression task on top of that. While doing the initial analysis and feature selection task, I did correlation of all the features with the target variable. When I got the results I saw one of the feature having negative correlation with the target variable but my business unit suggests it must have a positive correlation to proceed further. Now they are asking me to filter out the rows causing negative correlation. I don't know how to proceed in this context. Can anybody help me out or give me heads up in this? Thank you very much in advance


texts:
Beginner here!

I have a dataset, with reviews of a product as text, ratings for the product.

My previous motive was to use Naive Bayes classifier for sentiment analysis. But my data doesn't have the variables( sentiment) required - negative/positive.

Shall I use the ratings (1-5) and encode it as positive and negative?
Or using the lexicon-based methods is more valid?


texts: Visualize Softmax values in CNN prediction

texts:
What is the most convenient way to visualize Softmax values after calling the CNN prediction function? Do I have to collect different probability values and feed them to the matplotlib or are there any more convenient ways/libraries to do this?

Below is one example what I mean:


texts: Linear regression load model doesn't predict as expected

texts:
I have trained a linear regression model, with sklearn, for a 5 star rating and it's good enough. I have used Doc2vec to create my vectors, and saved that model. Then I save the linear regression model to another file. What I'm trying to do is load the Doc2vec model and linear regression model and try to predict a new review.

There is something very strange about this prediction: whatever the input it always predicts around 2.1-3.0.

Thing is, I have a suggestion that it predicts around the average of 5 (which is 2.5 +/-) but this is not the case. I have printed when training the model the prediction value and the actual value of the test data and they range normally 1-5. So my idea is, that there is something wrong with the loading part of the code (or even the reshape of the new vector). This is my load code:

from gensim.models.doc2vec import Doc2Vec, TaggedDocument
from bs4 import BeautifulSoup
from joblib import dump, load
import pickle
import re

model = Doc2Vec.load('../vectors/750000/doc2vec_model')

def cleanText(text):
    text = BeautifulSoup(text, "lxml").text
    text = re.sub(r'\|\|\|', r' ', text)
    text = re.sub(r'http\S+', r'<URL>', text)
    text = re.sub(r'[^\w\s]','',text)
    text = text.lower()
    text = text.replace('x', '')
    return text

review = cleanText("Horrible movie! I don't recommend it to anyone!").split()
vector = model.infer_vector(review)

pkl_filename = "../vectors/750000/linear_regression_model.joblib"
with open(pkl_filename, 'rb') as file:
    linreg = pickle.load(file)

review_vector = vector.reshape(1,-1)
predict_star = linreg.predict(review_vector)
print(predict_star)


texts: Is feature weighting is helpful for boosting Machine Learning model performance?

texts: I am a beginner in machine learning. I am looking to know, is feature weighting is helpful for ML models & For which models performance improved and for which not.


texts: General approach to extract key text from sentence (nlp)

texts:
Given a sentence like:

Complimentary gym access for two for the length of stay ($12 value per person per day)

What general approach can I take to identify the word gym or gym access?


texts: Where is the cost parameter C in the RBF kernel in SVM?

texts:
RBF kernel using SVM depends on two parameters C and gamma. If the equation of the kernel RBF as the following:

$K(X,X')= \exp(\gamma||X-X'||^2)$

In the equation I can see where can I use gamma, but I can't find the C parameter.
So, can enybody tell me please?

Thanks in advance,


texts: determining batch size and epoch size of neural network

texts:
what should be my batch size and epoch size for 2 layer neural network if I am working with 1700 data?
I also want to know about neurons.how to determine the number??


texts: Loss function for multivariate regression where relationship between outputs matters

texts:
I am attempting to build a sequential model with Keras (Tensorflow backend) that has multiple outputs. My targets are proportions of a whole so each observation is an array like [0.5, 0.25, 0.15, 0.1]. The sum of each observation's array equals 1.

When I train the model, minimizing mean squared error, I get OK results. But looking over the validation results, it's clear that mean squared error may not be the best loss function for my problem. It is important for me that the sequences within y_true and y_pred are highly correlated.

As an example, let's say I have one observation [10%, 20%, 30%, 40%]. And each of the following are potential predictions for this observation:

[20%, 30%, 20%, 30%]
[20%, 10%, 40%, 30%]
[00%, 30%, 20%, 50%]
[00%, 10%, 40%, 50%]

import numpy as np
from sklearn.metrics import mean_squared_error

y_true   = np.array([[0.1, 0.2, 0.3, 0.4]])
y_pred_0 = np.array([[0.2, 0.3, 0.2, 0.3]])
y_pred_1 = np.array([[0.2, 0.1, 0.4 ,0.3]])
y_pred_2 = np.array([[0.0, 0.3, 0.2, 0.5]])
y_pred_3 = np.array([[0.0, 0.1, 0.4, 0.5]])

preds = [y_pred_0, y_pred_1, y_pred_2, y_pred_3]

for i, pred in enumerate(preds):
    corr = np.corrcoef(y_true, pred)[0,1]
    mse = mean_squared_error(y_true, pred)
    print(f'y_pred_{i}: corr={corr:.2f}, mse={mse:.2f}')

Gives us:

y_pred_0: corr=0.45, mse=0.01
y_pred_1: corr=0.60, mse=0.01
y_pred_2: corr=0.87, mse=0.01
y_pred_3: corr=0.98, mse=0.01

So while each of these four predictions has the same error, the fourth is most preferable to me because the sequence within the predicted array is most correlated with the observation's sequence.

I've found others who have used a modified correlation coefficient function as a loss function. But if I optimize on correlation coefficient, an optimal prediction for the example observation might be [0%, 1%, 2%, 3%] since the two sequences are perfectly correlated. This doesn't work since the mean error is so large.

So I need to optimize both for high correlation and a small error. I'm unable to find anywhere where this kind of problem has been solved. Is there a way I can optimize for both of these objectives? Particularly in the Keras framework?


texts: How to Validate Decision Tree model by using *statistical tests*?

texts:
I'm reading sklearn Decision Trees reference page.

In the advantages section, it is mentioned that 'Possible to validate a model using statistical tests. That makes it possible to account for the reliability of the model.'

Can someone please explain what statistical tests are performed to validate the DT model?


texts:
I have 128 bin FFT data for an audio file. I want to convert these FFT into MFCC of size 14, 20, 40. Is there some library which converts directly FFT to MFCC.

P.S - I dont have the actual audio file.


texts: Is it legal to scrape YouTube videos for training data?

texts:
Is it legal, at least as long as I am not "selling" the video under my name, to scrape YouTube videos to train a neural network?

If it is not, is there a procedure to get permission for the above? I am in academia and require a massive amount of video data that YouTube would be the perfect source in.

Note: I couldn't figure out where to post this questions, as it is not technically technical. I am posting it here only because I believe someone here would know a bit or two. Would appreciate any suggestions regarding community this question might have a better chance in. What about Law?


texts: Which openCv function can be used to compute BEV perspective transformation given a point coordinates and the camera extrinsics/intrinsics?

texts:
I have the 3x3 intrinsics and 4x3 extrinsics matrices for my camera obtained via cv2.calibrateCamera()
Now I want to use these paramenters to compute the BEV (Bird Eye View) transformation for any given coordinates in a frame obtained from the camera.
Which openCv function can be used to compute the BEV perspective transformation for given point coordinates and the camera extrinsics and/or intrinsics 3x3 matrices?
I found something very related in the following post: https://deepnote.com/article/social-distancing-detector/ based on https://www.pyimagesearch.com/2014/08/25/4-point-opencv-getperspective-transform-example/,
they are using cv2.getPerspectiveTransform() to get a 3X3 matrix, but I don't know whether this matrix represents the intrinsics, the extrinsecs or something else. Then they are transforming the list of points using such matrix in the following way:
#Assuming list_downoids is the list of points to be transformed and matrix is the one obtained above
list_points_to_detect = np.float32(list_downoids).reshape(-1, 1, 2)
transformed_points = cv2.perspectiveTransform(list_points_to_detect, matrix)

I really need to know if I can use this cv2.perspectiveTransform function to compute the transformation or if there's another better way to do this using the extrinsics, the intrinsics or both, without having to reuse the frame, since I already have the coordinates of people detected in the frame saved into an array.
Somebody just told me I can reach this if I have at least 3 3D points by using the cv2.findHomography function. I'm trying this approach now. Does anyone else have another input?


texts: "Change the features of a CNN into a grid to fed into RNN Encoder?" What is meant by that?

texts:
So in the paper for OCR pr LaTex formula extraction from image What You Get Is What You See: A Visual Markup Decompiler, they pass the features of the CNN into RNN Encoder. But there is problem that rather than passing the features directly, they have proposed a solution to change it into the grid.
Extract the features from the CNN and then arrange those extracted features in a grid to pass into an RNN encoder. This is the exact language they have used.
What is meant by that? Theoratically speaking, if I have an CNN without any Dense/Fully Connected layer and produces an output of [batch,m*n*C], then how can I change it in the form of a grid??  Please see the picture below. So after getting the output from the CNN, they have chnged it somehow before passing it to RNN. What is the method that one can use to get this transformation?

So if I have to pass something to keras.layers.RNN()(that_desired_grid_format), what should be this grid format and how can I change it?


texts:
I am new to data science and I am currently playing around a bit. Data exploration and preparation is really annoying. Eventhough I use pandas.

I achieved imputing missing values in independant variables. For numerical data by using the Imputer with the means strategy and for one categorical variable I used the Labelencoder and afterwards imputed with the mode strategy.

But now I face the issue that the dependant variable $y$ also contains missing values. Should I delete those lines or should I impute $y$ which is numerical.


texts: Fine tuning a automatic speech recognition model with my own dataset

texts:
I'm using wav2letter to develop a speech-to-text system.
wav2letter has pre-built acoustic and language models which is great, however the audio that I am transcribing from is unique in comparison to what the model has previously observed.
The audio predominantly consists of people talking on CB radios using code words or lots of abbreviations. As a result, the pre-built models just don't cut it, they miss certain phrases, which are really key (there are also new words it needs to learn). So, I'm trying to fine-tune one of their pre-built models with my own dataset. I've got about 4 hours of audio , which I can use to fine-tune the model (this could be insufficient, I know).
When I run training and generate the resulting graph of the model's performance, it seems to me like the model is underfitting.

My understanding is that underfitting happens when the data has not been normalised (?). Well, in my case I think that means I just have to check that the sample rate and number of channels on my dataset matches what was used to build the original model.
Would this be the way to resolve or underfitting? If so, what other things do I need to do to normalise my data? I have also tried audio normalisation on each file so that they are all the same volume.
If this is not the 'right' direction, what methods/techniques would be recommendeded for my particular task?
Any thoughts would be greatly appreciated.


texts: Python function returning a 4x4 matrix instead of a floating number like in an equivalent Octave function in a Linear Regression problem

texts:
I am trying to translate code from Octave to Python, and I am stuck. I am aware they are libraries out there such as scikit-learn etc., but for my own learnin,g I would like to be able to implement the cost function from scratch. Furthermore, I managed to solve some issues by looking into numpy documentation, but I am still stuck with one last one. The function doesn't return the expected result.
I am providing everything you need to reproduce below. Nothing more is needed.
The python function returns a 4x4 matrices instead of a float like it does in Octave and I get this:
Actual results: an numpy.ndarray
Expected results: a floating number

The function in computeCost(X, y, theta) in Octave:

function J = computeCost(X, y, theta)
    m = length(y);
    J = (((X*theta)-y)'*((X*theta)-y))/(2*m);
end

The function compute_cost(X, y, theta) in Python:

from numpy import dot

def compute_cost(X, y, theta):
    m = len(y)
    ans = (X * theta) - y
    product = dot((1/(2*m)), dot(ans, ans.T))
    return product

The tests:

import unittest
import numpy as np
from ml.ml_utils import compute_cost

class TestComputeCost(unittest.TestCase):
    data = np.loadtxt('../data/data1.txt', delimiter=',')
    x = data[:, 0]
    y = data[:, 1]
    m = len(y)
    X = np.vstack((np.ones(m), x))

    def test_compute_cost_with_theta_zeros(self):
        theta = np.zeros((2, 1), dtype=int)
        result = compute_cost(self.X, self.y, theta)
        self.assertEqual(round(result, 2), 32.07,
                         'Result is wrong!')

    def test_compute_cost_with_theta_values(self):
        theta = np.array([[-1], [2]])
        result = compute_cost(self.X, self.y, theta)
        self.assertEqual(round(result, 2), 54.24,
                         'Result is wrong!')

if __name__ == '__main__':
    unittest.main()

data1.txt

6.1101,17.592
5.5277,9.1302
8.5186,13.662
7.0032,11.854
5.8598,6.8233
8.3829,11.886
7.4764,4.3483
8.5781,12
6.4862,6.5987
5.0546,3.8166
5.7107,3.2522
14.164,15.505
5.734,3.1551
8.4084,7.2258
5.6407,0.71618
5.3794,3.5129
6.3654,5.3048
5.1301,0.56077
6.4296,3.6518
7.0708,5.3893
6.1891,3.1386
20.27,21.767
5.4901,4.263
6.3261,5.1875
5.5649,3.0825
18.945,22.638
12.828,13.501
10.957,7.0467
13.176,14.692
22.203,24.147
5.2524,-1.22
6.5894,5.9966
9.2482,12.134
5.8918,1.8495
8.2111,6.5426
7.9334,4.5623
8.0959,4.1164
5.6063,3.3928
12.836,10.117
6.3534,5.4974
5.4069,0.55657
6.8825,3.9115
11.708,5.3854
5.7737,2.4406
7.8247,6.7318
7.0931,1.0463
5.0702,5.1337
5.8014,1.844
11.7,8.0043
5.5416,1.0179
7.5402,6.7504
5.3077,1.8396
7.4239,4.2885
7.6031,4.9981
6.3328,1.4233
6.3589,-1.4211
6.2742,2.4756
5.6397,4.6042
9.3102,3.9624
9.4536,5.4141
8.8254,5.1694
5.1793,-0.74279
21.279,17.929
14.908,12.054
18.959,17.054
7.2182,4.8852
8.2951,5.7442
10.236,7.7754
5.4994,1.0173
20.341,20.992
10.136,6.6799
7.3345,4.0259
6.0062,1.2784
7.2259,3.3411
5.0269,-2.6807
6.5479,0.29678
7.5386,3.8845
5.0365,5.7014
10.274,6.7526
5.1077,2.0576
5.7292,0.47953
5.1884,0.20421
6.3557,0.67861
9.7687,7.5435
6.5159,5.3436
8.5172,4.2415
9.1802,6.7981
6.002,0.92695
5.5204,0.152
5.0594,2.8214
5.7077,1.8451
7.6366,4.2959
5.8707,7.2029
5.3054,1.9869
8.2934,0.14454
13.394,9.0551
5.4369,0.61705


texts: Classifying text documents using linear/incremental topics

texts:
I'm attempting to classify text documents using a few different dimensions. I'm trying to create arbitrary topics to classify such as size and relevance, which are linear or gradual in nature. For example:

size: tiny, small, medium, large, huge.
relevance: bad, ok, good, excellent, awesome

I am training the classifier by hand. For example, this document represents a 'small' thing, this other document is discussing a 'large' thing. When I try multi-label or multi-class SVM for this it does not work well and it also logically doesn't make sense.

Which model should I use that would help me predict this linear type of data? I use scikit-learn presently with a tfidf vector of the words.


texts:
I am interested in creating heat maps for my CNN which regresses images to a real value. However, most of the methods I have seen (eg, grad-CAM) were developed for classification.

Is there a way to adapt these methods for a regressive, not classifier CNN?
If not, what other methods are there? I have so far used occlusion sensitivity and measured the deviation.

Thank you!


texts: Fast PR / ROC curves and corespondings AUPR / AUROC

texts:
I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC).
Its is quite easy to perform those calculations with standards implementations (I am using sklearn metrics) or even by 'hand'. However, with millions of instances, exact calculations are very time consuming for precision that do not seem necessary for exploratory analyses.
I have been looking for fast approximations for quite a while now. There are some people that propose to optimise the exact calculations, there are some people that propose some approximations for AUROC.
But do we have good approximations for fast PR / ROC curves calculations ?


texts: Recommender/Clustering data to support a hypothesis. Is this a valid use-case for unsupervised ML?

texts:
I have a dataset where some items have been labelled (categorized into 4 classes [A,B,C,D]). However, there is a vast majority of the dataset which has not been labelled. My hypothesis is that there are some characteristics which influence which category is applied to each item. Would clustering or maybe even a recommender system be able to suggest where each item should be placed? On a practical level, would I provide the "labels" within the model? Or would I keep it apart until the end and then overlay those labels on whatever the model managed to group together?
The above example seems like a clustering use case. However, can I spin the problem into a recommender system? As in, you labelled item X as A, and it has characteristics 1,2,3... item Y has similar characteristics, maybe you should label it A as well?


texts: Does GridSearchCV not save the best parameters?

texts: So I tuned the hyperparameters using GridSearchCV, fitted the model to the data, and then used best_params_. I'm just curious why GridSearchCV takes too long to run best_params_, unlike RandomSearchCV where it instantly gives answers. The time it takes for GridSearchCV to give the best_params_ is similar to the time it takes for GridSearchCV to tune hyperparameters, and fit the model to the data. It's as if it's doing it all over again when it has done so already. Is this the case? If not, what's taking it so long when it should have saved the best_params_ when I ran GridSearchCV the first time?


texts: Can continuous dataset have negative values?

texts: As in the description, I want to choose continuos dataset with temperatures and the minimun value is -40 and maximum is 51. Is that OK? Can we have negative values in continuous distribution?


texts:
i am new to neural networks using keras,
i have the following train samples input shape (150528, 1235) and output shape is (154457, 1235) where 1235 is the training examples,
how to put the input shape, i tried below but gave me a
ValueError: Data cardinality is ambiguous:
  x sizes: 150528
  y sizes: 154457
Please provide data which shares the same first dimension.

code:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dense
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import categorical_crossentropy

physical_devices = tf.config.experimental.list_physical_devices('GPU')
print("Num GPUs Available: ", len(physical_devices))
tf.config.experimental.set_memory_growth(physical_devices[0], True)

model = Sequential([
    Dense(16, input_shape=(150528,), activation='relu'),
    Dense(32, activation='relu'),
    Dense(154457)
])

model.compile(optimizer='sgd', loss='mse', metrics=[tf.keras.metrics.MeanSquaredError()])
model.fit(im_features, amp_vo_features, batch_size=10, epochs=30, shuffle=True, verbose=2)


texts: Transforming negative correlated non linear variable to linear positive correlated variable

texts:
At my office, I am stuck in a weird situation. I am asked to perform a regression algorithm on the data, in which the target variable is continuous having values range between 0.6 to 0.9 with 8 digits of precision after the decimal. Although I know and have applied many linear and non-linear regression algorithms in the past the case here is something different. There is one variable, which, according to my BU, should have a positive and linear correlation with the target variable. But when I ran Pearson's correlation, the variable is negatively correlated and by plotting a scatter plot I can see that the relationship is not linear at all.
What transformations can I perform on the variable so that it can show a positive correlation? I am fairly new to this problem so hoping to get it solved here. Thanks much everyone in advance.


texts: Statistical test to infer if brand preference depends on income

texts:
For the data in image can anyone please suggest the right statistical test. Want to test if brand preference depends on income level.

The percentages adds upto 100% row wise meaning the values in a row are percentage share of different brands in total purchase amount of an income group.


texts: Why can't I use 2D-arrays as features for CCA (Canonical Correlation Analysis) classifier?

texts:
The Problem
When using fit of the scikit learn CCA classifier it won't allow me to use arrays as features. The error ValueError: Found array with dim 3. Estimator expected <= 2. can be produced whit the following code
from sklearn.cross_decomposition import CCA
    CCA_model = CCA(n_components = 3, max_iter=20000)

    input_arr = [[[k*-1+j*-i*-1 for k in range(125)] for j in range(2)] for i in range(189)]
    input_arr = np.array(input_arr)
    print("INPUT SHAPE:", input_arr.shape)
    input_lbl = [[(-(-1+(-1)**(1+k+j)))/2 for k in range(3)] for j in range(189)]
    input_lbl = np.array(input_lbl)
    print("LABEL SHAPE:", input_lbl.shape)

    model = CCA_model.fit(input_arr, input_lbl)

>>INPUT SHAPE: (189, 2, 125)
>>LABEL SHAPE: (189, 3)

The question
Why is it so, shouldn't it be allowed to use arrays as single features? Is there any parameter I need to modify to do this?


texts:
Can someone help me how space transformation works on linear regression problems because I have been confused.

When we perform space transformation with a function e.g. $\varphi (x)$ we perform the transformation on dependent variable, independent or both? with which method?

For example if I have the following nonlinear model
$y = 2x_1^2 + 8x_2^2 + 4x_1x_2 + 8x_2 + 9x_1$
and I want to transform to a space where the problem of estimating the model is linear?

Also any link related to the problem.

Thanks


texts: Using local outlier factor score to detect outliers at run time

texts:
I am using LOF ( local Outlier factor) to detect outliers in my data. I get LOF score as outlier distance. this unsupervised learning doesnt help to detect outliers at run time. So I want to use my data points and LOF score to have a supervised model [regression /classification].

My question is which should be approach out of
 1) classification ( taking a cutoff of LOF score and after having binary variable , build classification model) and   use this classification model to predict outliers at run time.

How does any clustering/anomaly detection( using clustering)  is even used at run time ?? Any guidance will help.


texts: keras categorical CNN prediction always output [[ 0. 0. 0. 0. 0. 0. 1. 0.]] No probability for other classes?

texts:
print(model.predict(image))
print(model.predict_proba(image))
print(model.predict_classes(image))

Using the above different predict methods I always get predictions like

  [[ 1.  0.  0.  0.]]

but I want the probabilities of each classes something like this

  [[ 0.8  0.05  0.05  0.1]]

This is the model

model = Sequential([
    Conv2D(32, (3, 3), input_shape=input_shape, padding='same',activation='relu'),
    Conv2D(32, (3, 3), activation='relu', padding='same'),
    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),
    Conv2D(64, (3, 3), activation='relu', padding='same', ),
    Conv2D(64, (3, 3), activation='relu', padding='same', ),
    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),
    Conv2D(128, (3, 3), activation='relu', padding='same', ),
    Conv2D(128, (3, 3), activation='relu', padding='same', ),
    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),
    Conv2D(256, (3, 3), activation='relu', padding='same', ),
    Conv2D(256, (3, 3), activation='relu', padding='same', ),
    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),
    Conv2D(512, (3, 3), activation='relu', padding='same', ),
    Conv2D(512, (3, 3), activation='relu', padding='same', ),
    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),
    Flatten(),
    Dense(512, activation='relu'),
    Dense(num_classes, activation='softmax')
])

Compiled as

sgd = SGD(lr=0.0001,momentum=0.9)
model.compile(loss = "categorical_crossentropy", optimizer =sgd,metrics=['accuracy'])

Not sure if this has any effects, but this is how I loaded the data

  train_datagen = ImageDataGenerator(
      rescale=1./255,
  )

  train_generator = train_datagen.flow_from_directory(folder+'apex/train',
                                                      target_size=(200, 400),
                                                      batch_size=batch_size,
                                                      class_mode='categorical')


texts: Question about graphing the clusters in K means

texts: I've used K means to cluster my data. Before using K means, I had used StandardScaler on my data to standardize the data. Now, I'm wondering how can I show the clusters of the original data. Scikit-learn gives the labels on the standardized data but I want to have the labels on the original data and show the clusters of the original data on the graph.


texts: I understand the process and logic of why to perform stepwise regression. To me they should always arrive at the same function, just one adds coefficients and tests for significance while the other removes coefficients as it tests for significance. Are there any significant trade-offs between using one versus the other?


texts: Storing Sensor Data for Analysis of the Office

texts:
I have currently been tasked with designing an application that tracks several different measurements around the office, eg. the temperature, light, presence of people, etc. Having never really worked on data analysis before, I would like some guidance on how to store this data (which database design to use).

What we're looking at currently are around 50 sensors that only send data when an event of interest occurs: if the temperature changes by 0.5 degrees or if the light turns on/off or if a room becomes occupied/vacant. So, the data will only be updated every few seconds. Also, in the future, I'd like to analyse some of the data. Hence, the data must be persistent in the database. What kind of technologies would you suggest to carry out this task?


texts: Q-learning when minimising a total cost instead of maximising a total reward

texts:
I have a decision problem where the results are measured as a cost that I want to minimise. It seems like a good fit to Q-learning, but I am not sure how to adjust it to deal with a cost instead of a reward.

Which one is better:

Initializing Q-values for all actions with zeros, then getting the agent to learn the actions that maximize the Q-values, and later filter out the actions with minimum Q-values. The Q-values update would then be:

q_dict['state1']['act1'] +=
    r + (max([q_dict['state2'][u] for u in q_dict['state2']]))

Initializing Q-values with a big number then getting the agent to learn actions that minimize the Q-values, and later filtering out the actions with minimum Q-values. The Q-values update would then be:

q_dict['state1']['act1'] -=
    r + (max([q_dict['state2'][u] for u in q_dict['state2']]))


texts: Kernel regression results in diverse outputs


texts: How is cross validation used to prune a decision tree

texts:
As I understand it one can use cross validation to help find the optimal pruning of a classification or regression tree, for example, implemented here in Matlab.

I do not understand the following about this process. Since each cross validation fold has random data, it may be that the trees fit on each data set may be different and have differing depths. How can one find the best pruning level over all such trees then?


texts: Model Performance using Precision as evaluation metric

texts:
I am dealing with an imbalanced class with the following distribution :
(Total dataset size :  10763 X 20)

0 : 91%

1 : 9%

To build model on this dataset having class imbalance, I have compared results using

1) SMOTE and

2) Assigning more weight to the minority class when applying fit

and the latter seems to be working better.

After experimenting with Decision Tree, LR, RF SVM(poly and rbf) I am now using XGBoost classifier which gives me the below classification results(these are the best numbers I've got so far)

The business problem I'm trying to solve requires the model to have high precision, as the cost associated with that is high.

Here's my XGBClassifer's code :

xgb3 = XGBClassifier(
    learning_rate =0.01,
    n_estimators=2000,
    max_depth=15,
    min_child_weight=6,
    gamma=0.4,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.005,
    objective= 'binary:logistic',
    nthread=4,
    scale_pos_weight=10,
    eval_metrics = 'logloss',
    seed=27)
model_fit(xgb3,X_train,y_train,X_test,y_test)

And here's the code for model_fit :

def model_fit(algorithm,X_train,y_train,X_test,y_test,cv_folds = 5 ,useTrainCV = "True", early_stopping_rounds = 50):
if useTrainCV:
    xgb_param = algorithm.get_xgb_params()
    xgbtrain = xgb.DMatrix(X_train,label = y_train)
    xgbtest = xgb.DMatrix(X_test)
    cvresult = xgb.cv(xgb_param,xgbtrain,num_boost_round = algorithm.get_params()['n_estimators'],
    nfold = cv_folds,metrics='logloss', early_stopping_rounds=early_stopping_rounds)
    algorithm.set_params(n_estimators=cvresult.shape[0])

algorithm.fit(X_train, y_train, eval_metric='logloss')
y_pred = algorithm.predict(X_test)
cm = confusion_matrix(y_test,y_pred)
print(cm)
print(classification_report(y_test,y_pred))

Can anyone tell me how can I increase the precision of the model. I've tried everything I know of. I'll really appreciate any help here.


texts: Mathematically prove why sparsity leads to model overfitting

texts:
With respect to the stackoverflow post here: https://stackoverflow.com/a/59566478/9130959

I can't quite get why the logic stands: when # features increases, the hypothesis space is expanded, leading to sparse data, thus easily overfit. Is there a way to mathematically explain all these?
Thx in advance.


texts: Financial Time Series data normalization

texts: I'm using Keras in R to predict financial time series. It's easy to normalize price, simply compute returns or log returns, usually it's enough. I want to use Goldman Sachs Financial Conditions Index and MSCI World Index to predict other securitites and I want to use levels and their returns or first differences.  I think that using minmax or z-score normalization is not appropriate because series distribution will change. So, the question is how to normalize nonstationary time series data?


texts: Would K-means be Appropriate to Use with Four or More Variables?

texts:
Just a general question that I'm trying to mentally visualize. I'm fairly new to using k-means clustering and have used it before on two variables, which creates a 2-D plot of points. I also know, although I haven't done it before, that you can plot a k-means cluster with three variables utilizing the x, y, and z axes. But now I'm currently in a position where I have four variables, normalized by their z-scores, in which I'm not sure how to use the k-means clustering appropriately. Should I be using a k-means cluster in this circumstance?

Thanks


texts: Python: significance test (claim), one sample, test the proportion - simpler code

texts:
We have two sheets with pictures of dogs and dog owners. On one sheet - owners and their dogs are associated correctly. The other sheet shows random associations between owners and dogs.
61 students are asked to guess which sheet shows the real associations between pictures of owners and dogs. 49 of them guess correctly. If this was a random guess, 50% would guess right. Is this evidence that students are doing better than plain guessing?
H0: p = 0.5
Ha: p > 0.5
I can calculate the z score like this:
https://www.tutorialspoint.com/statistics/one_proportion_z_test.htm
And then in Python I can get the P-value like this:
from scipy import stats
P_value = stats.norm.sf(z_score)

which gives a P-value of 1.08e-06, so H0 is rejected. Students are doing better than guessing.
Is there a one-liner or two-liner code example in either scipy or statsmodels that would give me the P-value more directly? Just input the number of observations, number of successes, and the expected proportion, and get the P-value directly?
In other words, can I avoid memorizing the formula for the z score, and instead just remember some function in either scipy or statsmodels?


texts: Wiggle in the initial part of an LSTM prediction

texts:
I working on using LSTMs and GRUs to make time series predictions. For the most part the predictions are pretty good.

However, there seems to be a wiggle (or initial up-then-down) before the prediction settles out similar to the left side of this figure from another question.

In my case, it is also causing a slight offset. Does anyone have any idea why this might be the case? Below are the shapes of the training and test sets, as well as the current network structure. I've tried reducing the sequence lengths from 60 timesteps, switching between LSTMs and GRUs, and also having the inputs and outputs overlap slightly, all to no avail. Adding dropout does not seem to help either. The wiggles will not disappear!

My sequence lengths are 60 inputs and 60 outputs, currently.

Xtrain: (920, 60, 2)
Ytrain: (920, 60, 2)
Xtest: (920, 60, 2)
Ytest: (920, 60, 2)

def define_model():
    model = Sequential()
    model.add(LSTM(64, return_sequences=True, input_shape=(None, 2)))
    model.add(LSTM(64, return_sequences=True))
    model.add(LSTM(64, return_sequences=True))
    model.add(TimeDistributed(Dense(2)))
    model.add(Activation('linear'))
    model.summary()
    return model


texts: Meaning of absolute tolerance and relative tolerance

texts:
Hello Data science community,
I am using a library nnet in R. The documentation, page no. 4-5, shows that it has default values for abstol and reltol as a stopping criteria.
nnet uses optim library for optimizing. I checked its documentation also. But I could not find the meaning abstol and reltol
abstol and reltol stands for absolute and relative convergence tolerance. I wish to change these values. But can someone please explain its meaning.
Thanks


texts:
Why is the softmax activation function used in the output layer for CNNs?
Why not just take the highest value of the units in the output layer?