texts: Test dataset with categorical variable value not present in train dataset & transformer

texts:
I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ).

I am fitting a transformer doing just that on a train dataset df and then transform the test dataset df2.

How do I deal with a value appearing solely in the test dataset ?

When fitted on the train dataset the transformer received no mean value of the target variable on that value.

For example :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 'a' , 'x' , 0 ] ,
                   [ 2 , 2 , 2 , 'v' , 'b' , 'y' , 1 ] ,
                   [ 4 , 5 , 1 , 'w' , 'c' , 'z' , 1 ] ,
                   [ 2 , 1 , 9 , 'w' , 'c' , 'x' , 1 ] ,
                   [ 1 , 0 , 4 , 'w' , 'b' , 'y' , 1 ] ,
                   [ 2 , 2 , 3 , 'v' , 'b' , 'z' , 0 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'five' , 'six' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )

myarray2 = np.array([ [ 2 , 7 , 3 , 'v' , 'a' , 'x' , 0 ] ,
                      [ 9 , 2 , 2 , 'v' , 'a' , 'y' , 0 ] ,
                      [ 4 , 5 , 1 , 'w' , 'c' , 'k' , 1 ] ]  )

colnames2 = [ 'one', 'two', 'three' , 'four' , 'five' , 'six' , 'target' ]

df2 = pd.DataFrame( myarray2 , columns = colnames2 )

df is my train dataset, df2 my test dataset.

We can see variable 'six' has the k value not existing in the train dataset.

Next :

df[ 'target' ] = df[ 'target' ].astype( 'float64' )

Next ( my homemade transformer ) :

class Cat2TargetMean( BaseEstimator , TransformerMixin ) :

    def __init__( self , col2trans , tgt_col ) :
        self._col2trans = col2trans
        self._tgt_col = tgt_col

    def fit( self, X, y = None ) :
        self._dic_col_p = {}

        for col in self._col2trans :
            p = X.groupby( col ).mean()[ self._tgt_col ]
            self._dic_col_p.update( { col : p.to_dict() } )
        return self

    def transform( self , X , y = None ) :
        for col , dic_p in self._dic_col_p.items() :
            X.replace( { col : dic_p } , inplace = True )
        return X

Then :

 tsf = Cat2TargetMean( [ 'four' , 'five' , 'six' ] , 'target' )

 tsf.fit( df )

 tsf.transform( df )

 tsf.transform( df2 )

Result :

    one two three   four    five    six target
    0   2.0 7.0 3.0 0.333333    0.0 0.5 0.0
    1   9.0 2.0 2.0 0.333333    0.0 1   0.0
    2   4.0 5.0 1.0 1.000000    1.0 k   1.0

'k' value of column 'six' has not been transformed.


texts: Large general-purpose model vs ensemble of many smaller models

texts:
I am reading this paper - https://arxiv.org/abs/1503.02531v1 - devoted to knowledge distillation in neural networks.
One interesting approach is mentioned in this paper in sections 5.3, 5.4 - to create smaller specialists models, which are trained to distinguish only a subset of labels and mark all other labels as some dummy/background class.
The most straightforward application would be to split the data in some large dataset, for instance Imagenet, with large number of classes into smaller parts with images containing similar in some classes, grouped by a certain meaningful criterion. Then one creates a separate model for this subset of labels, for example, model which is trained to distinguish cars, or model which identifies different cat species.
On the inference stage when passing the image one gets probabilities for presence of certain classes,  and chooses n-most confident models and aggregates in a certain ways outputs of these models. Another option would be to create a general purpose model - generalist - and pass the input image to the specialists, according to the probabilities , obtained by the generalist.
Advantages
The advantage of this approach is that each model would learn features pertinent to the concrete task. At the same time, multi-purpose model learns about a lot of classes, but knowledge of features, relevant to different classes may interfere with each other.
Disadvantages and issues
Due to seeing less data, than the general purpose model, each of the specialist model is more prone to overfitting, as mentioned in the paper. However, superficially solving less sophisticated tasks models can be created much simplier, shallower and containing less layers. Ensembling effect from different specialists in case of partially overlapping domains- seems to remedy at a certain extent overfitting.
Another problem is an increase possible increase of the inference time, because one needs to evaluate several models for each sample. On the other hand, each of the model is smaller, and in prior it is difficult to say, whether evaluation of several simpler models would be more computationally expensive than the one pass through big model.
Moreover, large models, trained on Imagenet are useful as a feature extractors for various problems in Computer Vision tasks and in Transfer learning, whereas this ensemble of experts would not be useful in this case.
Nevertheless, this specialists ensemble approach seems to be interesting, but I've not seen a lot of literature devoted to this topic. Did this approach turn out to be unsuccessful or inefficient?


texts: How to place multiple graphs in same coordinate system (pandas,mathplotlib) -> look pic

texts: in short how to set graphs in same coordinate system, not separated as in pic each in its own coordinate system in one frame....


texts: How to insert two features in a model when a feature only applies to a certain group in the model

texts:
I'm building a machine learning model in Python to predict soccer player values. Consider the following feature columns of the dataframe:
         [features]
---------------------------------
position | goals | goals_conceded
-------- |-------|---------------
Forward  |  23   |     NaN
Defender |   2   |     NaN
Defender |   4   |     NaN
Keeper   |   NaN |     20
Keeper   |   NaN |     43

Since keepers don't usually score goals, they'll almost always have null values in the "goals" column, but they still can have this statistic, so it would be fine to fill the NaNs with 0. On the other hand, since line players can't have "goals_conceded" stats, they'll also have null values in that column, but in this case, players will never have this statistic, since this is a keeper only stat. How do I build a machine learning model considering these two columns as features?
I thought about putting them together in one single column, but that can't happen since for a line player, the more goals he makes the better it is. For goalkeepers it's the opposite, the less goals he conceeds the better. I also can't fill the columns with zeros since it would affect the model prediction in the "goals_conceded" column for example, since 98% of the rows contain in line players info.
This happens with many of the columns in my dataframe, such as "clean sheets" (only Keepers will have this stat) and "shots at target" (only line players will have this stat). How do I deal with them?


texts: What algorithms should I use to perform job classification based on resume data?

texts:
Note that I am doing everything in R.

The problem goes as follow:

Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: based on the text on their CVs, I want to classify them into different job sectors. I am particular in those cases, in which the candidates do not have any experience / is a student, and I want to make a prediction to classify which job sectors this candidate will most likely belongs to after graduation .

Question 1: I know machine learning algorithms. However, I have never done NLP before. I came across Latent Dirichlet allocation on the internet. However, I am not sure if this is the best approach to tackle my problem.

My original idea:  make this a supervised learning problem.
Suppose we already have large amount of labelled data, meaning that we have correctly labelled the job sectors for a list of candidates. We train the model up using ML algorithms (i.e. nearest neighbor... )and feed in those unlabelled data, which are candidates that have no work experience / are students, and try to predict which job sector they will belong to.

Update
Question 2: Would it be a good idea to create an text file by extracting everything in a resume and print these data out in the text file, so that each resume is associated with a text file,which contains unstructured strings, and then we applied text mining techniques to the text files and make the data become structured or even to create a  frequency matrix of terms used out of the text files ? For example, the text file may look something like this:

I deployed ML algorithm in this project and... Skills: Java, Python, c++ ...

This is what I meant by 'unstructured', i.e. collapsing everything into a single line string.

Is this approach wrong ? Please correct me if you think my approach is wrong.

Question 3: The tricky part is: how to identify and extract the keywords ? Using the tm package in R ? what algorithm is the tm   package based on ?  Should I use NLP algorithms ? If yes, what algorithms should I look at ? Please point me to some good resources to look at as well.

Any ideas would be great.


texts: How to cope with new/unseen targets classes in incremental learning algorithms

texts: According to scikit-learn documentation1, the sklearn incremental learner itself may be unable to cope with new/unseen targets classes. Is there any available python machine learning library which support this functionality?


texts: What is the best way to normalize histogram vectors to get distribution?

texts:
l have the following sample of 4 vectors of dimension 5 . They are sparse vectors, in a way that each value in a vector represent the frequency (number of occurrence of values). For instance

v_1=[0,4,0,0,1]

4 at index 1 means we have four values of 1 and 1 at index 4 it means we have one value of 4.

The purpose of normalizing these histograms is to get a distribution.

Here is a sample of my data.

Vectors=[[0,4,0,0,1],[5,0,0,3,1],[1,0,0,0,0],[0,0,6,0,0]]

What l tried ?

vectors=(vectors-np.mean(vectors,axis=0)) / np.std(vectors,axis=0)

and l tried :

vectors=sklearn.preprocessing.scale(vectors)

A)Is it correct what l have tried ?

B)l have question to normalization by subtracting means and dividing by standard deviation:
through all the vectors, through each dimension, means( of a dimension) is computed. Than each data point of this dimension is subtracted from its mean. Is it correct what l'm saying ?
in other way for each dimension, we have it's own mean.

For instance, if we have dim(vectors)=(400,2000) such that 400 is the number of examples.
we compute 2000 means and 2000 std since we have 2000 dimensions. And then each data point is subtracted from its means and divided by its std (mean and std of that dimension)

Thank you for correcting me.


texts: Coloring labels using scatterplot3d in R

texts:
I am trying to visualize data using R and scatterplot3d.

I have loaded data and used:

colors <- c("#999999", "#E69F00", "#56B4E9" )
scatterplot3d(output$X2,output$X6 , output$X7 , color=colors, pch="X9")

X9 is label column in my dataset. it contains 3 categories : A , B , C.

By documentation:

  color : colors of points in the plot, optional if x is an appropriate
  structure. Will be ignored if highlight.3d = TRUE.

  pch: plotting  "character", i.e. symbol to use.

Yet I still get this error

  Error in scatterplot3d(output\$X2, output\$X6, output\$X7, color =
  colors,  :    length(color) must be equal length(x) or 1

I assumed I had to put color for every collumn in dataset, but creating such array where:

length( colors ) = length( num of columns )

yields the same error.

What is the right way to add colors?


texts:
I have multiple time series sequences and I want for each new time series to find the most alike old one.

I found that I can use the sum of errors between points. Is this a good approach?
Is there a way to be able to compare sequences with different lengths (maybe a sequence look like a subsequence of another sequence)?
Will scaling the data before comparing make difference?


texts: Good approach to increase accuracy for a continuous value that is highly variable/sensitive to the inputs?

texts:
I am trying to predict a continuous 'Y' variable using a variety of algorithms and feature engineering techniques. My issue is that Y is extremely variable and I reached a asymptote in accuracy.
This is the structure of my feature variables (with swapped variable names)

Screw Width (mm)
Screw Height (mm)
Screw Angle (degrees)
Screw Type
Screw Material
Car Model Id
Car Age (Years)

0.53
0.24
43
Eye Bolt
Carbon
1
3

My target variables go as followed:

Speed without Screw
Speed with Screw

24
29

Which I merge into a single variable Y:

Speed Delta

5

The range of the Delta can go from -2000. to 165,000. Please note I swapped the variable names again so I'm not in reality predicting speed.
Currently my R2 score is 0.9 and the mean prediction error rate is 25%. I want to get it down to 15%
What I have tried so far:

Constraining the Input and Sensitivity Analysis on the variables for current Regressor Choice

For some time I tried:

Splitting the data (e.g. run on Screw Width < 0.5 mm)
Build a model on the split data
Record the new R2 score/Mean Error

To my surprise it wasn't one variable or range of that variable that was causing huge variance as repeating this experiment gave me similar results. For example filtering heavily on the variables and repeating steps 1-3 gave me results like as followed:

Using different regressors to predict a single continuous variable

I tried using a simple Linear Regressor, MLP Regressor, Gradient Boosting and Random Forest. I haven't played too much with the parameters of each regressor and am not sure if that is my downfall. Currently a RandomForestRegressor is giving me best results but each regressor converges around a similar range.

Simple Feature Engineering:

I tried OHE for non numerical variables, Log Transforming the Data. The difference was noticeable but still small.

Running more data

I have close to 250,000 simulations (or rows) and reached a bit of an asymptote at 25% accuracy after 150,000 simulations with the strategies/algorithms I deployed.

Summary: I need advise on how to tackle this problem. Do I just keep on randomly trying methods/manipulating the data until something sticks. Or is there a better path moving forward to get a better accuracy?
What have I not tried?


texts: classification performance metric for high risk medical decisions

texts: What is the best classification performance metric for risky medical treatments like surgery? for example a patient should NOT suggest a surgery (negative) if he/she can be treated by medicine (positive). Does Negative predictive value (TN/TN+FN) works for this situation?


texts: Any Machine Learning algorithm to know a yes/no answer?

texts: Any Machine Learning algorithm to know the meaning of sentences? Specifically, I have sentences like, "we do not allow managers to trade derivatives", "We do not have policy on hedging", "our policy do not permit trading on stock", "trading is not allowed"... The actual sentence is longer. Any algorithm I can know whether trading on stock is allowed or not by programming? Thank you for your help!!!


texts: Why i get OOM error although my model is not that large?

texts:
I am a newbie in GPU based training and Deep learning models. I am running cDCGAN (Conditonal DCGAN) in tensorflow on my 2 Nvidia GTX 1080 GPU's. My data-set consists of around 32,0000 images with size 64*64 and 2350 class labels. If I set my batch size 32 or large I get OOM error like below. So I am using 10 batch size for now.

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,64,64,2351] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[Node: discriminator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_41, _arg_Placeholder_3_0_3/_43, discriminator/concat/axis)]]
Caused by op 'discriminator/concat', defined at:
  File "cdcgan.py", line 221, in <module>
    D_real, D_real_logits = discriminator(x, y_fill, isTrain)
  File "cdcgan.py", line 48, in discriminator
    cat1 = tf.concat([x, y_fill], 3)

The training is very slow which I understand is down to the batch size (correct me if I am wrong). If I do help -n 1 nvidia-smi, I get the following output.

The GPU:0 is mainly used, as the Volatile GPU-Util gives me around 0%-65% whereas GPU:1 is always 0%-3% max. Performance for GPU:0 is always in P2 whereas GPU:1 is mostly P8 or sometimes P2. I have the following questions.

1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?

2) How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

Model Details are as follows

Generator:

I have 4 layers (fully connected, UpSampling2d-conv2d, UpSampling2d-conv2d, conv2d).
W1 is of the shape [X+Y, 16*16*128] i.e. (2450, 32768), w2 [3, 3, 128, 64], w3 [3, 3, 64, 32], w4 [[3, 3, 32, 1]] respectively

Discriminator

It has five layers (conv2d, conv2d, conv2d, conv2d, fully connected).
w1 [5, 5, X+Y, 64] i.e. (5, 5, 2351, 64), w2 [3, 3, 64, 64], w3 [3, 3, 64, 128], w4 [2, 2, 128, 256], [16*16*256, 1] respectively.

Session Configuration
I am also allocating memory in advance via

gpu_options = tf.GPUOptions(allow_growth=True)
session = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))


texts: How to explain decision tree algortihm in layman's terms?

texts:
I have a task at hand, where I have to explain decision tree algorithm to a person who has not much understanding of machine learning. I have been looking around, but find it difficult to explain the algorithm in layman's terms, so that a person will understand what is happening in the process.

What is the best way to describe the algorithm, possibly using some very simple basic example, so that the whole process of the algorithm is broken down to simple steps for understanding?


texts: Extreme Learning Machine - why random weights rather than space-spanning?

texts:
Putting aside the pros and cons of Extreme Learning Machines (ELMs), there is something pretty fundamental that I don't understand: why are the weights and biases in the hidden layer randomised?

Would it not be better to thoroughly span the weight-bias space with ordered values so that we can be sure that all of the space has been spanned? Surely randomised weights and biases might leave regions of the space sparsely populated and thus compromise the outcome, i.e. you don't have a node in a region where it would be beneficial to have one?

Compare the two figures at the top-right of the Sobol sequence Wikipedia page, for instance. The pseudorandom sequence produces a lot of empty regions in comparison with the sobol sequence.

Pseudorandom Sequence

Sobol Sequence

Images by Jheald [CC BY-SA 3.0  (https://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia Commons


texts: SVM radial basis generate equation for hyperplane

texts:
I need to generate an equation for hyperplane, I have two independent variables and one binary dependent variable.

Regarding this following equation for svm , $f(x)=sgn( sum_i alpha_i K(sv_i,x) + b )$

I have two independent variables (say P and Q) with 130 point values for each variable. I used svm radial basis function for binary classification (0 and 1) and I calculated for radial basis kernelized case, and now I have

One column of 51 y (i) alpha (i) or (dual coefficients).
Two columns of 51 sv (support vectors)for P and Q.
One single value for b .

I received these using scikit SVC.

So, how can I generate the equation now?

Can I multiply those 51 y (i) alpha (i) or (dual coefficients) with 51 sv (support vectors) for each variable P and Q so that I have two coefficients for P and Q so that finally my equation appears as f(x)=sgn( mP + nQ +b) where m = sum of the (product of 51 sv of P with 51 dual coefficients) and n = sum of the (product of 51 sv of Q with 51 dual coefficients)?


texts: Bellman operator and contraction property

texts: Currently, I am learning about Bellman Operator in Dynamic Programming and Reinforcement Learning. I would like to know why is Bellman operator contraction with respect to infinity norm? Why not another norm e.g. Euclidian norm?


texts:
I am trying to train a Siamese network for an application very similar to this and this. From what I have read about training Siamese networks dissimilar pairs of images outnumber the similar pairs and obviously so. In the papers I have linked to, the authors talk about a 1:20 ratio for similar to dissimilar pairs, i.e., for every similar pair of images, the training set consists of 20 dissimilar pairs. If I have understood it correctly, for a batch size of 64, my batch will consist of 3 similar pairs and the corresponding 20 dissimilar pairs for each of the similar pair.

I implemented the same in Keras, and the network overfits from the very first epoch. I have approximately 6000 similar pairs of images and I am generating the dissimilar pairs randomly. How do I reduce the overfitting? Any tips would help.

NOTE: I am using the last two Inception modules of Inception V3 to train my data.
EDIT: We managed to fix this issue with getting more training data, and using a network pretrained on a very similar task.


texts: How to calculate the distance between two locations using Haversine Formula?

texts:
I have the columns of Latitude and Longitude of city like shown below :
             City                        Latitude                 Longitude

1) Vauxhall Food & Beer Garden           -0.123684                  51.485020

2) 14 Hills                              -0.129212                  51.507426

3) Cardiby                               -0.123234                  52.476264

Now I want to calculate distance between the coordinates of specific place with all cities as shown in  data frame . Like
Lon = 0.1245
Lat = 51.685
These above two coordinates should be subtracted with Lat/Lon cities of DataFrame . Output should look like:
      City                        Latitude                 Longitude            Distance

1) Vauxhall Food & Beer Garden           -0.123684                  51.485020    275km

2) 14 Hills                              -0.129212                  51.507426    856km

3) Cardiby                               -0.123234                  52.476264    584km

My code :
from haversine import haversine

from math import sin, cos, sqrt, atan2, radians

df['Latitude'] = [radians(i) for i in df['Latitude']]
df['Longitude'] = [radians(j) for j in df['Longitude']]

lat2 = radians(50.0863)
lon2 = radians(14.4139)

dlon = [i-lat2 for i in df['Latitude']]
dlat = [j-lon2 for j in df['Longitude']]

df['distance']=haversine(df['Latitude'],df['Longitude'],dlat,dlon)

print(df['distance'])

It is not working ,


texts: why naive is needed in Naive Bayes ,what happens if naive is not included in Bayes theorem?

texts:
Im trying to understand why naive is needed in Naive Bayes and everyone says Naive Bayes assumes the input features (predictors) are not correlated hence they are not dependent on each other .

i want to understand what will happen if features are dependent i.e Non Naive (the opposite part of being naive )

if we have a sentence "You won lottery for 1million" and we need to classify it as spam and not spam using naive bayes .

p(y|x)=p(x|y).p(y)

In the likelihood partwe will model the probability as p(x|y)

here x="You won lottery for 1million" and y=spam or not spam

p('You won lottery for 1million'|y=spam)

p('You won lottery for 1million'|y=notspam)

what is the correct way of writting this probability and finding its value without considering independence of event among X ?

should it be written as

**to find probaiblity of spam given feature are depenent**
p('You |won, lottery, for, 1million,spam) *
p('won| lottery, for, 1million,spam) *
p('lottery| for, 1million,spam)*
p(for| 1million,spam)*
p( 1million|spam)

**to find probaiblity of not spam given feature are depenent**
p('You |won, lottery, for, 1million,notspam) *
p('won| lottery, for, 1million,notspam) *
p('lottery| for, 1million,notspam)*
p(for| 1million,notspam)*
p( 1million|notspam)

is this correct way of finding the probability of the X considering its events are dependent on each other ? Should the spam/notspam also be included in the dependence part ?

What is the probelm in finding the above 2 probability and why is it so hard that naive has to pitch in and make the features as independent inorder to calculate probability .


texts: How to estimate a user's gender based on what apps the user download?

texts: I have a dataset of what apps users downloaded and I try to estimate these users' gender based on what apps they downloaded, using machine learning algorithm. However, what kind of features of the apps should I focus on? As far as I know, app category plays a big role. Do Google Play and Apple App Store estimate users' gender based on what apps they download?


texts: How to version data science projects with large files

texts:
I am working on a project with large data files (~300MB). I want to version my work along with the data files so that it is always available online. I tried using git-lfs but it has a 1GB/month bandwidth limit, beyond which you're blocked for a month.
What are versioning tools using by data scientists for projects that use >100MB data files (both static and generated)?


texts: Use YOLO v4 to detect smaller items than what it was trained on

texts: What is the best way to use Yolov4 to predict smaller objects (that looks exactly the same) using a model trained on bigger objects without retraining the network? Which parameters in the config file should I be change?


texts: Training with data of different shapes. Is padding an alternative?

texts:
I have a dataset of about 1k samples and I want to apply some unsuspervised techniques in order to clustering and visualization of these data.

The data can be interpreted as a table of a spreadsheet and unfortunately it doensn't have a very defined pattern of structure. The number of table lines varies, but not the columns.

The data is structured like this:

sample 1:

{
  "table1": {
    "column1": [
      "-",
      "-",
      "-"
    ],
    "column2": [
      "2017-04-16 10:00",
      "2017-04-16 10:00",
      "2017-04-16 10:00"
    ],
    "column3": [
      "-",
      "-",
      "-"
    ],
    "column4": [
      "name X",
      "name Y",
      "name Z"
    ],
    "column5": [
      "0",
      "0",
      "0"
    ],
  }
}

sample 2:

{
  "table1": {
    "column1": [
      "-",
      "-",
      "-",
      "-",
      "-",
      "-",
      "-",
      "-"
    ],
    "column2": [
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00"
    ],
    "column3": [
      "-",
      "-",
      "-",
      "-",
      "-",
      "-",
      "-",
      "-"
    ],
    "column4": [
      "name A",
      "name Z",
      "name B",
      "name X",
      "name C",
      "name D",
      "name E",
      "name F"
    ],
    "coumn5": [
      "",
      "",
      "3",
      "1",
      "0",
      "3",
      "0",
      "0"
    ]
  }
}

These samples comes from alarms generated by a systems that collects informations from a lot of nodes (these nodes are named as "name A", "name B"...). My objective is to transform these data into a matrix (n_samples x n_features) to apply clustering and visualization algorithms.

How can I work with these data for unsupervised training? Is padding a way forward for this problem? If so, how can I apply the padding on this case?


texts: How Object oriented programming approach can be utilised in data science domain?

texts:
I have been asked many times that why you are not using classes, objects, and object oriented concepts while coding?
What I do?

Write python scripts to fetch data from different sources, apply
pre-processing code, create necessary variables, and take these
refined variables to modeling process.

How I do?

Write necessary functions to carry-out specific task, all the
functions will be called one-by-one as top down, procedural approach.
In case of different modules, I keep them into different python files,
and then simply import them.

In the current process, I never found a motivation to use OOPs concepts.
Can you suggest how can I improve my coding practice, and what are the flaws I am putting into my current architecture which can be fulfilled by OOPs?


texts: Build a model to classify given string/text input

texts:
I need to build ML/NN model to classify/predict a given string pattern. Sample training data looks as shown in the image. Input will be the string in the column "Id Number", i need to tell to which class it belongs to in column "Id Type".

How do i move forward in building a model for text classification?
How to convert string to digit for using embedding in keras?


texts: Format for time series data with non-trivial sampled data

texts:
I have a data stream that I would like to share with some data scientists.

It is a regularly captured time series with some fields that are simple scalars, booleans. Each sample has a UTC time and fractional seconds since start of capture.

Also captured is a 3d spline that varies in size. These splines are also regularly sampled.

Additionally, there are some other multi dimensional fields as well ([x,y,z,pitch,yaw,roll]).

For simple data sets I would normally use CSV. However due to the nature of the more complex data I need a more appropriate format.

What are my options for formats that would allow easy loading into Matlab or other common data science tools?


texts: How can I compute the ideal variance threshold value for my data?

texts: I have a dataset that contains n features scaled between [0,1]. I would use an unsupervised feature selection algorithm (variance thresholding). How can I compute the threshold value?


texts:
I have trained Text classifier using simpleTranformer.ai I am struggling to save  and load the model in docker container. Please let me know how can I save the trained model and then load it into different environment smoothly.
I am using this library to : https://simpletransformers.ai/ to train a text model using these commands
 model = ClassificationModel('xlmroberta', 'xlm-roberta-base',use_cuda=cuda_available, num_labels=78, args={'learning_rate':1e-5, 'num_train_epochs': 1,'train_batch_size':256,'eval_batch_size':1048, 'n_gpu':4, 'reprocess_input_data': True,
'overwrite_output_dir':True, 'overwrite_output_dir': True})

model.train_model(train_df)

I am saving the trained model using pytorch function:
torch.save(model, 'classifier')

But its showing error of some missing files when I tried to load this model from different virtual machine. So, I am looking for best alternative to save and load the simpleTransformer model.


texts:
I have built my network and would like to see how the activation of a particular layer change after each epoch of training. For example, as code shown below, I want to see the activation values of "act_layer_1". What should I do so that I can see the activation values? Thanks in advance!

W1 = tf.get_variable('W1', [n_input,n_hidden_1], initializer = tf.contrib.layers.xavier_initializer(seed = 1))
b1 = tf.Variable(tf.constant(0.0, shape=[n_hidden_1], dtype=tf.float32), trainable=True, name='b1'),
act_layer_1 = tf.nn.relu(hidden_layer(x_image, W1) + b1)
drop_layer_1 = tf.nn.dropout(act_layer_1, keep_prob)


texts: Using neural networks with jumps in stock returns

texts:
I am using an LSTM network to analyse stock return patterns. A problem is that, there is usually huge jumps in stock returns but if you are only using the trading data, the jumps would seem pretty random. (For example, the jumps from SEC ruling against or in favor of a company.)

Thus, if the neural network learns too much from the jumps, the results would not generalize well. One might cap the returns or use auto encoders. What are some other methods to regularize such jumps and limit the changes the jumps cause to the network?


texts: How can I detect the frame from video streaming that contains the clearest shot of a graffiti on city walls?

texts:
I am working on a graffiti detection project. I need to analyze data stream from a camera mounted sideways on a vehicle to identify graffiti on city walls and notify authorities with the single best capture of graffiti and its geolocation, etc.

I am trying to use a ResNet50 model pre-trained on ImageNet using transfer learning for my graffiti image dataset. The classification will be done on an edge device as network connectivity may not be reliable.

Suppose I have a series of frames that have been detected to contain graffiti, as the vehicle goes past it, but I only need to report one image (so not all frames containing graffiti in the series). How can I do that?

Ideally, I would like to report the frame where the camera is perpendicular to the wall. Why perpendicular? I think that images containing the graffiti when the camera is perpendicular to the wall will more clearly show the graffiti.


texts: Unsupervised Text Classification with Python: Kmeans

texts:
I am working on a project to build a text classifier of questions being asked. There are no labels provided in my data so I have chosen to go with an unsupervised approach. This solution needs to read a new question, and determine which category of classification it falls into
I have been working on a KMeans model, but despite everything I try I just cannot get any decent results. I have even hand selected a data set, cleaned it further by hand to make it as straightforward as possible, and my clusters are still overlapping, and I am getting a low silhouette score of 0.26015371238537227 (this is the highest I've been able to get). My raw data isn't very great to work with, but I still don't understand how it performs so poorly when I hand select/clean the best examples to use as a test. I have included my code for kmeans section below.
Does anyone have any suggestions? I'm a kind of at a loss here. Are there any other good ways to classify unlabeled text? I'm also relatively new to this type of work and this is my first ML based project too so I'm sure there is some knowledge gap to account for as well. I'm about ready to give up on the unsupervised approach and do some manual work to label the data myself and build a supervised model. Figured I'd check here first to see if all hope is lost or not. Any help is greatly appreciated!
    data = pd.DataFrame(text)
    data.columns = ['KM']
    tfidf = TfidfVectorizer(max_df=0.80, min_df=5, max_features=10000)
    text = tfidf.fit_transform(data['KM'].values.astype('U'))

    kmeans = KMeans(n_clusters=8, init='k-means++', max_iter=300, n_init=10,
    random_state=20)
    kmeans.fit(text)

    clusters = kmeans.predict(text)

    pca = PCA(n_components=2)
    two_dim = pca.fit_transform(text.todense())

    scatter_x = two_dim[:, 0]
    scatter_y = two_dim[:, 1]

    plt.style.use('ggplot')

    fig, ax = plt.subplots()
    fig.set_size_inches(20,10)

    cmap = {0: 'green', 1: 'blue', 2: 'red', 3: 'orange', 4: 'black', 5: 'purple', 6:
    'yellow', 7: 'pink'}

    for group in np.unique(clusters):
        ix = np.where(clusters == group)
        ax.scatter(scatter_x[ix], scatter_y[ix], c=cmap[group], label=group)

    ax.legend()
    plt.xlabel('PCA 0')
    plt.ylabel('PCA 1')
    plt.show()

    order_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]

    terms = tfidf.get_feature_names()
    for i in range(8):
        print("Cluster %d:" % i, end='')
        for ind in order_centroids[i, :10]:
            print(' %s' % terms[ind], end='')
        print()

    score = silhouette_score(text, labels=kmeans.predict(text))
    print(score)


texts: How does personalized machine learning work?

texts:
Many services (such as Netflix, Amazon, and Google Search, Apple's Siri) are said to get better by learning the 'habits' of their users. As I understand, they somehow create a customized machine learning model for each individual because a generic model would not work well. However, I would like to know how this can be achieved in practice. For the sake of discussion, suppose I am developing a new massage chair that changes its "massage recipe" based on the respiration pattern of its user. I suppose (and I might be wrong here) that the respiration pattern varies from person to another but that the variations in a person's respiration pattern indicate what kind of message the person would like to have (again, these suppositions are wrong but they are presented for the sake of discussion). I have questions on how these models are customized. Specifically:

How would I train the model for my hypothetical device?
How would customize the said model? Do big companies (Amazon et.al) create a person-specific model for every user? And if this is the case, wouldn't this be impractical (Google serves billions of people; therefore, having billions of models would be hard to maintain)
In the case of Google for example, when setting a new android phone for the first time, the user is requested to repeat three-times "Ok Google" to train their voice recognition model. Is this small sample of sound enough to create the voice recognition model? Or is the sample used to customize an existing generic model? And if so, how is this done?
Is there any references (blog, paper, etc...) that discuss this topic in details?


texts: Is there a python package that includes decision tree structures that can be used with a genetic algorithm?

texts:
I'd like to use decision tree / forest but I need to use a special objective function that can't be differentiated and hence I can't use XGBOost etc. That leaves a genetic algorithm where I use the fitness function to incorporate my special objective.
Other then building everything from scratch, are there any python packages that will allow me to do this easily?


texts:
In a paper, it mentioned: ANN, RNN, and LSTM NN are optimized to contain three hidden layers with 1000 hidden units in each layer.

I would like to model the RNN model in Keras. But my code fails in an error!

My code:

model=Sequential()
model.add(SimpleRNN(1000,input_shape=(320,15),activation='relu'))
model.add(SimpleRNN(1000))
model.add(SimpleRNN(1000))
model.add(Dense(1600))

Error:

    ValueError                                Traceback (most recent call last)
<ipython-input-49-ff01ce62eb30> in <module>()
      1 model=Sequential()
      2 model.add(SimpleRNN(1000,input_shape=(320,15),activation='relu'))
----> 3 model.add(SimpleRNN(1000))
      4 model.add(SimpleRNN(1000))
      5 model.add(Dense(1600))
.....
....
...
..
.

ValueError: Input 0 is incompatible with layer simple_rnn_2: expected ndim=3, found ndim=2

How can I code for the RNN model which is optimized to contain three hidden layers with 1000 hidden units in each layer?

Thank you so much


texts:
I have heard that Random Forest and other tree based machines apply some kind of implicit feature selection.

My Question is: Does this also apply for machines like the SVM? As far as I understand is the choice of the support vectors also some kind of feature selection.


texts: What are the ways to merge data from different sources? Is there any software framework?

texts:
Is there any software library or framework that allows merging data from a different source on regular basis? As I understand it should store and support a different kind of rules for cleaning and reformatting data from a particular source as it is the only way from my point of view.

Exmaple:

Source 1: New Great Sneakers NGS1
Source 2: ngs1 sneakers

So it should be merged after applying rules: convert to lower case -> find some pattern 'NGS*' -> be sure that a number after pattern 'NGS' is the same.

It is some kind of naive way to solve the problem. Is there a better way to perform such task on regular basis?


texts:
I am reading this paper "Sequence to Sequence Learning with Neural Networks" http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

Under "2. The Model" it says:

  The LSTM computes this conditional probability by first obtaining the fixed dimensional representation v of the input sequence (x1, . . . , xT ) given by the last hidden state of the LSTM, and then computing the probability of y1, . . . ,yT′ with a standard LSTM-LM formulation whose initial hidden state is set to the representation v of x1, . . . , xT:

I know what an LSTM is, but what's an LSTM-LM? I've tried Googling it but can't find any good leads.


texts: Shaping data for ConvLSTM for many-to-one image model

texts:
Ultimately, I am trying to obtain a binary segmentation mask for an image sequence.

I have n number of image sequences, each with 500 greyscale images of size 256px by 400px. Each of these sequences is matched to a single segmentation mask, or image of shape 256px by 400px. This gives:

input shape : (n, 500, 256, 400, 1)
output shape: (n,   1, 256, 400, 1)

Understandably, and as explained in this blog post it is necessary to reshape my input array into sequence fragments (time steps) because LSTMs prefer sequences smaller than sequence_length = 400 (my GPU cant handle anything larger than 100 anyway). So, with this in mind, I have chosen a sequence "fragment" length of 50, or 50 time steps. Without overlap this brings the shape of a single input sample:

from:      (500, 256, 400, 1)
to  : ( 10,  50, 256, 400, 1)

In my mind it made sense to feed a single sample through the ConvLSTM2D at a time with a data generator, to have:

input shape : (10, 50, 256, 400, 1)
output shape: ( 1,  1, 256, 400, 1)

I thought this meant that the network would associate the 500 images with a single output image ie. remembering the state across the 10 fragment sequences of 50 images each in order to result in a single image output and computing the loss function only when the network has seen all 500 images for example. It did not make complete sense to me when I did it and now I have proven it to make even less sense because I have gotten the following error:

 ValueError: Input arrays should have the same number of samples as target arrays. Found 10 input samples and 1 target samples.

So I am thus a bit stuck with a flawed understanding of Keras's LSTMS and how to get this ConvLSTM2D to manage these image sequences in the way that I want ie. how I might be able to get a many-images (of a fairly long sequence) to one-image model to work. I can simply duplicate the output image for each of the sequence fragments so that there is a dimension match but this is not the problem I am trying to solve because the segmented image is a summary across the entire sequence, and not across sequence fragments.


texts:
I don't understand how to form my dataset from activity(logins etc.) and characteristic(location, age etc.) raw user data.

Ultimately, each row of the training set will have N activity features for a certain period, M characteristic features and a binary outcome - churn or not after the end of this period.

My problem comes from defining the period and the number of rows per users.

The options I see are the following:

Define period from start of user lifetime, 1 week for example. Then each row is 1 user (activity and characteristics) and outcome is whether they churned in week 2 or not.
Break down a user's lifetime into periods. Predict all users every day on the data from their last week. Let's say user has 2 weeks lifetime. Training data will be:

data_week_1, not churn

data_week_2, churn

Looking for any advice or links related to the viability of these or other methods of dataset formation.


texts: How to get back stationary data from non stationary data after performing diff operations in var (time series)?

texts:
I have applied var model on multiple features ,but var model accepts the stationary data so i have converted non-stationary data to stationary data by doing diff() ,given the data to var model.
what my question is ,is their option get back  data normal data from stationary to non-stationary  in time series?


texts: Removing irrelevant faces from a photograph

texts:
I try to do clustering over a collection of photographs from parties/weddings/etc... (where each photograph would usually contain multiple faces and large background crowds).
I use the face_recognition library in Python that uses a CNN internally to detects the faces in the photograph and extract their embeddings.
The problem is that when I then use HDBSCAN to cluster similar faces together, I would get a lot of noise, because the face recognition library would capture a lot of "background" faces that I wouldn't necessarily want to capture.
I tried to think about a way to filter "background" faces from each photograph, I came up with some metrics for each face, such as brightness (the mean of the HSV value over the entire face  region), blurriness (the variance in the laplacian over the entire face region) and the size of the face over the size of the photograph, I picked a few individual photographs and found out that by playing with some thresholds for these metrics, I can indeed filter out unwanted background faces.
Knowing this, would you advise me to manually label all the photographs (select the wanted faces while ignoring the background ones) and then train some sort of a classifier?
What kind of classifier would you use for that and how large of a dataset would I need?


texts:
I saw a similar question, but I think my problem is something different.
While training, the training loss and the validation loss move around one number, not decreasing significantly.
I have 122707 training observations and 52589 test observations with 55 explanatory variables and one dependent, One CONN1D with 24 filters, 2 Lstm years with 24 units and one dense layer. I've added a dropout rate of 0.2 between the layers. Total parameters 13417.
Seems like my model is not learning at all. Does it mean that the dataset is not a good representation of the specific problem? Should I increase the number of epochs? I use Adam optimizer with default learning rate.

Adding additional info:
I am trying to predict next hour air pollution based on previous value air pollution concentration, previous hour meteorological data as temperature, windspeed etc. Day, Hour and Month are also included and encoded with One Hot Encoding. Additionaly the wind degree is decomposed according to its sin and cos components. Previously I tried normalization of data but it didn`t seem to give any difference. Haven't try any other models. Here is the model:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv1D(filters=24, kernel_size=3,
                  strides=1, padding="causal",
                  activation="relu",
                  input_shape=[None, 55]),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.LSTM(24, return_sequences=True),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.LSTM(24, return_sequences=True),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(1),
tf.keras.layers.Lambda(lambda x: x * 1000)
])

Somewhere I saw Labda layer after Dense layer in regression. I noticed that adding Lambda layer at the end speeds up the learning. I multiply the output with 1000 because it is the maximum value for the variable I want to predict.


texts: Feature selection for data with both continuous and categorical features?

texts:
I am working on a classification problem with 4 ordinal classes to predict, labelling/predicting samples as either a number from 1-4. My training dataset has 284 features by ~40,000 samples and I am looking to explore feature correlation and variance and relate that to using a filtered feature selection method. I've been looking to learn from this guide: https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/

However, my dataset has predominantly continuous features, but it does have ~5 categorical features. How do I consider both types in my statistics and feature selection method? Is it common to partition the features and explore correlation separately (e.g. select out my categorical feature and use Chi-square, then select my continuous and use ANOVA?). Or should I be transforming either my continuous or categorical variables into the other and applying one statistical method to all features?

I aim to be systematic and explore different statistics and selection methods, but I am not sure how I should be accounting for the continuous and categorical variables together or apart. I am new to machine learning (with a biology background) so any help or resources to learn more would be appreciated.


texts: I am combining several vectors, where each vector is a certain kind of embedding of some object. Since each embedding is very different (some have all components between $[0, 1]$ some have components in the range of around 60 or 70 etc.) I want to rescale the vectors before combining them. I thought about using something like min-max rescaling, but I'm not sure how to generalize it to vectors. I could do something of the sort - $\frac{v-|v_{min}|}{|v_{max}|-|v_{min}|)}$ but I don't think it makes much sense. Is there some generalization of min-max scaling for vectors, or generalization of some other kind of scaling?


texts:
I'm new to the machine learning and working on a supervised classification problem. I used  discretization  process to transform continuous variables into discrete variables. So I followed this article to implement it. But when repeat same process with same values it generate different boundary values. Can anyone explain about it?

X_train, X_test, y_train, y_test = train_test_split(train[['tripid', 'Hour', 'is_FairCorrect']],train.is_FairCorrect , test_size = 0.3)

tree_model = DecisionTreeClassifier(max_depth=2)
tree_model.fit(X_train.Hour.to_frame(), X_train.is_FairCorrect)
X_train['Age_tree']=tree_model.predict_proba(X_train.Hour.to_frame())[:,1]

pd.concat([X_train.groupby(['Age_tree'])['Hour'].min(),
           X_train.groupby(['Age_tree'])['Hour'].max()], axis=1)


texts: Identifying most informative (sub)words/vectors that help classify a sample

texts:
I am classifying text using fastText which is a word2vec library that can also create vectors for character level n-grams and I have successfully trained a binary classifier.

Now I’d like to see what words or subword n-grams are the most predictive of a class for the two classes (e.g. if classifier sees a word forest or a subword res then that might be a strong indication that the document has label Nature, but if it sees word “and” then that is probably not very informative for this classification task).

Therefore, I guess the question could be phrased as:

Given vectors representing words and subwords and a trained fastText classifier, what would be the best way to get a list of e.g. top 10 most informative words and subwords for deciding which class a sample belongs to?

Even though I’d be glad if you could make specific suggestions that consider my current setup with fastText, I’m also open to the more general solution suggestions.

Thanks


texts: What is the intuition behind transposed conv layers being able to upscale images?

texts: I was reading the ZF Net paper and it used the term Deconvnet on some searching it seems this is the wrong term and rather we use transposed convolutions instead. I understood how transposed convolutions work but I still don't get how does this show the information stored in a particular layer or the features captured by a particular layer


texts: Data leakage when setting class_weight to tackle imbalanced time series data?

texts:
I'm using a random forest classifier from sklearn to predict whether a stock's return for the next period is greater than a certain threshold (say -2%), so negative is 0 and positive is 1, a binary classification problem. As stock returns are usually centered around 0, setting the threshold large and negative will create imbalances in the observations. In addition, in my application, having false positive is much more costly than having false negative. After doing some research, setting the class_weight variable can tackle this, a good way to start would be inverting the historical ratio between 0 and 1. For example if the ratio between 2 classes are respectively 1:10 then:
class_weight = {0:10.0, 1:1.0}

However, my question is whether setting the class_weight variable like this is equivalent to data leakage when I train the model on past data, but the classes' weights (1:10 as above) are only known at the time of training, which would not be available at all data points up to t-1 (t is the time of training). If this is in fact data leakage, would it be more appropriate to punish false positives on an observation basis with the cost of false positive being the inverse of the ratio between the 2 classes using historical data up until that point only?
Thanks for your help.


texts: How to find out the percentage of contribution of a variable for another variable/feature?

texts:
I have an e-commerce dataset for a month consisting of five columns where the fifth column is "Revenue". The other four columns are different factors for the revenue such as "Page-Views","Visitors","Bounces" and "Click Through Rates". Each row is basically a particular day of the month(hence 30 rows in total).

So, if I would like to find out the contribution/importance of each factor for the revenue, will correlation values between the variables be sufficient or are there better metrics ?


texts: Extracting individual emails from an email thread

texts:
Most of the open source datasets are well formatted i.e each email message is separated well like the enron email dataset. But out in the real world it is highly difficult to separate a top email message from a thread of emails.

For example consider the below message.

Hi,

Can you offer me a better discount.

Thanks,
Mr.X
Customer Relations.

---- On Wed, 10 May 2017 04:05:16 -0700 someone@somewhere.com wrote ------

Hello Mr.X,

Does the below work out. Do let us know your thoughts.

Thanks,
Mr.Y

Sales Manager.

Now the reason why we want to split the emails is because we want to do sentiment analysis. When we fail to split the email then the results will be wrong.

I searched around and found this very comprehensive research paper. Also found an implementation by mail gun called mail gun talon. But unfortunately it does not work well for certain kind of patters.

For example when the second message in the email thread breaks like

---------- Forwarded message ----------

instead of the above

---- On Wed, 10 May 2017 04:05:16 -0700 someone@somewhere.com wrote ------

My question is many people who are trying to do such stuff would have definitely faced such problems, but yet the area remains pretty shady. Is there any pretty solid implementation of the paper or something else that splits email pretty well.


texts:
I'm new to data science. I'm trying to increase the time-series length for a special calculation. In the original time-series I have 20 weekly reports and I want to increase the amount of occurrences to 200. Is it ok just to use the range from the first date value to second date value between two neighbor dates?

For example if I have 1 for the first date and I have 5 for the second one, is it fine to populate the empty space between them with 2, 3, 4. Or do I need to use more advanced techniques here?


texts:
I've heard sum encoding mentioned as a method of encoding categorical variables, but I haven't been able to find a clear explanation of what it actually is.
I found the following explanation on Towards DataScience:

Just like OneHot except one value is held constant and encoded as -1
across all columns.

But this confused me as it doesn't involve any summing.
Is it really just?:
Value      Variable 1. Variable 2.
Red            1            0
Blue           0            1
Green         -1           -1


texts: XGBoost: How to set the probability threshold for multi class classification

texts:
I am using the XGBoost for classification of text data.
There are the 3 different classes in the training dataset.

classifier = Pipeline([
    ('features', FeatureUnion([
        ('text', Pipeline([
            ('colext', TextSelector('Issue')),
            ('tfidf', TfidfVectorizer(tokenizer=Tokenizer, stop_words='english',
                     min_df=.0025, max_df=0.25, ngram_range=(1,3))),
            ('svd', TruncatedSVD(algorithm='randomized', n_components=300)), #for XGB
        ]))
    ])),
    ('clf', XGBClassifier(n_estimators=300,max_depth=3, learning_rate=0.1)),
#    ('clf', RandomForestClassifier()),
    ])

As per the classification results, the class for which prediction probability is highest is assigned to the data point.
For example, if the prediction probability for class A is .67, then that data point is assigned to that category(Class A).

predictionProbability=classifier.predict_proba(X_test)

But the requirement is to assign the data point to the 4th Category that is "UnDetermined" if the prediction probability for the data point does not differ much among the classes. For example, if the prediction probability of the datapoint for three classes is .32,.33,.35, then can we mark it as Undetermined.
So that the user can review the undetermined category and assign that to the appropriate class.

But I am not sure how to set the cutoff probability for multiclass classification problem?  PLease let me know how to identify the cutoff probability. So that i can mark the data points with prediction probability less than cutoff probability as "Undetermined"


texts: Python: How to make model predict in a generalized manner using ML Algorithm

texts:
I have a dataset for which I am trying to predict target variables.

Col1    Col2    Col3    Col4    Col5
  1      2       23      11     1
  2      22      12      14     1
  22     11      43      38     3
  14     22      25      19     3
  12     42      11      14     1
  22     11      43      38     3
  1      2       23      11     4
  2      22      12      14     2
  22     11      43      38     3

I have provided a sample data, but mine has thousands of records distributed in a similar way. Here, Col1, Col2, Col3, Col4 are my features and Col5 is target variable. Hence prediction should be 1,2,3 or 4 as these are my values for target variable. I have tried using algorithms such as random forest, decision tree etc. for predictions.
Here if you see, values 1 and 3 are occurring more times as compared to 2 and 4. Hence while predicting, my model is more biased towards 1 and 3 whereas I am getting only less number of predictions for 2 and 4 (Got only 1 predicted for policy4 out of thousands of records when I saw the confusion matrix). I am planning to do feature extractions and try to see how my model behaves, but is there any way to make my model predict in a generalized way where it can generalize some values 2 and 4? I just read of undersampling and oversampling concepts, but I am beginner here, is there any good approach I could try? Should I split the training dataset based on Col5? I am implementing this in python using pandas.


texts:
I was looking at this repository (https://github.com/jeffheaton/jh-kaggle-util) and was wondering if there's a better approach to creating a scalable machine learning pipeline for data science projects?

I feel inspired to try and make a pipeline that incorporates the structure outlined in the repo above, but I'm having trouble following the code implemented by the author. I also found it difficult to find more examples of scalable machine learning pipelines online.

Would love to have a second opinion on this subject and recommendations for any relevant resources. Thanks!


texts:
Working on an classification problem with images at the pixel level using either keras(tf) or pytorch. The images are all 10,000 pixels wide and high. To elaborate there are two kinds of pixels in this image "x" pixels and "y" pixels. It is impossible to tell them apart in a single image. However, the pixels for "x" change in a very specific way over time and the idea is to identify "x" as soon as possible. I have a stack of images (timestep of 50 images) involving these pixels. Additionally, "x" pixels are about 25% of the image while "y" pixels are the remaining 75%. The final image I have is a black and white image in which constitutes the labels named key.jpg. Basically "x" is black and "y" is white this is what is used to identify "x" and "y" pixels over time. The idea is to have the nn determine if a pixel is an "x" or a "y" pixel by the 5th timestep, that way I can use the nn on a new series of images that have "x" and "y" pixels in a different configuration have it classify them enabling the creation of a matching key.jpg for them.

I theorize that I can solve this with a cnn or an rnn (lstm), but I'm not sure of how to implement them. The general idea I have stack the images above one another in the order of time (an array) then select 500 random pixels following them over time with 25% of them being "x" and 75% of them being "y". Use 200 pixels as training, 200 as validating and 100 for testing (while keeping the pixel ratios).

The problem I have here is that I cannot envision what my next step should be like and I'm facing two problems.

I'm clearly working with time series data so a cnn might not be the right choice.
I cannot figure out how to use this in an rnn. Logically I can envision using a single time series, but since each pixel's tuple/array is an individual time series and I'm not sure how to proceed.

Any help/suggestions would be greatly appreciated.


texts: Vowpal Wabbit Online Normalization -- Possible to parallelize?

texts:
Vowpal Wabbit (VW) uses online normalization as explained here [1].

When running VW with multiple workers, workers synchronize their models with an AllReduce at the end of each epoch.

Is it possible or is there any code/paper that explores the idea of doing online learning with multiple workers in a parameter server setting?

[1] https://arxiv.org/abs/1305.6646


texts: samples for different objects with unique labels

texts:
Lets say i have a samples taken from veterinary hospital , one of my feature will be the type of  the animal and some other features such as fever , size,symptom etc.. , my labels are the medicine given to that animal.

if each medicine is unique to that type of animal (Medicine A should be given to animal A , and there are no Animal B that taken Medicine A) . What will be the cons and pros to building one classifier for the whole data set vs split classifier for each animal (since there wont be valid generalization between animals)


texts:
What exactly is the difference between model-based boosting and gradient boosting? For an intro to model-based boosting see https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf
It seems to me that both terms are equivalent. However, both are used in various literature...


texts:
I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, I have noticed that some of the entries contain the following symbols: "-", "0x000b", "0xc0a8", and many others that it is difficult to track them and see how many of these unformatted data are present, specially when the database is so big. Is there a way to take the whole dataframe, spot all the possible unformatted and error data and substitute them by NaN, to treat them later as missing values?

Thanks in advance!


texts: What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn?

texts:
I've found two different approaches online when using the Elbow Method to determine the optimal number of clusters for K-Means.

One approach is to use the following code:

distortions_2.append(sum(np.min(cdist(data,
                                      kmeanModel.cluster_centers_,
                                      'euclidean'),
                                axis = 1)) / data.shape[0])

Another is to use inertia_ from sklearn.cluster.KMeans:

distortions_3.append(kmeanModel.inertia_)

When I plot the results (using the same random states) both give different results but I'm not sure what the differences are, can anyone help?

Edit: If I replace the normalisation factor / data.shape[0] with squared **2 as suggested below, then I still don't get the same as for the inertia plot:

distortions_2.append(sum(np.min(cdist(data,
                                      kmeanModel.cluster_centers_,
                                      'euclidean'),
                                axis = 1)) ** 2)

Using squared just makes the plot a little smoother, but definitely not the same as using intertia_, I'm just not quite sure how inertia_ is calculated and what it means.


texts: How to plot/visualize clusters in scikit-learn (sklearn)?

texts:
I have done some clustering and I would like to visualize the results.

Here is the function I have written to plot my clusters:

import sklearn
from sklearn.cluster import DBSCAN
from sklearn import metrics
from sklearn.preprocessing import StandardScaler

from sklearn.cluster import DBSCAN
from sklearn import metrics
from sklearn.datasets.samples_generator import make_blobs
from sklearn.preprocessing import StandardScaler

def plot_cluster(cluster, sample_matrix):
    '''Input:  "cluster", which is an object from DBSCAN,
       e.g. dbscan_object = DBSCAN(3.0,4)
"sample_matrix" which is a data matrix:
X = [
    [0,5,1,2],
    [0,4,1,3],
    [0,5,1,3],
    [0,5,0,2],
    [5,5,5,5],
    ]
        Output: Plots the clusters nicely.
    '''
    import matplotlib.pyplot as plt
    import numpy as np

    f = lambda row: [float(x) for x in row]

    sample_matrix = map(f,sample_matrix)
    print sample_matrix
    sample_matrix = StandardScaler().fit_transform(sample_matrix)

    core_samples_mask = np.zeros_like(cluster.labels_, dtype=bool)
    core_samples_mask[cluster.core_sample_indices_] = True
    labels = cluster.labels_

    # Black removed and is used for noise instead.
    unique_labels = set(labels)
    colors = plt.cm.Spectral(np.linspace(0, 1, len(unique_labels)))
    for k, col in zip(unique_labels, colors):
        if k == -1:
            # Black used for noise.
            col = 'k'

        class_member_mask = (labels == k)  # generator comprehension
        # X is your data matrix
        X = np.array(sample_matrix)

        xy = X[class_member_mask & core_samples_mask]

        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=col,
                 markeredgecolor='k', markersize=14)

        xy = X[class_member_mask & ~core_samples_mask]
        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=col,
                 markeredgecolor='k', markersize=6)

    plt.ylim([0,10])
    plt.xlim([0,10])
#        plt.title('Estimated number of clusters: %d' % n_clusters_)
    plt.savefig('cluster.png')

The function above is copied almost verbatim from the scikit-learn demo here.

Yet, when I try it on the following:

dbscan_object = DBSCAN(3.0,4)

X = [
    [0,5,1,2],
    [0,4,1,3],
    [0,5,1,3],
    [0,5,0,2],
    [5,5,5,5],
    ]

result = dbscan_object.fit(X)
print result.labels_

print 'plotting '
plot_cluster(result, X)

...It produces a single point.  What is the best way to plot clusters in python?


texts: What sort of models work for unsupervised reinforcement learning, or is deep learning the way?

texts:
I'm setting out on an adventure to automate the statuses of the lights around my home. The lights should have different brightness in the range [0, 100] depending on some factors, which I have boiled down to:

Time till sunset
Sun Intensity
Overcastness
Possibly whether it's a weekend or not

Now, obviously, I can implement an algorithm that computes the answer as some equation involving these metrics, but I would like it if, if over time, me manually adjusting the lights in the house because they aren't right should affect the way the algorithm computes. As far as I see it, this is clearly reinforcement learning: when I have to change the lights manually, this is a negative reinforcement for the predictor.
I know that Deep Learning is one way of performing reinforcement learning, but traditionally, from what I understand, it needs a lot of initial data, which I don't have. Realistically, you could expect a data-point every 10 minutes but that's still not a lot. Obviously, in this setting, accuracy isn't so critical, because it's just a light, but I'm unsure about whether this is a reason to avoid Deep Learning. The problem is, other than having done a course on Deep Learning during my degree, I haven't had any training on any classical machine learning approaches or algorithms. I'm aware of some, but don't know enough about them to know which ones would be applicable to me.
So my question is, for reinforcement learning problems like this one, is there a good model that can be chosen that is not Deep Learning, or is it the case that the lack of data doesn't really have an impact?


texts: Lightweight execution of Spark MLLib models

texts:
I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API.
I have a code set up to train this model every day to get the most up-to-date model. (the real-world use case is that I am predicting vehicle unloading times, and my model must always be recently trained since the characteristics of the vehicles and locations change over time.)
However, when I use my model for inference I want to do it from an existing Java codebase. I need fast inference for individual data points, not batch inference. So I need a lightweight low latency way of calculating inference.
One solution that I have found is to export the parameters of my MLLib model into PMML or another representation, and then re-implement the inference code in pure Java without any of the boilerplate that would come with Spark. So I have a function like this:
    public static double predict(double[] parameters) {
        double prediction = bias + weights[0] * parameters[0] +  ....;
        return prediction;
    }

where the array weights are updated every day with values exported from a trained MLLib model.
However, this seems inefficient, since the logic of the model design is now reproduced unnecessarily in the Java code, and also restricts me to the kind of simple models that can be represented this way. For example, I could not do this for random forest regression.
Ideally, I would like a lightweight inference call to Spark MLLib within Java, without any of the overhead of Spark sessions, servers, APIs, URLs, etc.
Is there such a lightweight Spark Java function that I could use which allows the inference of a single instance? I don't imagine it being an uncommon situation that somebody needs the benefit of Hive and parallel processing for training, but just fast and simple inference with minimal overheads.


texts:
In the last layer of CNNs and MLPs it is common to use softmax layer or units with sigmoid activation functions for multi-class classification. I have seen somewhere, I don't remember where, that softmax is used whenever the classes are mutually exclusive and the layer with units containing sigmoid activation function are used in tasks with multiple labels, e.g. recognizing animals in an image which can contain numerous animals. Am I right? Is there any other deduction for distinguishing between them?

I have seen here and here but they do not contain what I want.


texts:
According to xgboost paper, regularization is given by:
$$\Omega(f) = \gamma T + \lambda || w||^2$$
where $\gamma$ is the complexity of a tree (i.e., number of leaves in the tree).
The parameter gamma in xgboost library, on the other hand, controls the minimum split at a node in order to proceed. Hence, is the $\gamma$ in the equation above used by xgboost software package? I could not find any reference to it.


texts: Implement and train YOLO 3 with Opencv and C++

texts:
I want to implement and train YOLO 3 with my dataset using Opencv and C++. I can't find an example to start with, or a tutorial to explain how to train YOLO with my own data. All of the tutorials I found are in Python and don't use Opencv. Do you have any examples, or an explanation of how to code an object detector with YOLO 3, opencv with C++?

I find this tutorial.

The code for this tutorial is here.

but I can't find the C++ code. I think it exists only with Python.


texts: What is the state-of-the-art method/algorithm to extract Keywords from text?

texts:
What is the state-of-the-art method/algorithm to extract data from text without regarding the language of the text?

The methods I am reading about are Rake, Yake or using LTSM Networks to identify Keywords.

Are there any ressources on what Amazon, Google or Facebook use for this?

Can anybody here share his experience regarding Keyword extraction?


texts: Calculating Rank Ordering Error Metric for implicit recommendation

texts:
I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula:

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}$$

This is the same formula that Datacamp defines as the appropriate error metric for implicit recommendation engines, except they call it "Rank Ordering Error Metric". I'm implementing the system in Spark, so I've defined a test dataset to try things out:

test_df = spark.createDataFrame(
  [
    ("A", "Fish", 1, 1),
    ("A", "Dogs", 2, 2),
    ("A", "Cats", 3, 3),
    ("A", "Elephants", 4, 4),
    ("B", "Fish", 1, 1),
    ("B", "Dogs", 2, 2),
    ("B", "Cats", 3, 3),
    ("B", "Elephants", 4, 4)
  ], ["Customer", "Item", "ImplicitRating", "PredictedRating"]
)

rankWindow = Window.partitionBy("Customer").orderBy(desc("PredictedRating"))
test_df\
  .withColumn("RankUI", percent_rank().over(rankWindow))\
  .withColumn("RankUIxRating", col("RankUI") * col("ImplicitRating"))\
  .show()

and the output is:

+--------+---------+--------------+---------------+------------------+------------------+
|Customer|     Item|ImplicitRating|PredictedRating|            RankUI|     RankUIxRating|
+--------+---------+--------------+---------------+------------------+------------------+
|       B|Elephants|             4|              4|               0.0|               0.0|
|       B|     Cats|             3|              3|0.3333333333333333|               1.0|
|       B|     Dogs|             2|              2|0.6666666666666666|1.3333333333333333|
|       B|     Fish|             1|              1|               1.0|               1.0|
|       A|Elephants|             4|              4|               0.0|               0.0|
|       A|     Cats|             3|              3|0.3333333333333333|               1.0|
|       A|     Dogs|             2|              2|0.6666666666666666|1.3333333333333333|
|       A|     Fish|             1|              1|               1.0|               1.0|
+--------+---------+--------------+---------------+------------------+------------------+

I'm effectively modelling a perfect prediction here by setting the Predicted "Rating" to match the ImplicitRating. My problem is that plugging those values into the formula above gives me...

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}} = \frac{0.0+1.0+1.\dot{33}+1.0+0.0+1.0+1.\dot{33}+1.0}{4+3+2+1+4+3+2+1} = \frac{6.\dot{66}}{20} = 0.\dot{33}$$

Given the paper is explicit in saying that lower values of $\overline{\text{rank}}$ are better and that they achieved values as low as ~ 8%, I'm confused as to how that can be given my experience in this experiment.

What am I doing wrong?


texts:
I'm having difficulty searching for the original published paper proposing k-Means as an algorithm. I have been inspired to find it as reference for similar work, inspired by this TowardsDataScience article.
I have tried wiki's references and Google Scholar but with no luck.


texts: What is the best way to use Early Stopping in an ensemble (stacking) model?

texts:
I have a training and a test dataset. I would like to use the output of Model A in an ensemble model. However, I would like to use early stopping.
Usually, I would create Model A for each K-fold (on training) and predict the OOF to create the meta-model dataset. Then I would repeat the methodology but for hyperparameter tuning the 2nd layer of the stacking model, trained with the meta-model dataset.
Finally, I train the meta models on the whole training set, train the 2nd layer of the stacking model on the whole training set, and predict\evaluate the results for the test set.
How would the final step work if I am using early-stopping? should I just create the meta-model dataset without CV and without predicting the OOF? or should I forgo early stopping and find the right hyper-parameters\training steps.


texts: Is gradient descent slower for finite differences?

texts:
In gradient descent, we updated each parameter $\theta_i$ in the direction which minimizes a function $f(\theta_1,\theta_2,\dots,\theta_N)$ by doing $$\theta_1 \leftarrow \theta_1 - \alpha \frac{\partial f}{\partial \theta_1}(\theta_1,\theta_2,\dots,\theta_N)$$
$$\theta_2 \leftarrow \theta_2 - \alpha \frac{\partial f}{\partial \theta_2}(\theta_1,\theta_2,\dots,\theta_N)$$
$$\vdots$$
$$\theta_N \leftarrow \theta_N - \alpha \frac{\partial f}{\partial \theta_N}(\theta_1,\theta_2,\dots,\theta_N).$$

If we have $N$ parameters, then it will involve $N$ evaluations.

If possible, obviously, we want to use this analytical form of the gradient.

But we could, of course, approximate it using finite differences,
$$\theta_1 \leftarrow \theta_1 - \alpha \frac{f(\theta_1+\varepsilon,\theta_2,\dots,\theta_N)-f(\theta_1,\theta_2,\dots,\theta_N)}{\varepsilon}$$
$$\theta_2 \leftarrow \theta_2 - \alpha \frac{f(\theta_1,\theta_2+\varepsilon,\dots,\theta_N)-f(\theta_1,\theta_2,\dots,\theta_N)}{\varepsilon}$$
$$\vdots$$
$$\theta_N \leftarrow \theta_N - \alpha \frac{f(\theta_1,\theta_2,\dots,\theta_N+\varepsilon)-f(\theta_1,\theta_2,\dots,\theta_N)}{\varepsilon}$$

This involves $N+1$ evaluations, which is negligible if $N$ is large, as is the case with deep neural networks.

Obviously, the analytical gradient is preferable since it is more accurate, and it avoids numerical problems due to $\varepsilon$ being too large or too small.

My question is: I have heard people claim that using gradient descent with the analytical derivative is faster than approximating the derivatives. Is this true? To me, it sounds like it only avoids one extra evaluation.

EDIT: Practical example:

Let's say we have
$$y=\sigma(ax)$$

$$z=by$$
where $a$ and $b$ are the parameters we want to optimize.

Gradients

Using the gradients, we would have $\nabla=\left(\frac{\partial z}{\partial a},\frac{\partial z}{\partial b} \right) = \left(bx\sigma(ax)(1-\sigma(ax)), \sigma(ax)\right)$

(I hope I didn't make a mistake there. $\frac{\partial z}{\partial a}=\frac{\partial z}{\partial y}\frac{\partial y}{\partial a}=bx\sigma'(ax)=bx\sigma(ax)(1-a\sigma(ax))$.)

For each back-propagation, we would need two evaluations for $a$ and for $b$: $\nabla(a,b)$, so that we can do...

These are the computations required:

$a \leftarrow a-\alpha bx\sigma(ax)(1-\sigma(ax))$

$b \leftarrow b-\alpha \sigma(ax)$

Finite differences

We have the differences $$\Delta=\left((z(a+\varepsilon,b)-z(a,b))/\varepsilon, (z(a,b+\varepsilon)-z(a,b))/\varepsilon \right)$$

$z(a,b)=\color{blue}{b\sigma(ax)}$

$z(a+\varepsilon,b) = \color{blue}{b\sigma(ax)} + b\sigma(\epsilon x)$

$z(a,b+\varepsilon) = \color{blue}{b\sigma(ax)} + \varepsilon\sigma(ax)$

These are the computations required:

$a \leftarrow a-\alpha b\sigma(\varepsilon x)/\varepsilon$

$b \leftarrow b-\alpha \sigma(ax)$

The terms in blue disppear in the graph because they are subtracted by themselves.


texts: Collaborative Social Network Visualization

texts:
I do movement building work for Effective Altruism (http://en.m.wikipedia.org/wiki/Effective_altruism), and would like to level up our growth strategy. It occurred to me that a social network visualization tool which allowed us to strategically find and recruit new influencers/donors would be mega useful. I'd love to find something (preferably free), similar to InMaps, which would allow us to:

Combine all of our social media connections into a single map
Easily see who the superconnectors are
Weight each person by their degree of social influence (perhaps some function of things like Klout score * amount of social media connections * number of Google mentions, etc)

Does such a thing exist? If not, is anyone interested in pro bono work for an amazing cause? =)

Disclaimer: I am a data science noob, so preferably the solution would be one with a nice GUI and minimal involvement of R or Python.


texts: Best regression model to use for sales prediction

texts:
I have the following variables along with sales data going back a few years:

date # simple date, can be split in year, month etc
shipping_time  (0-6 weeks) # 0 weeks means in stock, more weeks means the product is out of stock but a shipment is on the way to the warehouse. Longer shipping times have a siginificant impact on sales.
sales # amount of products sold

I need to predict the sales (which vary seasonally) while taking into account the shipping time. What would be a simple regression model that would produce reasonable results? I tried linear regression with only date and sales, but this does not account for seasonality, so the prediction is rather weak.

Edit: As a measure of accuracy, I will withold a random sample of data from the input and compare against the result.

Extra points if it can be easily done in python/scipy

Data can look like this

--------------------------------------------------
| date           | delivery_time| sales          |
--------------------------------------------------
| 2015-01-01     | 0            |10              |
--------------------------------------------------
| 2015-01-01     | 7            |2               |
--------------------------------------------------
| 2015-01-02     | 7            |3               |
...


texts: Does Homoscedasticity applies only for linear regression models?

texts:
In statistics, a sequence of random variables is homoscedastic if all its random variables have the same finite variance. This is also known as homogeneity of variance (Wikipedia).
Is this assumption only for linear regression models?
If not, is there any relative method is available to check for non-linear models?


texts: Can linear regression or any other models be used to judge if Y increases as X increases?

texts:
I have a database like:

Site    X    Y
S1      1    1.5
S1      1    1.3
S1      2    1.7
S1      1    1.1
S1      4    5.9
S2      3    4.0
S2      2    2.5
S2      4    9.1
S2      4    9.2
S2      1    2.3

I need to find if $Y$ increases as $X$ increases for every site. In other words, Bigger $X$ corresponds to bigger $Y$.

I know linear regression might suit this problem. But please take a look at the following graph:

Figure 1 is not what I want because small $X$ corresponds to big $Y$. However, Figure 2 is what I want. When I use a linear regression model and RMSE as a measure, it cannot tell the difference between Figure 1 and Figure 2.

Another thing about my database is: $X$ are like levels, which are the same to all the sites. But the $Y\text{'s}$ of every sites are different. For example, for one site, $X=1$ and $Y=20$ means $20$ is a small value because it corresponds the lowest level of $X$. But for another site, $Y=15$ and $X=6$ means $15$ is the highest value because $X$ is the highest level.

So, my problem is: for every site, I need to use a linear model or any other algorithms to judge if $Y$ increases as $X$ does. Then, I need to use a measure to select some sites.


texts: Set for building ROC curve and and choosing logistic regression cut-off

texts:
I'm building a logicsic regression classifier for binary classification. I have trained it and going to choose a cut-off value using ROC curve.

But what set to use for it: training or validation?


texts: Understand clearly the figure: Illustration of a Convolutional Neural Network (CNN) architecture for sentence classification

texts:
I am studying the blog: Understanding Convolutional Neural Networks for NLP. It is very good blog.

One thing I can't understand clearly about this blog. As the figure Illustration of a Convolutional Neural Network (CNN) architecture for sentence classification as following:

I want to ask:

I know the region sizes(2,3,4) is like 2-gram, 3-gram, 4-gram word, but what’s the meaning of number filters? Here is 2 filters for each region. Why in the author's code about sentence classification is the number of filters defined to 128? Could you give examples to explain the meaning of the number of filters? for example using the sentence of ‘I like this movie very much’ would be great.

2) I understand the height of region size (4) is 4, but in the figure, the height of region(2, 3) are 5 and 6 respectively, I don't know why? I think the height of region is 2 and 3.


texts: Adding a new custom column in a python data frame

texts: I need to add a new column in a python data frame that has the dates of January in a column. Each date is repeated 24 times in the column without any intervention. The total number of entries will thus be 31*24 = 744 entries. Can you please help me code this part ?


texts:
I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. Accuracy increases till some point (approximately half) of first epoch and then decreases. Loss also first decreases and then increases rapidly (it is not Nan). What can be reason of this trend ?
I noticed same trend when I tried to run it with roberta-base. But any such trend was not noticed when I trained with distilbert, accuracy goes on increasing in this case.
Here is graph for accuracy VS steps:

[EDIT]
File containing running output of the model:
https://drive.google.com/file/d/1r5AWftyHTLf5sqtgWnQm_4lqB84UrJex/view?usp=sharing


texts: Does it make sense to visualize data with a linear relationship using tSNE?

texts:
I have used tSNE several times to visualize high dimensional data for cluster analysis, and it has always worked quite well when data falls into clusters.

However, does it make any sense to use tSNE to visualize the independent variables in a dataset which has a linear relationship $y = X \theta + b$, where $y$ is the target value and $X$ are the independent variables (b is the bias)?

As I understand tSNE, it is used more for capturing local structures in high dimensional data, so it is probably not suited for visualizing data for linear regression - is this a wrong assumption?


texts: Delete strings with a specific last character in tibco spotfire

texts:
I am using tibco spotfire and I have columns with data like this "0001V", "0002V", "0003N". I know data with the 'N' at the end is not valid so I need to delete the numbers which have that. I think I should use "limit data using expression" but I don't know how to write the expression. So it should be something like "if X ends in 'N' delete X"... Could you help me please?

Thanks!


texts: Cook's distance, altering diagnostic plot in R,?

texts:
I have a linear model in R, named Q1fit. I am interested in the plot of the residuals vs leverage, with Cook's distance. When I run

plot(Q1fit, which=(5))

I get

There is no Cook's distance line visible here. Does that mean the Cook's distance goes beyond the axes, and I need to change the limits? If so, how can I do that without giving an error (as my attempts keep doing).


texts:
I have implemented PCA algorithm and I understood it very well but still I have some questions. My code is below and it's very simple implementation.

import numpy as np

x = np.loadtxt('CCPP', delimiter=',')
row, column = x.shape

# Mean normalization
for i in range(column):
    x[:,i] = (x[:,i] - x[:,i].mean()) / (x[:,i].max() - x[:,i].min())

sigma = x.transpose().dot(x) / row
u, s, v = np.linalg.svd(sigma, 0)
z = x.dot(u[:,:3]) ## new features

new_x = z.dot(u[:,:3].transpose()) ##reconstruction

First Question

As you can see above my sigma variable is

  x.transpose().dot(x) / row

It's giving to me an nxn matrix (n is number of features). But sigma's formula is $$\Sigma = \frac{1}{n} \sum_{i=1}^n x^{(i)} {x^{(i)}}^T$$

Why there is a summation symbol in formula? I mean, if I use this sigma formulation then sigma is going to be a number, not a matrix. I have to get nxn matrix, right? So is my sigma implementation correct? or am I missing something about the formula?

Second Question

When we are reconstructing X (at the bottom in the code), should new_x equal to my first X? I mean, I reduced dimension of the data set, then I reconstructed it, original dataset and reconstructed dataset must be the same, right? This is my second question.

Third Question

This one is easy. Should I use data compression for each of my dataset which has 1000, 100000 or more features? I mean, can I always use it? Is it a good choice to use it every time?


texts: Tensorflow MirroredStrategy() looks like it may only be working on one GPU?

texts:
I finally got a computer with 2 gpus, and tested out https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html and https://github.com/tensorflow/models/tree/master/tutorials/image/cifar10_estimator and confirmed that both gpus are being utilized in each(The wattage increases to 160-180 on both, Memory is almost maxed out on both, and GPU-Util increased to about 45% on both at the same time).

So I decided I would try out tensorflow's MirroredStrategy() on an exitsting neural net I had trained with one GPU in the past.

What I don't understand is that the wattage increases in both, and the memory is pretty much maxed in both but only one GPU looks like it is being utilized at 98% and the other one just chills at 3%.  Am I messing something up in my code?  Or is this working as designed?

strategy = tensorflow.distribute.MirroredStrategy()
with strategy.scope():
    model = tensorflow.keras.models.Sequential([
        tensorflow.keras.layers.Dense(units=427, kernel_initializer='uniform', activation='relu', input_dim=853),
        tensorflow.keras.layers.Dense(units=427, kernel_initializer='uniform',activation='relu'),
        tensorflow.keras.layers.Dense(units=1, kernel_initializer='uniform', activation='sigmoid')])
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    model.fit(X_train, y_train, batch_size=1000, epochs=100)

nvidia-smi:

Fri Nov 22 09:26:21 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp COLLEC...  Off  | 00000000:0A:00.0 Off |                  N/A |
| 24%   47C    P2    81W / 250W |  11733MiB / 12196MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp COLLEC...  Off  | 00000000:41:00.0  On |                  N/A |
| 28%   51C    P2    64W / 250W |  11736MiB / 12187MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2506      C   python3                                    11721MiB |
|    1      1312      G   /usr/lib/xorg/Xorg                            18MiB |
|    1      1353      G   /usr/bin/gnome-shell                          51MiB |
|    1      1620      G   /usr/lib/xorg/Xorg                           108MiB |
|    1      1751      G   /usr/bin/gnome-shell                          72MiB |
|    1      2506      C   python3                                    11473MiB |
+-----------------------------------------------------------------------------+


texts: True Negatives for Named Entity Recognition

texts:
Are True Negatives always zero for a Named Entity Recognition task because TN in NER would mean a not entity being classified as Not entity?
Actual Entity
[Microsoft Corp.] CEO [Steve Ballmer] announced the release of [Windows7] today
Model prediction
[Microsoft Corp.] [CEO] [Steve] Ballmer announced the release of Windows7 [today]
What will be the TNs in the above example?


texts:
I have been using the mlp library on the sklearn on some unsupervised data. I have been getting training set scores between 0.7 - 0.9 .However my test set scores have been really low. I assume this is due to over fitting and have read and understood that regularization (to help smooth weight/bias) and early stopping would help prevent this issue. However I am finding it hard to implement it, could someone help me with an implementation or guide me on the right path with a suitable link.

(The tutorial on regularization for clf on sklearn did not really help me as I found it to be confusing with little explanation)


texts: How to decide who to market? Clustering or Decision Tree?

texts:
I am working with a dataset that has enough observations and ~ 10 variables,

half of the variables are numeric
another half of the variables are categorical with 2-3 levels (demographics)
one ID variable
one last variable that has sales value, 0 for no sale and bill amount for sale

Using this information, I want to understand which segments of my customers to market. I am using R for code but that's not relevant here. :)

I am confused about which statistical technique to use. Since I want to identify what types of customers I want to acquire and build my campaigns, I was initially thinking of using k-means clustering, i.e. go with unsupervised learning.

However, considering I know who purchased and how much they purchased from the sales value figures, I believe it's worth including this information and have decided to go with predictive modeling instead. Here regression will only tell the importance of variables, but I am interested in nodes (for example, I want rules that can support my marketing campaign such as age 45+, from LA region...etc.) so decision tree would be a better fit.

What are your thoughts? Clustering or decision tree? Or actually something else?


texts: Difference between rand index and adjusted rand index?

texts: I am unable to understand what is the adjusted term in ARI. The expected index term in the ARI is from a prior. Kindly explain


texts:
I want to do some data augmentation on MNIST and therefore must manually label the Y set. This can be automated if MNIST is unshuffled, however mnist.load_data() appears to be returned in a shuffled state. How do I load this with Keras or TensorFlow in an unshuffled state?

UPDATE: It's trivial to unshuffle manually with:

perms=np.where(Y==0)
x[perms] #all X that are 0

Nevertheless would be nice if Keras had a way to import MNIST directly without shuffle.


texts: What's the correct objective function for cosine similarity of two vectors to be 1 or 0?

texts: The representation learning model produces vectors for objects. I want the cosine similarity of some vector pairs to be (close to) 1, some to be 0. What objective function should I use? MSE as training a regression model?


texts: Cannot see what the "notation abuse" is, mentioned by author of book

texts:
From Sutton and Barto, Reinforcement Learning: An Introduction (second edition draft), in equation 3.4 of page 38.

  The probabilities given by the four-argument function p completely characterize the dynamics of a
  finite MDP. From it, one can compute anything else one might want to know about the environment,
  such as the state-transition probabilities (which we denote, with a slight abuse of notation, as a threeargument function

  $p(s^{'} | s, a) \dot{=}Pr\{S_t=s^{'} | S_{t-1} = s, A_{t-1}=a\} =
\sum_{r\in{R}}{p(s^{'},r|s,a)}$

The author mentioned, with a slight abuse of notation.
where is the abuse in the notation please? I didn't see anything that is not proper.

Thank you.


texts: What should return doc.ents if the doc have no entities, in spacy?

texts:
I want to answer this question: "How many sentences contain named entities given a doc?"
and I have this piece of code as solution

nb = 0

for sent in list(doc.sents):
    if sent.ents:
        nb = nb+1

print(nb)

But I am a little confused about how the if-statement work;
consider that the actual sentence(sent) does not have any entities(ents), so sent.ents will return an empty list I suppose and I can't understand why an empty list is considered as a false statement because the program when executed does not enter the if.


texts: Handle unbalanced data by implementing Edited nearest neighbors, smote and Tomek links in r?

texts:
Imbalanced data is a big problem in classification problems. I have a binary classification problem with imbalanced data.
I have researched and found that a possible method of dealing with this is preprocessing the data before plugging it into a model. My data looks like this example using iris:
# Create unbalanced binary dataset (Remove 'versicolor')
 data <- iris[-c(95:130),]
 data <- data[data$Species == c("setosa","virginica"),]

 # Show that data is imbalanced
 table(data$Species) / sum(table(data$Species))

How can I implement

Edited nearest neighbors
Smote
Tomek links

in R in order to preprocess the data before fitting a model? Which packages could I use particularly for a binary classification problem rather than the multiclass case?


texts: Which version of spectral clustering in sklearn library?

texts: Which version of spectral clustering is implemented in sklearn library? Is it Shi, Malik or Ng, Jordan, Weiss from this tutorial? In sklearn user guide, both versions are mentioned in reference. From the source code, it is not trivial to understand what is implemented as the authors used some tricks for code optimization.