texts:
i m doing dimensionaly reduction using PCA. I don't understand why some dataset already had a target ad example in Iris database or other like this (https://scikit-learn.org/stable/datasets/index.html)
had a target_names useful when plot data.

Ad example in iris database choose like color= target_names to do this

i found this code online in example.

fig = plt.figure(figsize = (8,8))
ax = fig.add_subplot(1,1,1)
ax.set_xlabel('Principal Component 1', fontsize = 15)
ax.set_ylabel('Principal Component 2', fontsize = 15)
ax.set_title('2 component PCA', fontsize = 20)
targets = ['Iris-setosa', 'Iris-versicolor', 'Iris-virginica']
colors = ['r', 'g', 'b']
for target, color in zip(targets,colors):
    indicesToKeep = finalDf['target'] == target
    ax.scatter(finalDf.loc[indicesToKeep, 'principal component 1']
               , finalDf.loc[indicesToKeep, 'principal component 2']
               , c = color
               , s = 50)
ax.legend(targets)
ax.grid()

I m trying to do the same with my dataset, where i dont have a target

i have a table in this way

User    Movie
        0 1 2 3 4
      0 2 0 5 0 0
      1 0 1 1 0 0
      2 0 5 5 5 0

for each user i have all film and him review (0 if don't review)

When plot my graph i tried to do this

fig = plt.figure(figsize = (16,12))
ax = fig.add_subplot(111)

ax.scatter(a,b, alpha = 1)
plt.title('Method: PCA')
#plt.savefig('PCA.png', dpi = 300)
plt.show()

but i really don't know where is my target. I try to add another column to my dataset with gender for user for cluster user for gender but give an error of the shape because i just have a 2 gender but 6000 user.

I really don't know to apply a target in this way,
Same suggestion?


texts:
Normally, I am familiar with precision and recall evaluation metrics but as you know recall@k and precision@k are different things and used in ranking evaluations especially recommendation systems.
I checked many sources, I understood everything I could not understand a point.
One more thing,
Every source is different between each other in terms of calculation ( 1 , 2, 3, 4 )
let's get this example

I'll give you an example so you can explain to me
Let's say we have 5 users. We are trying to give location recommendations for the next visit to each user.  We are analyzing users' historical check-in data and we are giving recommendation for the next visit.
User 1 is visiting: Museum1, Park1, Night Club, "?" (What is next)
We are trying to find the next visiting locations. Let's say our ground truth "Restaurant"
How can I calculate precision@5 and recall@5?
Extra:
This youtube video is explaining very good (go to 51:45 on video)
What is 5-6 relevant item means? If we are giving recommendation it should be just 1 item that is gonna be relevant for the user. They are trying to make a movie recommendation but they are saying we have 5 relevant movies. What is that mean?


texts: How to make an MNIST classifier work with blank images?

texts:
I am trying to make a Sudoku solver and for the image recognition I trained a CNN but the problem that I am facing is that I don't know how to make it see a clear distinction between numbers and blank images.
(My neural network is trained for MNIST data set only)
For example in a Sudoku like this :

I want the classifier to classify the blank spaces as "0"
Here is what I have already tried:
import numpy as np
import cv2
from PIL import Image
import pytesseract
import matplotlib.pyplot as plt
from tensorflow import keras

#open the image
img = Image.open(r'D:\\D_Apps\\Sudoku Solver\\image\\1_9Tgak3f8JPcn1u4-cSGYVw.png').convert('LA')

#take only the brightness value from each pixel of the image
array = np.array(img)[:,:,0]

#invert the image (this is how MNIST digits is formatted)
array = 255-array

#this will be the width and length of each sub-image
divisor = array.shape[0]//9

puzzle = []
for i in range(9):
    row = []
    for j in range(9):
        #slice image, reshape it to 28x28 (mnist reader size)
        row.append(cv2.resize(array[i*divisor:(i+1)*divisor,
                                    j*divisor:(j+1)*divisor][3:-3, 3:-3], #the 3:-3 slice removes the borders from each image
                              dsize=(28,28),
                              interpolation=cv2.INTER_CUBIC))
    puzzle.append(row)

model = keras.models.load_model(r'C:\Users\Ankit\MnistModel.h5')

template = [
    [0 for _ in range(9)] for _ in range(9)
]

for i, row in enumerate(puzzle):
    for j, image in enumerate(row):
        #if the brightness is above 6, then use the model
        if np.mean(image) > 6:
            #this line of code sets the puzzle's value to the model's prediction
            #the preprocessing happens inside the predict call
            template[i][j] = model.predict_classes(image.reshape(1,28,28,1) \
                                                   .astype('float32')/255)[0]

print(template)

(read about this in a blog)
This algorithm took the average brightness and checked if the other cells had brightness less than 2 and classified them as blanks.
But this algorithm dose not work if the image did not have a white background for example the output for this image:

the output was :
[[7, 7, 0, 7, 7, 7, 1, 7, 1], [2, 1, 8, 1, 8, 1, 1, 0, 8], [7, 7, 1, 8, 8, 1, 7, 7, 7], [7, 1, 1, 1, 1, 1, 1, 0, 1], [7, 7, 1, 1, 7, 8, 1, 7, 7], [8, 7, 8, 1, 7, 7, 7, 4, 9], [7, 1, 1, 1, 1, 0, 7, 8, 7], [7, 4, 7, 8, 8, 7, 7, 7, 4], [2, 7, 7, 7, 8, 0, 4, 7, 7]]

What can I do to improve this? Should I retrain my model to work with other image colors?
Or Should i retrain the model with blank spaces? If so how can I find the dataset?
I have done a lot of research but can't find a clear answer to my questions


texts: How to train the same RNN over multiple series?

texts:
I have multiple separate time series and would like to train the same LSTM network on them. How to do in this situation? I can't just concatenate timeseries (along time), because I am afraid network will be confused by jumps at the points of concatenation.

How to overcome?


texts: Parameters for OnlineLogisticRegression function in Mahout

texts:
Can anyone tell me where do I find any documentation for parameters like:

-stepOffset

-alpha

-decayExponent

in an OnlineLogisticRegression function in Mahout?

I am interested in what do they change in calls like this one:

int FEATURES = 10000;
OnlineLogisticRegression learningAlgorithm = new OnlineLogisticRegression(20, FEATURES, new L1())
                        .alpha(1).stepOffset(1000).decayExponent(0.9).lambda(3.0e-5).learningRate(20);


texts: ValueError:invalid literal for int() with base 10: 'No''

texts:
Trying to do a binary classification and have the class names as string but when it gives me the error mentioned in the title. I tried to give integer labels as 0 and 1, the code works perfectly fine. I tried all the suggestions from stackoverflow regarding the same error but nothing is working.Here is the code snippet:

labels = ["No", "priorlocaltx"]
#labels = [0, 1]
keyword_identifiers_no = ['surgery', 'radiotherapy', 'brachytherapy', 'Newadjuvant', 'neo', 'Adjuvant', 'Mets at first diagnosis', 'M1HSPC']
keyword_identifiers_yes = ['no prior tx', 'BCR', 'Neoadjuvant', 'neo', 'Adjuvant', 'Hormone refractory', 'Mets', 'No mets']

cancer_df.insert(2, 'Result', 0, allow_duplicates = False)
#print(cancer_df.head())
for i in range(1, len(cancer_df)):

    for word in keyword_identifiers_no:
        if word in cancer_df.iloc[i]['Patient Segment(s)']:

            cancer_df.at[i,'Result'] = labels[0]
            #print(cancer_df['Patient Segment(s)'].dtype)

    for word in keyword_identifiers_yes:
        if word in cancer_df.iloc[i]['Patient Segment(s)']:
            cancer_df.at[i,'Result']= labels[0]

    cancer_df.to_csv('Result1.csv')


texts:
I'm working on a dataset with more than 2000 features. Most of the features contain both numerical values and categorical features. For example, in the feature represents how long has the user been living in the current address, the value could be numbers or some letters which mean the value cannot be obtained due to some reasons.
I don't know how to process these features. If they are just pure numerical or categorical values, things will much easier. But since they are mixed, I'm really confusing. Could anyone give me some advice?
Update: I may not express clearly that it is not a dataset include both numerical features and categorical features. I mean in one feature, there are both numerical values and categorical values.
For example: (here M, C, T mean that because of different reasons, no exact values can be found)
TOTAL INCOME
3000
5000
M
8000
C
4000
T


texts:
How to create anchor-positive and anchor-negative pair for feature X in a signature data set for training Siamese network?

Im have a cedar signature data set with 55 peoples signatures(classes) with  24 original image(instance) and 24 forged image(instance ) .
so for person 1(class) there will 24 original signature images and 24 forged signature images(48 instance )

i need to create (Anchor-positive pair ,anchor -negative pair ) to represent the feature X((img1,img2),(img1,img9)) .
feature y(1,0) 1-> similar 0-> non similar

Anchor-> original image
positive-> this also represent the  original image but with minute variations
Negative->  forged image

The below code does not give the exact pair for X labels and y labels because the negative pair is not chosen randomly.

positive -positive pair -> must be of image1 and image1 original
instances (total - 24 pairs )
i. positive -negative pair -> must be of image1 and image1 negative
instances(NEED only 12)
ii. positive -negative pair -> must be of image1 and any other random
image  instances(NEED only 12)
final positive-negative pair will be 24

below line takes image1 and pair with other instances of image1 and labels it as 1 -> anchor-positive  pair

z1, z2 = digit_indices[d][i], digit_indices[d][i+1]

below line takes image1 and pair with other instances of other images eg:image9 and labels it as 0 -> anchor-negative pair

inc = random.randrange(1, nb_classes)
        dn = (d + inc) % nb_classes
        z1, z2 = digit_indices[d][i], digit_indices[dn][i]

the issue there is always perfect 50/50 positive-negative pair and model is doing random choice prediction (i.e) accuracy score is 50% always

please suggest how to create a meaningful pair of positive-positive images and positive-negative images

def create_pairs(x, digit_indices, nb_classes):
"""      x:         X_train, array of array of all train samples.
    digit_indices:  List of an array, length = no of classes; each sublist consists of train sample indices
                    belonging to that particular class index/class
"""
""" Positive and negative pair creation.
    Alternates between positive and negative pairs.
"""
pairs = []
labels = []

print ('\n\n')
print ('X_train shape:  ', x.shape)
print ('Digit_indices shape:    ', np.array(digit_indices).shape)
print ('Length of digit indices:    ', len(digit_indices))
print ('No of classes:  ', nb_classes)
print ('\n\n')

n = min([len(digit_indices[d]) for d in range(nb_classes)]) - 1
for d in range(nb_classes):
    for i in range(n):
        z1, z2 = digit_indices[d][i], digit_indices[d][i+1]
        pairs += [[x[z1], x[z2]]]
        inc = random.randrange(1, nb_classes)
        dn = (d + inc) % nb_classes
        z1, z2 = digit_indices[d][i], digit_indices[dn][i]
        pairs += [[x[z1], x[z2]]]
        labels += [1, 0]    #check label based on similarity = 0 or 1
        # labels += [0,1]     #similar pairs = 0 in this case
return np.array(pairs), np.array(labels)


texts: How to use fresh data when target prediction period is long?

texts:
I'm using supervised learning on monthly activity data to predict when a customer buys a particular product. This product is typically bought infrequently and at the moment my target variable is whether the customer buys the product in the next twelve months.

Assume that for every customer I get a set of features every month, $x_1,x_2,\ldots,x_n$. The goal is to use these features to predict whether $y=0$ or $y=1$ ($y$ is 1 if the customer did buy the product in the next twelve months, otherwise it is zero).

However, this creates a dilemma. If I use this approach for $y$ then my freshest training data is twelve months old as I do not know the true value for $y$ for data that is younger than twelve months old. My main question is thus the following: Is there a way for me to make use of newer data in this setting?

Also, I should note that I have tried changing $y$ into: "Does the customer buy the product in the next month?". It works but not nearly as well as the other approach. My data is imbalanced so by allowing the target period to be composed of the following twelve months instead of a single month I get many more positive data points.


texts:
My academical background is in physics and analysis (PDE's), but now I'am reading about data science.

I'm reading about Gaussian Process implementation in Sci-Kit Learn

I could not find a simple explanation about those hyper-parameters theta0, thetaL and thetaU. Can someone explain to me what those parameters are, and how to choose then?


texts: Keras input for multivariate classification with LSTM using current features and previous timesteps features and y values

texts:
I am working on a multivariate binary classification problem. What I want to do is to predict a binary classification given the features at the current timestep and the data (features+real classification) from past timesteps

Keras seems to have a problem with the shape of my inputs so I want to know what I am doing wrong:

X_train = (nb_samples, nb_timesteps, nb_features)
y_train = (nb_samples, nb_timesteps, binary_result)

model = Sequential()
model.add(LSTM(nb_units,
               input_shape = X_train.shape[1:]
              ))
Dense(1,activation='softmax')
model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=ep, validation_data=(X_train, y_train), verbose=2, shuffle=False)


texts: Support Vector Machines (SVM) vs Minimum distance between two convex hulls

texts:
Imagine that we have two sets of seperable points, X and Y, in the plane (R^2) that we want to classify. To find the optimal line (hyperplane) that separates these two sets of points we can:

Run a SVM algorithm
Find the minimum distance line segment between the convex hulls of X and Y and take the line perpendicular to the midpoint of this line.

2 can be done using an algorithm of H. Edelsbrunner in O(logn+lognm) time. But I guess creating the convex hulls of the point sets would be O(nlogn), so overall it should take O(nlogn) time. I do not know the time complexity of the SVM algorithm.
Do both algorithms achieve the same result? If not, why are they different? If yes, why would we prefer the SVM algorithm over 2?


texts: Features selection/combination for random forest

texts:
I am working on using random forest to predict 1 or 0. I have about 20 variables available for modeling. I realized that if I put different variables will have different accuracy/sensitivity/specificity. I am wondering if there is a test or method can tell me which variables combination has the highest accuracy? Or which variable combination has the highest sensitivity and specificity respectively?

Thanks in advance!


texts: Need some tips regarding starting out with the field's specific programming languages, with a heavy focus on data visualization

texts:
I've got rather expansive development experience, but I'm new to data science. I have been trying to familiarize myself with the main concepts and deciding between R and Python to put my time into.

I know you can write Python in R and vice versa, but since data visualization in a beautiful manner would be a very high priority for me in the future and both Python and R use additional packages and libraries for data representation, I'm not entirely sure whether it's possible to use the additional libraries when integrating code into one another.

I would really appreciate it if someone could give me some advice regarding this and also provide me with some general tips about the best approaches, practices and tools when it comes to data visualization.


texts:
Im new to Big Data, so please just dont blast me.

Can you explain me why HDFS works
best when the file blocks are evenly spread across the cluster ?


texts:
My task is to estimate a person's age based on a rgb image of the face of that person. I'm using ResNet-50 to that end. At first stage I trained my net on a dataset which is called WIKI-IMDB (after filtering and expanding my dataset by horizontal flipping it contains ~300k images) and I initialized the weights of the network with the weights of a pre-trained ResNet-50 network which trained on ImageNet. When training on the WIKI-IMDB dataset I calculated the mean of each channel (RGB) of the train set input and subtracted it from each example in the batch (I also used the same mean to subtract it from the validation set which I'm using for early stopping). after I finished training my network on WIKI-IMDB I want to test it on another dataset set which is a benchmark for this task. my new dataset which I'm testing on is called CACD and contains ~160k images which I'm dividing to train set and test set. I need to fine-tune my network on the CACD train set and finally report the metric I'm using on the CACD test set.

My question is: when learning on the new CACD train set, should I subtract the mean of the CACD train set from each example in the batch or should I keep subtracting the mean of the WIKI-IMDB train set?
and if I need to subtract the new mean of the CACD train set, should I use that mean also when evaluating on the CACD test set?


texts: Using Vector Auto Regression for multiple time series at once

texts:
Say I have a dataframe like so:
                       Avg Temp       Avg Precipitation
City         Month

New York       Jan        56               12
               Feb        63               17
               Mar        68               24

Boston         Jan        56               12
               Feb        63               17
               Mar        68               24

I would like to use Vector Auto Regression to forecast Avg Temp and Avg Precipitation for the month
of April. Would I need to create two seperate VAR models, one for each city? Or could I train one model using both cities, then predict seperate values for both?
How would this scale if I had around 200 cities, and possibly hundreds of variables rather than just
Avg Temp and Avg Precipitation? Would a neural network be capable of accomplishing a single model?


texts: Does it make sense to combine PCA with an artificial neural network?

texts:
I have a Dataset of around 200 features. Most of them are categorical and only a few are numerical. It seems that an artificial neural network with an Autoencoder has some problems with that kind and amount of features. Therefore, I thought to use PCA to reduce the dimensions and then apply the Autoencoder afterwards.

Does the combination of a PCA before neural nets makes sense as the neural nets also reduces the information in internal layers? Does anybody has experiences with such a combination?

Edit: There is also an interesting answer in the stats forum.


texts: How to deal with possible data leakage in time series data?

texts:
I have historical consumer data who have taken out a loan at some point in time. The task is to predict if a consumer will default when requesting a loan.

My issue is that for some customer in the data set, historical transactions are only available after the loan was issued. I believe using data after the loan event for prediction will cause data leakage.

This is a subtle leakage because it does not involve using information not available at prediction time. My concern is more about behavioral change when the customer is indebted that create a shift in the underlying distribution.

To test my hypothesis I was wondering if comparing whether the two samples before and after the loan is issued come from the same distribution will be a good approach.

These are my questions:

Is there really data leakage in the scenario I described
If yes, can I test it in any way?
Can a two samples test provide an answer? Which one? Note that the sample is composed of multivariate data
Can I do testing using any machine learning approach? I was thinking of using a Mixture Model to test for instance.

Any suggestion on how to best deal with this problem other than what I suggested will be appreciated.

Thanks


texts: How to count occurrences of values within specific range by row

texts:
I have a data frame of 3000 rows x 101 columns like as follow:

Time   id0  id1  id2     ………… id99

1      1.71 6.99 4.01    ………… 4.98

2      1.72 6.78 3.15    ………… 4.97

.

.

3000   0.36 0.23 0.14    ………… 0.28

Using Python, how could we add a column that counts for each row the number of values (in column id0, to id99) that are within a specific range?


texts: Creating bar plot with frequency distribution based

texts:
I have a dataframe with this kind of structure:

ID | Size | Value
1  | 1    | 10
2  | 2    | 20
3  | 1    | 10
4  | 1    | 10
5  | 1    | 15
6  | 1    | 12
7  | 1    | 20

I want to create bar plot of the distribuition of column "Values" when the "Size" column is equal to 1.

With the current example:

the x-axis should have: 10, 12, 15 and 20
the y-axis should have 1 (for the 12, 15 and 20) and 3 for the 10

I don't have a lot of code. Basically I just created a new dataframe with only the rows where Size=1 and then performed value_counts():

data1 = data.loc[(data['Size'] == 1)]
count = data1['Value'].value_counts()

The variable count now has a Series object with a Value and a counter.

How can I split the value of count in order to have two lists (values and counters) to send to the plotting code?


texts: How would be plotted the logistic functions associated to a multiclass logistic classifier in a X-Y plot?

texts:
Logistic function is well known to be a good binary classificator, as it can be easily shown with this image (let x be the dot product of [x1,x2] and [w1, w2]): enter image description here

I am currently learning how a multiclass logistic classifier works, and this time with x1 and x2 as axes, three classes can be represented as follows (from this link):

According to several sites, both the "one vs the rest" and the multinomial logistic classifiers, would have one logistic regression for each of these 3 classes, with the difference of training them "together" or separately. If this is true, and we assume the next figure to be related to the previous example, I can't figure out how these 3 regressions (as in the first example) would be plotted in there.

As much I could figure, in the "one vs the rest" case, how to draw the logistic function for blue and yellow categories against the rest, but how would it be for the red category against the rest?


texts: R lm(log(y)~x,data) models and predict, need to remember the exp. R2 differences

texts:
I noticed a difference between R and Excel when it comes to trend lines.  Basically I am automating a simple process that someone does.  He/she takes web traffic and finds the best trend line by looking at the R squared to predict what the web traffic will be a few months from now to make sure we will have the servers to handle it.

In R, one can do: lm(log(y)~x,data) but you must "remember" to do exp(predict(...)).  Is there a better predict function that remembers the left hand operations?

Also the R-squared reported for log(y)~x is the r squared for x vs that y's log.  The R-squared against the original y was better.  Is there a nifty function that calculates that?

As you can see from this output, what summary reports as the rsquared is not "correct" while my rsq data accurately shows the best rmse. (See source code below.)

[1] "rmse is 305221.416535481 rsq is 0.749525713697497 summary$r.squared 0.705953722113025"
[1] "rmse is 304961.752311906 rsq is 0.749815047530537 summary$r.squared 0.706163612870978"
[1] "rmse is 318083.254498832 rsq is 0.723564406971294 summary$r.squared 0.723564406971402"
[1] "rmse is 317352.614485029 rsq is 0.724832898371528 summary$r.squared 0.724832898371531"

I want to learn the best way to do this in R...

rmse=function(x,y){
  return( sqrt(sum((x-y)^2)/length(x)));
}
rsq=function(actual,pred){
  return( cor(actual,pred,method=c("pearson"))^2);
}

findBestFormula=function(data,yvar,xvar){
  #This figures out the best Excel-like trend function based on lowest rmse
  data$y=data[,yvar];
  data$x=data[,xvar];
  models=list(lm(log(y)~log(x),data),
              lm(log(y)~x,data),
             lm(y~log(x),data),
              lm(y~x,data));
  preds=list(exp(predict(models[[1]],data)),
    exp(predict(models[[2]],data)),
    (predict(models[[3]],data)),
    (predict(models[[4]],data)));
  rs=c();
  for(i in (1:4)){
    #say(names(models[[i]]))
    rs[i]=rmse(data$y,preds[[i]]);
    print(paste("rmse is",rs[i],"rsq is",rsq(data$y,preds[[i]]),
                "summary$r.squared",summary(models[[i]])$r.squared));
  }
  best=min(rs)
  #say(best);
  for(i in (1:4)){
    if(best==rs[i]){
      return(list(model=models[[i]],pred=preds[[i]],modelNum=i,rmse=best));
    }
  }

}

summary(findBestFormula(weekDayDf,'daily','d')$model)

Any tips on niftier code to write the above?


texts:
I have a column as month in my excel data set, when i load the data in the file, orange changes it to some numerical values.. Help! I want the Month as it is. Tried changing the value type as short or long date in excel too.

The orange loads Dec-16 as 42055.00 something.


texts: Are parametric method and supervised learning exactly the same?

texts:
Parametric methods make explicit assumptions about the functional form of $f$, and supervised learning are pretty the same right? It's a way to build a statistical model for predicting, or estimating, an output based on one or more inputs.

If they are the same, why would they have to create different names. Is there a reason behind this?


texts: Keeping DataFrame in Memory while working - R vs Python in VS Code

texts:
I am considering migrating from R-Studio to Python. So I am just starting Python with VS Code as my editor. My purpose is mainly to analyze data, build predictive models and make such models available via a web API.

When working in R-studio with a large data set, I can run a script to read in the data from say a CSV file, then I can explore the data using the normal statistical approaches. I only need to load the data set ONCE, which is great since loading a large data set often takes some time. And data exploration requires lots of repeated "probes" to view histograms, compare counts etc. So not having to reload the data each time I change a script or run a script to view a plot, is a great advantage.

However, using Python in VS Code, it seems to me that if I load the data frame say in a method at the top of a script, then have a method to draw a graph, then each time I run that script, the data is reloaded. So if I want to draw 20 histograms, each time changing only the title, I need to do the data reload each time I modify and run the script.

Am I missing a feature of Python and VS Code or is my summary correct?


texts: Random selection of variables in each run of python sklearn decision tree (regressio )

texts:
When I put random_state = None and run Decision tree for regression in python sklearn, it takes different variables to build tree each time?
Shouldn't there be only few top variables which should be used to split and should throw me similar trees everytime?
Also, if I use integer for random_state and run the decision tree, it gives me a different tree for each random_state setting. Which tree should be selected in case of so many trees?


texts: How's frequent pattern mining in data mining?

texts:
I am reading a paper Frequent pattern mining:current status and future directions.

From this paper I learned that a lot of people did some interesting jobs in this field. But this review is published in 2007. And as the size of data becomes bigger and bigger, in my opinion, frequent pattern mining which based on sparse data may not compete with Neural Network anymore?

Hope someone can tell some recent news about frequent pattern mining.


texts: How to use 3-Axial acceleration data for CNN?

texts:
Currently, i am using 1-D acceleration by dividing data in 10 seconds segments and then making a test train split into these segments. But i am not sure how to use 3-D data, especially how to make test train split out this data because for each label there will 3 channel input.

EDIT
My dataset looks lie this :

Right now I am only using X-Axis. I am segmenting X-acc in 200 samples and for each sample, I have an activity label. So now I have a number of segments and a label for each of these segments. After that, I am splitting these segments and their corresponding labels in test train split. This all works fine but no i want to use Y-axis and x-axis also. so now i will have three arrays each segmented in 200 samples, now i am not sure how toperform test train split.

Note Activity labels are not shown here, but there are total 6 labels and are one hot encoded.


texts: How to extract all information by using id

texts:
I need to apply classifier algorithm after clustering. Now after clustering I find the id numbers that which id belongs which cluster. I clustered them into 2 cluster.

Now I need to collect those data by using those id. But I don't know how I can collect all information by using those id.

As I use jupyeter notebook and in main data I have no attribute named id and those id assigned jupyter notebook when I load data from main data file.

This is my main data

Here is my code to find which data belongs which cluster.

x = 0.10
i=0
C_i = np.where(labels == i)[0].tolist()
n_i = len(C_i) # number of points in cluster i

# (2) indices of the points from X to be sampled from cluster i
sample_i = np.random.choice(C_i, int(x * n_i))
print (i, sample_i)

and after clustering I find these ids

New addition:

suppose my loading file name is train.
now using train.loc[26] command I get the info of that specific id.

But I need to collect all info into a new data frame like as train dataframe


texts:
I’m doing an “analytics” project in which I have to collect data from small businesses (SMEs) in South America and try to come up with some insights about what are their needs in general. The project’s coach basically wants to collect data from social media and business association websites and do some text analytic on it to see if we can gain some insights.

Ideally, the idea of the project is that given a country and a sector, find what the companies in those sectors are up to and see what are their problems.

This is what I've thought:

Find keywords in the twitter API and facebook API.
Find News about SMEs in a given country and crawl the content.
Find associations web sites. I dis this but the content here is useless most of the time.

The goal of this project is to gain some insights and see if some new products can be customized to these companies.

Thus, my question is: how would you solve a problem like this if you do not have any data? I was also thinking about getting some contact information from a sample of this companies and send them all a survey or something.


texts: Time series forecasting with RNN(stateful LSTM) produces constant values? Is there any solution out there?

texts:
I tried lstm  but results were getting constant.
Same question as in the link


texts: How to measaure the similarity between two series?

texts: I'm confused about how to measure the similarity between two time series with the same length. For example, both time series are 2 hours in length and every 5 minutes a point. I really want to know which Distance Algorithm should I use. I have tried the Euclidean Distance but it didn't work well on this type of data. Should I find some point-to-point distance algorithms?


texts: Error while trying to merge two neural networks

texts:
I'm trying to merge two neural networks with Keras.

The code:

left_branch = Sequential()
left_branch.add(Dense(512, input_shape=(7000,)))
left_branch.add(Activation('relu'))
right_branch = Sequential()
right_branch.add(Dense(512, input_shape=(14012,)))
right_branch.add(Activation('relu'))
merged = Concatenate([left_branch, right_branch])

final_model = Sequential()
final_model.add(merged)
final_model.add(Dense(3, activation='softmax'))
final_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
final_model.fit([np.array(review_matrix), np.array(X_train)], labels,epochs=2, verbose=1)
final_model.save('model.merged')

I get the following error: AssertionError (assert len(inputs) == 1)

I guess the problem comes from the fact that final_model should not be sequential. However, I don't know how I can do  otherwise. In a lot of links, it works with sequential model (for example: https://statcompute.wordpress.com/2017/01/08/an-example-of-merge-layer-in-keras/)

Thanks !


texts:
Im studying the following tutorial on the Keras website and I'm trying to understand how to create a sequence for a Conv1D layer.
This is their method:
TIME_STEPS = 288

def create_sequences(values, time_steps=TIME_STEPS):
    output = []
    for i in range(len(values) - time_steps):
        output.append(values[i : (i + time_steps)])
    return np.stack(output)

Let's suppose we have TIME_STEPS = 3 for example, and a dataset which contains [0, 1, 2, 3, 4, 5].
If we run that method, we will get a sequence structured as well:
0 1 2
1 2 3
2 3 4
3 4 5

If we train our neural network on these sequence we'll get more training values and it is a great choice, right? But when we test it on the test set, should't we avoid to repeat the values? I made my own method:
def create_sequences_test(values, time_steps=TIME_STEPS):
    output = []
    for i in range(len(values) - time_steps):
        output = [values[i: i + time_steps] for i in range(0, len(values) - time_steps, TIME_STEPS)]
    return np.stack(output)

Now, if use the same values I should get:
0 1 2
3 4 5

Should't be this one the correct format for the test set?
So I would train the NN creating the sequence with first criteria and then I would create the test with the second one:
X_train = create_sequences(dataset[0:500, time_steps=3)
X_test = create_sequences_test(dataset[500:600, time_steps=3)


texts: Cat2Vec implementation X = categorical and y = categorical

texts:
I am trying to convert categorical values (zipcodes) with Cat2Vec into a matrix which can be used as an input shape for categorical prediction of a target with binary values.
After reading several articles, among which: https://www.yanxishe.com/TextTranslation/1656?from=csdn
I am having trouble to understand two things:
1) With respect to which y in Cat2Vec encoding are you creating embeddings. Is it with respect to the actual target in the dataset you are trying to predict, or can you randomly choose any y column as target? What would be wise?
2) With respect to the y target, if it should be the actual target you are trying to predict with the model. In my case I am trying to predict a binary target (0/1). Would this still work? As the dimensionality of the target is very limited.
Thank you for helping me!


texts: Should the bias value be added after convolution operation in CNNs?

texts: Should we add bias to each entry of the convolution then sum, or add bias once at end of calculating the convolution in CNNs?


texts: sentence classification with RNN-LSTM - output layer

texts: i have read a few  blogs and papers on the IMDB exercise w.r.t sentiment classification using LSTM's (and at times in conjunction with CNN) but there the output layer can contain just 1 neuron with a sigmoid since the sentiment can either be good or bad. But if i need to use the same technique to classify, say, 30,000 sentences into 20 different labels then what should my output layer look like ? if i am not mistaken i should have the same number of neurons in my output layer as the number of labels i am training the data for (20 in this case) each of which has the same sigmoid. Can you please let me know if this makes sense ?


texts: What model is recommended: I am using text features in a regression and want to interpret coefficients

texts:
I am using the text of comments on a forum to predict how many upvotes it will get. I want to be able to say, "Reviews with X, Y, Z words are more upvoted". So to do this, I want to use text features in a regression. In particular,

What model should I use to maximize interpretability of coefficients?


texts: TypeError: doc2bow expects an array of unicode tokens on input, not a single string

texts:
import re
from nltk.tokenize import sent_tokenize ,word_tokenize
from nltk.corpus import  stopwords
from nltk.stem import WordNetLemmatizer
from gensim.corpora.dictionary import Dictionary
from collections import Counter

 article1 = """Indian landscape is rich and replete with art, but not all of it gets a platform to be exhibited. With the upcoming edition of Hindustan Times Imagine Fest 2019, arts from across India will get a chance to become visible to people, who can also buy these artworks since the concept of this festival is to have all artworks priced below ₹2,00,000. """
 word = word_tokenize(article1)
 islower = [t.lower() for t in word if t.isalpha()]
 dictionary = Dictionary(islower)
 computer_id = dictionary.token2id.get("computer")
 corpus = [dictionary.doc2bow(article) for article in article1]
 print(corpus[4][:10])

  I used all libraries but I cant understand whats the problem, A little guidance would be
  appreciated.


texts: What advantage does Guassian kernel have than any other kernels, such as linear kernel, polynomial kernel and so on?

texts: Guassian kernel is so important in SVM as we know. The parameter gamma is designed for this kind of kernel. My question is what makes Guassian kernel so unique? What advantage does it have over other kernels?


texts:
I am currently struggling to understand how i should train my regression network using keras. I am not sure how I should pass my input data to the network.

Both the input data and the output data is stored as a list of numpy arrays.

Each input numpy array is a matrix which has (400 rows, x columns) Each output numpy array is a matrix which has (x number of rows, 13 columns)

So input dimension is 400 and output is 13. But how do I pass each of these sets within the list to the training?

# Multilayer Perceptron
model = Sequential()    # Feedforward
model.add(Dense(3, input_dim=400, output_dim=13))
model.add(Activation('tanh'))
model.add(Dense(1))
model.compile('sgd', 'mse')

I know that model.fit trains the model given the parameter, how it actually takes in the data seems a bit like magic to me, and how it knows that the columns of matrix A should be mapped the rows of matrix B this has to be done for the whole matrix, appended to the list.


texts: Finding p values and estimates of external variables in facebook Prophet for forecasting in Python

texts:
I am using facebook Prophet for multivariate forecasting which has an objecting of forecasting prices. My target variable is affected by n number of external variables too.

I am using add_regressor of fb prophet library to add these variables  but cant figure out a way to know the strength of these variable in forecasting y.

Can we get a summary which displays pvalues , beta estimates , standard error etc.


texts:
I have a few doubts in backpropagation....

1.) My first doubt is that we can write the cost function as all the neural elements involved from all layers and then we can differentiate with respect to each weight... But this takes a lot of time and hence we use chain rule. Is my understanding right?

2.) In order to calculate the derivative of cost function with respect to a particular weight, We just take the derivative of cost function with the corresponding input neuron multiplied by the derivative of input neuron by that particular weight...  Is this right?

3.) This is a crucial doubt. Delta term which we use is as I understood nothing but derivative of cost function with rest to the INPUT (Z).. IS MY UNDERSTANDING RIGHT?. In order to calculate the derivative of a cost function with respect to input layer L... We need the all the next layers Delta... AND THATS WHY WE CAN CALCULATE EFFICIENTLY USING BACKPROPAGATION... Right?

4.) When calculating the derivative of cost function with respect to input in previous layers,, WHY DO WE TAKE SUMMATION OF THE NEXT LAYERS... WHY?

5.) MY FINAL DOUBT IS WHAT DOES DELTA SIMPLY MEAN? DOES IT MEAN HOW MUCH THE INPUT NEURON IS AFFECTING THE COST FUNCTION?

THANKS AND I AM A BEGINNER.. IS MY FUNDAMENTALS CLEAR OR AM I CONFUSED?


texts: Detecting anomalies from time series correlated variables

texts:
I have a daily set of data that I need to detect if there are any anomalies in it. These are variables related to the stock market so I'd just give an example (totally fictitious) here:

Date     Type  Notional Underlying Expiry Market_Value DV01 X    Y    Z
1/1/2018 Bonds 100,000  USTreasury Mar18  100,000      2.5  2.5  5.0 15.0
1/2/2018 Bonds 150,000  USTreasury Mar18  149,000      4.0  4.0  8.0 12.0

What I'm trying to say is that there're some unknown correlations a,b,c between DV01 and parameters X, Y and Z. However I'm not entirely sure what the relation is, for simplicity sake I'm just letting a=1, b=2, c=3. For other type of instruments such as Options, the correlations will be entirely different. What I hope to achieve here is to be able to pick out values which do not conform to these relations. If the following data comes in the next day, the 50.0 in Z should be flagged.

Date     Type  Notional Underlying Expiry Market_Value DV01 X    Y    Z
1/3/2018 Bonds 200,000  USTreasury Mar18  200,500      5.5  5.5  11.0 50.0

Unfortunately a,b,c are not constants; they change with time. The model should be able to learn from the past, say 10 days, and establish what type of relation it is and check the next day if there're any anomalies. What would be a good way to start?


texts: Difference of sklearns accuracy_score() to the commonly accepted Accuracy metric

texts:
I am trying to evaluate the accuracy of a multiclass classification setting and I'm wondering why the sklearn implementation of the accuracy score deviates from the commenly agreed on accuracy score: $\frac{TP+TN}{TP+TN+FP+FN}$

For sklearn the sklearn.metrics.accuracy_score is defined as follows(https://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score):

$\texttt{accuracy}(y, \hat{y}) = \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples}-1} 1(\hat{y}_i = y_i)$

This seems like its completly neglecting the true negatives of the classification.

Example:

Predicted        1                     2                 3
Actual
1               5                      2                 0
2               8                      6                 2
3               3                      4                 12

And here the TP,TN,FP and FN:

                TP                     TN                   FP                FN
1               5                      24                   11                2
2               6                      20                   6                 10
3               12                     21                   2                 7
SUM             23                     65                   19                19

In the "standard" average score I would calculate:
$\frac{23+65}{23+65+19+19}=0,698$

In the sklearn implementation however it would be:
$\frac{1}{42}*23= 0,548$

Why is this different? And is the other metric somewhere mentioned in the literature, I couldn't find anything so far.


texts: What is the dimension reduction method to large numbers of independent features while only two of them are important?why?

texts:
What is the dimension reduction method to model a data with large numbers of independent features (for instance 5k features), while only two of them are important (are effective in cost function)?
I have tried on PCA!


texts: Improving a basic neural network with only 3 input columns

texts:
I am just getting started with neural networks, and I was hoping someone could shed some light on an issue I keep facing with a model I am working on. I have a dataset that looks like this:
df:

x1     x2    x3      y1
70     3     0.25    0.4
80     4     0.5     1.0
70     4     1.25    0.5
...    ...    ...     ...

...with a shape of (400, 4)
My objective is to train a deep learning model to predict the y1 value using the x1, x2, and x3 inputs.
Here is my code so far adapted from this source:
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# define base model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(3, input_dim=3, kernel_initializer='normal', activation='sigmoid'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_absolute_percentage_error', optimizer='sgd')
    return model

kfold = KFold(n_splits=10)

estimator = KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=0)
results = cross_val_score(estimator, X_train, y_train, cv=kfold)
print("Results: %.2f (%.2f) MSE" % (results.mean(), results.std()))

However, my results are a little unusual:
Results: -122.97 (87.82) MSE
y_pred = model.predict(X_test)
y_pred

array([[0.9994005],
[0.9994005],
[0.9994005],
[0.9994005], ...)
My main questions here are why are the final values always the same, in this case "0.9994005", and what can I do to improve this model? What the main things you look for in deep learning models that direct you in the right path?


texts: Python and data analytics how to get the unique number of orders per seller using groupby?

texts:
I am totally new to data analytics and python.

I took a good course on Machine learning. But I didn't really worked with real life data.

I managed to get data from a store who wants to create a recommendation system. I convinced them, that I will learn and get experience, and you will get benefited with a recommendation system, plus some insights on your previous orders.

I've got a json file having the following structure (I transformed it into excel):

Each order_id is connected through one seller_id. And it may contain multiple product_id.

With that I mean, that a user could have orderd from the seller_id=1, 2 products and the data would be like this:

{
    order_id: 12,
    seller_id: 1,
    product_id: 3,
    ...
},
{
    order_id: 12,
    seller_id: 1,
    product_id: 89,
    ...
}

The main things I need to export from these data is the following:

Number of unique orders per seller;
Products that are being ordered the most per seller;
Products that are being ordered the most in general;
Sum of the quantity field for each product to check how many times has been order;
Number of orders for each customer;

And other things that I may come up with later.

I started with the number 1, wherre I need to get the unique number of orders per seller. I tried the following:

data.groupby('order_id')['seller_name'].value_counts().sort_values(ascending=False)

But I didn't get the right result. A seller having 110 unique orders, but using the line above, it shows he is having more than 500 orders.

Here is a snippet of the data available:

Even though, seller_id=1 having 4 rows, but the actual number of orders is uniquely 2, despite the fact that the quantities of items within these 2 orders are 7.

The same for seller_id=5, he is having 4 rows, but they are only for one single order.

So how can I get the unique number of orders of each seller as a start? I think I will figure the other points out once I get to do the first one.


texts: shapes (127,1) and (13,) not aligned: 1 (dim 1) != 13 (dim 0)

texts:
i am try to find score of linear regression
it gives me this type error my code is below

from sklearn import datasets
bostan=datasets.load_boston()

x=bostan.data
y=bostan.target

from sklearn import preprocessing

x_scale=preprocessing.scale(x)

yfrom sklearn import model_selection

x_train,x_test,y_train,y_test=model_selection.train_test_split(x_scale,)

from sklearn.linear_model import LinearRegression

clf=LinearRegression()

clf.fit(x_train,y_train)

y_pred=clf.predict(x_test)

y_pred=y_pred.reshape(-1,1)

y_test=y_test.reshape(-1,1)

y=clf.score(y_pred,y_test)   # problem is in this line
```


texts: A pretrained model for mathematical equations characters detection

texts: I am working on a project to convert equations to LaTeX code. After segmenting out the characters, I got stuck on the detection part and was looking for some pre-trained model that could detect characters of the equation for later conversion to LaTeX. Is there any such pre-trained model available on the internet that could be used in Python to identify characters. If not then can somebody share some source to find a dataset to train such kind of model in Keras?


texts: Visualization of transformed features in BERT

texts:
So I'm trying the Intent Recognition with BERT using Keras and TensorFlow 2 available at kdnuggets.com and this is the code for the results evaluation.
sentences = [
  "Play our song now",
  "Rate this book as awful"
]

pred_tokens = map(tokenizer.tokenize, sentences)
pred_tokens = map(lambda tok: ["[CLS]"] + tok + ["[SEP]"], pred_tokens)
pred_token_ids = list(map(tokenizer.convert_tokens_to_ids, pred_tokens))

pred_token_ids = map(
  lambda tids: tids +[0]*(data.max_seq_len-len(tids)),
  pred_token_ids
)
pred_token_ids = np.array(list(pred_token_ids))

predictions = model.predict(pred_token_ids).argmax(axis=-1)

for text, label in zip(sentences, predictions):
  print("text:", text, "\nintent:", classes[label])
  print()

What I want to add in here is a visualization of the transformed features in BERT using a graph that is similar to clustering problems graphs. It should look like this:
Image taken from: towardsdatascience.com

Each color represents an intent and this graph is meant to show the predicted intents distribution in order to visualize how well BERT is doing.
Is there a graph similar to this meant to visualize model predictions? In this case, what are the hyperparameters related to the transformed features that should be considered since the predicted intents are not stored in a dataframe?
Thank you for your help peeps.


texts: NN Making Poor Averaging Fit whilst LGBM Regressor Fits Perfectly

texts:
I have a simple toy dataset for which the features have been encoded using a Encoder-Decoder NN.
I am using the hidden feature vector from the Encoder as the X input for training a 1-step lookahead model of the data.
I know the feature vector from the Encoder is good, because when I use this X as input into an vanilla out of the box LGBM tree model it fits almost perfectly to the Y output.
However, trying to train a simple 2 layer NN network (ReLU activation in first layer, Linear in second) which should learn the relatively easy mappings of the X feature vector to the Y output, it basically just learns to just average the output.
Now there is a reason why I want to use a NN model to predict the output here rather than just using the LGBM model, so why (if the LGBM model can do it) is my NN struggling so hard.
Below is the Prediction/Target from the LGBM Model:

And here is the same but from the NN model:

The reference Pytorch code for the NN model is:
 alpha_hidden_1 = nn.Linear(encoder.hidden_size, hidden_alpha_size)
 alpha_out = nn.Linear(hidden_alpha_size, 1)

 x = F.relu(alpha_hidden_1(hidden_state_vector))
 x = alpha_out(self.dropout(x))

I'm training it for 1000 epochs and have tried varying various hyperparameters to no avail.


texts: How to train an ML to predict improvement in patient recovery?

texts:
I am interested in training an ML model that would predict improvement in a patient's injury.
Assume doctors have collected two sets of MRI images from hundreds of patients who have suspected brain injury. The first set was taken at 2 weeks after the injury and the second set was taken 6 months after the injury. Additionally, doctors have their own assessment of the injury (say a score indicating the strength of the injury). So the patient's recovery from the injury can be better (increase in the score and hence positive) or worse (decrease in the score and hence negative) from 2 weeks to 6 months.
How to frame this problem so that I can train an ML model that takes the 2 weeks image and 6 months image as its inputs and predicts if the recovery is positive or negative? I can use the injury score provided by the doctors as a ground truth. But what sort of loss function to use to train this model in an end-to-end fashion?
I know I need an ML model that takes in two inputs X and X' and outputs a score s and the score should be any positive number if the recovery is good or any negative number if the recovery is bad. I am not sure what loss function to use so that I can train the model. Note that X represents the MRI scan taken at 2 weeks and X' represents the MRI scan taken at 6 months.


texts: Extracting MM-YYYY from python date and creating a new column with the same

texts:
I want to extract the month and year from one of my date columns in the dataset and create a new column in the data-frame with the new MM-YYY format.

My current solution is working fine but its way to long. I am looking for an efficient way to do this.

Date point looks like this: 2017-07-26 and its in datetime64[ns] format and I want the output to look like this: Jul-2017


texts: What can be done so that 'teacher' and 'teaches' are treated similar?

texts:
Who teaches English?

Now, after tokenizing, stemming..
    it gives me

Who, teach, English

In my list of word, I have a word called

teacher

Lemmatizing, stemming teacher gives teacher and lemmatizing, stemming teaches gives teach

Even, calculating edit_distance will not solve this.. As, edit_distance comes out to be 2.

Now, What do I do to have  teacher and teach treated as similar?
Similarly, there may be other cases with extra 's' at the end. Is there some stemmer that solves this problem? Is there any solution?

Other similar example can be: instructor and instructs


texts: Classifcation of non-linear regressions based on their shapes

texts:
I have a data set of thousands of individual y ~ x relationships that can have varying shapes in their relationships. For example, they can follow an exponential, asymptotic, logistic or hump-shaped (with different skewness) pattern. I actually don't know all the shapes the relationships can take, these are just a few patterns that I hypothesise to emerge.

I need a way to classify each of these relationships into one of the non-linear patterns without looking at each individual relationship. I was wondering whether there is way to do this in machine learning. I am willing to sit down and classify maybe some of the patterns myself but would like to avoid classifying all of them by hand.

Ultimately, I'm not interested into prediction. The regression should only serve as a way to describe different non-linear patterns and classify each individual into a category based on the shape of the regression.

What I am doing so far is fitting non-linear regressions with different equations to each relationship and decide which of the non-linear regressions (exponential, asymptotic etc) is fitting best by evaluating the residual variance and the p-value. However, I was wondering whether there is a way to do this with machine learning instead.


texts: Distinction of different types of missing values is lost after importing data from SPSS into R

texts:
I've got a file containing survey data in SPSS. There are 3 types of missing values defined: invalid (coded as 900), not applicable (990), not filled in (999).
After importing the SPSS file into R using haven library, all types are coded as "NA", the distinction is lost.
How do you deal with this if distinction is important?


texts: Rain radar image noise reduction and cleanup

texts:
An application that I am building is plotting rain radar images on map. The images are transparent PNGs that are sourced straight from the local meteorological service.

As can be seen in the example, the images have two types of noise I'd like to get rid of:

Directional interference (can be seen at around 11 o'clock on the image below)
General noise (small dots that are not real rain clouds)

A human eye can easily see what the "real" clouds look like when viewed as an animation.

What are some recommended methods to clean up this image that reduce as much noise as possible? References to algorithms and tools (Python, prefereably) alike would be appreciated.

Methods can include cleaning up each image separately, or using previous frames as context.


texts: Small number of estimators in gradient boosting

texts:
I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models and Bayesian optimization algorithm for the hyperparameters search (hyperopt)

One of the hyperparameter which is being tuned in the number of estimators used for each model. For the 1st round of testing, I starting with number of estimators in the range of 100-300. The outcome from hyperparameters optimization algorithm was that the best number of estimators is 100.I repeated the same analysis where the range of number of estimators was modified to 50-300, this time the outcome from hyperparameters optimization algorithm was that the best number of estimators is 50. I repeated the same analysis for the 3rd time setting the range of possible number of estimators to 2-300 (Just to check the extreme case) this time the outcome from hyperparameters optimization algorithm was that the best number of estimators is 2. The same outcome was noticed both for the XGBoost and lightGBM models.

What does this say about my model and data? Does this mean that the best model is Random forest instead of gradient boosting? is it smart 'force' the number of estimators by fixing the hyperparameter value (e.g., 150) and tuning all other parameters?

The training dataset has > 0.5M Samples where ~ 90% of the data is zero and the rest is mix of positive and negative values. Should I consider using a different object function rather than MSE?

The range of the tuned parameters for LightGBM are:

'max_depth': 5 ==> 15
'colsample_bytree': .6==> .9
'subsample':.5 ==> .8
'reg_alpha': np.log(1e-4) ==> np.log(1e-1)
'reg_lambda': np.log(1e-4) ==> np.log(1e-1)
'n_estimators': 50 ==> 300
'num_leaves': 10 ==> 150, 2
'min_child_samples': 20 ==> 800
'subsample_for_bin': 20000 ==> 300000
'subsample_freq': 1 ==> 20

Output:
{'subsample_freq': 16, 'num_leaves': 10, 'max_depth': 6, 'colsample_bytree': 0.7577614465604802, 'subsample_for_bin': 80000, 'min_child_samples': 415, 'n_estimators': 56, 'subsample': 0.6531478473538894, 'reg_alpha': 0.025744268683186224, 'reg_lambda': 0.0001729942781329532}

The range of the tuned parameters for XGBoost are:

'colsample_bytree',.6 ==> .9
'max_depth', 4 ==> 9
'n_estimators', 50 ==> 300
'reg_alpha', np.log(1e-4) ==> np.log(1e-2)
'subsample',.5 ==> .8
'gamma', 0 ==> 4

Output:
{'gamma': 2.4257700330471357, 'max_depth': 4, 'n_estimators': 57, 'subsample': 0.5568564232616263, 'reg_alpha': 0.0009876777981446033, 'colsample_bytree': 0.7073279309167877}


texts: Can LSTM have a confidence score for each word predicted?

texts:
LSTM networks can be used to generate new text. Given a sequence, I can predict the next word. Is there a way to get a score associated to each word predicted?

In particular, if the new word has never been seen by the LSTM network, can we train the LSTM to output a score of "no confidence"?

For example, this article gives the following example:

For example, let us consider the following three text segments:

1) Sir Ahmed Salman Rushdie is a British Indian novelist
and essayist. He is said to combine magical realism with
historical fiction.

2) Calvin Harris & HAIM combine their powers for a magical
music video.

3) Herbs have enormous magical power, as they hold the
earth’s energy within them.

Consider an LM that is trained on a dataset having the
example sentences given above — given the word “magical”,
what should be the most likely next word: realism, music, or power?

Say, that, in fact, the next word is neither one, but "power", but my LSTM has never seen that word before. So, the LSTM is going to predict one of the three words it has seen, but I would like it to output a low confidence score. Is this possible?


texts: What is the best way to train a gradient boosted model on a binomial dataset where the number of "observations" for each instance varies?

texts:
I have been trying to figure out the best way to train a gradient boosted model on a binomial dataset. To be more clear my dataset is in a format similar to this:
[link to toy dataset].
https://i.stack.imgur.com/b8bRL.png
The Goal is to predict the probability of "success" (# of successes/# of trials) for each row (set of predictor variables).
The Problem I am having is I am not sure how to incorporate the # of trials (of each row) into the models. This is clearly important as we have much more confidence in instances with more trials (100 successes in 100 trials is more trustworthy than 1 success in 1 trial). I know that to deal with this, binomial linear regression treats each trial as an instance (instead of treating the sum of trials with the same predictor variables as an instance). I would like to try to mimic this approach in a gradient boosting framework.
My Solutions  The first approach I am considering is using an XGBoost regressor to predict the % of successes in each row. I would set the sample weights of each row equal to the number of trials that were in that row. Would this properly deal with the fact that we have increased confidence in rows with more trials?
My alternate approach is "expanding" the data so that each trial is its own row. So if one row had 5 successes out of 8 trials I would expand this into 8 rows (5 successes and 3 failures). I would then use a XGBoost classifier to predict the probability of success of each instance/trial. Since all of those rows that originally belonged to the same row would have the same predicted probability of success I would take this as the probability of success of the original row. This approach seems likely to work, but very inefficient.
Which of these approaches would better account for the fact that we have more confidence in instances with more observations? Are there any other better ways to deal with this in a gradient boosting framework?


texts:
Could you please explain, in simplest way, the algorithm (mathematical equation) of back-prop?
I have read lot of articles on it so I know what it is and understand the intuition behind it, but I still do not understand the equation of "upgrading/changing" neurons' attributes in layers.


texts:
I'm trying to get list of n-gram tokens for text

Ex: 'How to use build_analyzer in sklearn feature extraction '

output :['How', 'use', 'build_analyzer', 'sklearn', 'feature', 'extraction', 'How use', 'use build_analyzer', 'build_analzer sklearn', 'sklearn feature', 'feature extraction']

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

vectorizer = TfidfVectorizer(stop_words = 'english',ngram_range=(1, 2), token_pattern=r'\b\w+\b', min_df=1)

 df['Text'].apply(lambda x : vectorizer.build_analyzer(x))

TypeError: build_analyzer() takes 1 positional argument but 2 were given


texts:
There's a Twitter thread going around that claims there are signs of voter fraud due to anomalies in the election vote count data set. You can download the dataset here and find the script to generate the analysis here.
Was the dataset analysed correctly? If yes, are these 'anomalies' significant enough to indicate fraud? If no, in what way is the analysis lacking?
PS: I'm not looking to start a political discussion, or to make any claims of fraud myself.


texts: Multi output regression analysis learning curve

texts:
I am quite new in data science. I have been following Andrew Ng's machine learning course.
I am employing neural networks for a 2-input 2-output regression problem for a robotics project
The architecture of the network has only 1 hidden layer with 10 neurons. The training size is 2160 and the cross validation set size is 720.
I have plotted the learning curves for 100 samples and this is what I get . (Please refer figure). I am not sure what to make of it, whether it is overfitting or underfitting, since the errors are low even from the beginning itself. How can I improve the performance?


texts: Performance profiling and tuning in Apache Spark

texts: Beyond the immediate suspects defined in the spark documentation, what are some ways to profile, tune and boost performance of an Apache Spark application?


texts: How to retrive the results saved in model of gensim?

texts:
I have a file that contains words. I have to calculate cosine similarity of each word with every other words. For that I used the following formula in gensim:

model.most_similar([word])
model.save('text.txt')

I want to save the result of cosine-similarity of each words with respective to other words in such a way that I can used them further in classification and clustering. But don't know how to retrieve the results which are saved using gensim model ?


texts:
I was reading this article talking about a form of targeted internet bullying which involves sending flashing images via Twitter to people with epilepsy.

I was wondering whether there is a way to detect these malicious GIFs? I first though of it as a classification problem but in reality the content can be anything - what we would need to detect is the flashing lights which have a particular frequency.

Any idea on how to approach this? Which technique can be used?


texts: Generate a balanced batch with ImageDataGenerator() and flow_from_directory()

texts:
Hi I am new to python and deep learning. I am doing a multiclass classification. My 3-classes dataset is imbalanced, the classes take about 50%, 40%, and 20%. I am trying to generate mini batches with balanced classes. I am using class_weight to generate a balanced batch in fit_generator() but I am doubting if it actually works because the batches generated by train_datagen.flow_from_directory() is not balanced. the generated batches have weights around [0.43, 0.38, 0.19].
 My code are as follow:

train_datagen = ImageDataGenerator(rescale=1./255,
                                    featurewise_center=True,
                                    rotation_range=30,
                                    width_shift_range=0.3,
                                    height_shift_range=0.3,
                                    shear_range=0.2,
                                    zoom_range=0.2,
                                    horizontal_flip=True,
                                    fill_mode='constant')

#Training Set
train_set = train_datagen.flow_from_directory(
                                             directory=train_folder,
                                             target_size=input_shape[:2],
                                             batch_size=32,
                                             shuffle=True,
                                             class_mode='categorical')
#Validation Set
val_set = test_datagen.flow_from_directory(
                                            directory=val_folder,
                                            target_size=input_shape[:2],
                                            batch_size = 32,
                                            class_mode='categorical',
                                            shuffle=True)

call_backs = [EarlyStopping(monitor='val_loss', patience=train_patience),
             ModelCheckpoint(filepath=output_model, monitor='val_loss', save_best_only=True)]

class_weights = class_weight.compute_class_weight(
               'balanced',
                np.unique(train_set.classes),
                train_set.classes)

history = model.fit_generator(
          train_set,
          steps_per_epoch=2000 // batch_size,
          epochs=300,
          validation_data=val_set,
           validation_steps=800 // batch_size,
           class_weight= class_weights,
           verbose=1,
           callbacks=call_backs)

Does this code enough to generate balanced batches for training? I checked the batches generated from the train_set they are around [0.43, 0.38, 0.19]. Any advice would be appreciated Thank you.


texts: train NER using NLTK with custom corpora (non-english) must use StanfordNER?

texts:
I have searched about customization NER corpora for trainig the model using NLTK library from python, but all of the answer direct to nltk book chapter 7 and honestly makes me confuse how to train the corpus with correct flow and data set that has structure like this below:
Eddy N B-PER
Bonte N I-PER
is V O
woordvoerder N O
van Prep O
diezelfde Pron O
Hogeschool N B-ORG
. Punc O

I have some questions:

I found so many article that if you will train customed corpora using NLTK, there uses StanfordNER library too, should it be? or we can use pure of NLTK library for it?
Should the grammar pattern be included if you want to apply it to other languages? How is the flow?

And please give me example of code to train custom corpora until give the tag of POS Tag and NER label output using data like data structure above if you have. Thank you.


texts: What is Deep learning approach to count the number of Diamonds in an image?

texts:
I am working on a project which involves counting the number of diamonds in the provided image. I have a set of images and a VIA annotated .json file which has all the annotations. How do I proceed with this problem.

what approach to start i.e Object detection vs Instance segmentation ?
what models to choose from MaskRCNN,U-Net, etc (Please suggest) ?

Please guide.
Thanks


texts: Which is best FaceNet or dlib_face_recognition_resnet_model_v1?

texts:
I am newbie in face recognition related things... As far i observed dlib's frontal_face_detectoris widely used to find the faces in an image and after that, to extract face_descriptor vectors which is better for real time face authentication system ?

FaceNet by google
dlib_face_recognition_resnet_model_v1 by face_recognition

It looks both working fine.. but in real-time implementation,

Is there some thing important to understand the performance ?

Or any comparison checks done on real time / large data sets ?
Thanks in advance.


texts:
3 is a big file, but I would like to reset the state after mini_batch_size of 50.

n_epoch=10000
n_batch=50

# create and fit the LSTM network
model = Sequential()
model.add(LSTM(3,batch_input_shape =(n_batch,trainX.shape[1], trainX.shape[2]),stateful=True))
model.add(Dense(1))
model.add(Activation("linear"))
model.compile(loss="mse", optimizer="adam")
model.summary()

#fitting model
for i in range(n_epoch):
    history=model.fit(trainX, trainY, epochs=1, batch_size=n_batch,verbose=2, shuffle=False)
    model.reset_states()

I am getting the error:

value error:  In a stateful network, you should only pass inputs with a number of samples that can be divided by the batch size. Found: 63648 samples<

How can I train a LSTM in mini_batch size of 50 (a number which is not divisible by trainX)?


texts: Predict arguments of indicator function or the value the indicator function itself

texts:
I have an evenly-spaced timeseries $a_1, a_2, a_3,..., a_n$ and a function $f_k=\mathbf 1(a_k-a_{k-5})$, where $\mathbf 1$ is the unit step function.

I want to predict $f_k$ using $a_1, a_2, a_3,..., a_{k-5}$, that is, I want to predict if $a_k>a_{k-5}$, given the values $a_1..., a_{k-5}$ .

My question is:

Is it better to construct and train an ANN to:

directly predict values of $f_k$

OR

predict values of $a_k$, and then apply $f_k$ to the result?


texts:
There are 2 sns.pairplot, tell me how to interpret them.

As I understand it, sns.pairplot allows us to look at the diagonal distribution of these signs, and on the non-diagonal linear relationship between the signs, i.e. it is possible to identify in which space (a pair of signs) the classes will be well separated from each other. If you look at the picture, I understand that, on the first there are very few dependencies, unlike the second. By the way, this is confirmed by the correlation matrix. If so, is it necessary to remove one of the dependent signs? Are my statements true or not?


texts: Handling features with multiple values for clustering

texts:
Suppose I have a movies dataframe in pandas. One of the features is Genre.

It has a list of genre names. for example:

Movie_ID    Genre

1        [Action, Thriller, Drama]
2        [Romance, Comedy]
3        [Action, Romance]

How do I use this column for a clustering problem, say K-means clustering?


texts:
In R I have data where head(data) gives

day   promotion   profit   new_users
1           105    45662          33
2            12    40662          13
3            44    46800          20
4           203    54102          46

Now day is simply the day (and is in order). promotion is simply the promotion-value for the day, the profit is the profit that day and new_users is the number of new users that day.

I want investigate the relationships between promotion to profit and new_users. We see a clear positive correlation between promotion and profit, and there is also a positive correlation between promotion and new_users.
In R I simply test correlation

cor.test(data$promotion, data$profit, method="kendall", alternative="greater" )
cor.test(data$promotion, data$new_users, method="kendall", alternative="greater")

which both gives a low p-value, ie we have a positive correlation.

I want to find a point where where the increase of promotion don't increase profit or new_users that must, ie a sweet spot.

Here is 2 plots and the R code for these

plot(data$promotion, data$profit, col="brown")
plot(data$promotion, data$new_users)

How should this be done?

My thoughts where to make a regression model.
For the first one "promotion vs. new_users" one could use a poisons model because it's a count-process, so a model like this would be a good chose?

glm(formula= data$new_users ~ data$promotion, family="poisson", data=data)

Next what regression model should one chose for the next one. Is it fair to say that this regression model is a good chose ? (I use sqrt command)

glm(formula=data$profit ~ sqrt(data$promotion) , data=data)

Or maybe it's not even necessary to use a regression model at all to find a sweet spot?

Thanks.

I have now looked at 'good' new users. For each day we have a promotion value and we have a count value which is the number of new good users. This plot shows us the number of good new users we get for a promotion for each day. For example for promotion value 90 we have a day where we got 8 new good users and a day where we got 14 new good users.

What would be the right approach to find a sweet spot for the use of promotion ?


texts:
I got the following warning:

94: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 1200012120 elements. This may consume a large amount of memory.

For the following code:

from wordbatch.extractors import WordSeq
import wordbatch
from keras.layers import Input,Embedding
...
wb = wordbatch.WordBatch(normalize_text, extractor=(WordSeq, {"seq_maxlen": MAX_NAME_SEQ}), procs=NUM_PROCESSOR)
wb.dictionary_freeze = True
full_df["ws_name"] = wb.fit_transform(full_df["name"])
...
name = Input(shape=[full_df["name"].shape[1]], name="name")
emb_name = Embedding(MAX_TEXT, 50)(name)
...

That is I make use of WordSeq (from WordBatch) output from the Embedding layer of a GRU network. How should I modify the code to make it work without converting to dense tensor?


texts: Traditional ML Model or Deep Learning for ~200-300 samples?

texts:
Good morning all!

I'm working on a resume parser that is integrated with an RPA (robotic process automation) platform.

The robot has OCR to extract text from a PDF resume, and it supplies the tokenized contents as well as the X and Y coordinates of each word.

My goal is to create a predictive model to identify key sections of a resume, and then use traditional resume parsing for the remainder of the workflow.

Some of the features included in the model are as follows:

| Word | X_norm | Y_norm | Length_Word | Num_Word_per_Line | Special Char? | Contains numbers? |

Where normalization is min-max.
We are also working on getting font size and normalized color (i.e boldness).

My gut feeling is that traditional ML model might be better as we only have the resources to label 200-300 resumes, not thousands. But I wanted to be open to deep learning (something I don't have much experience with). The RPA workflow is such that a Resume Specialist can give feedback to the model in real time, and something that can improve itself on the fly might be better for this use case.

Can traditional ML models also efficiently self-improve?

Thanks for your time!


texts: I have the following averaged − curves with 4 models. Which one is the best?


texts: How to get periodicity from timeseries data?

texts:
I would like to create a recommendation system for a smart home application. I gather the data in a time-series database. The app monitors the on/off state of a smart lamp and can create daily routines. I would like to create notifications to the user like "It looks like you are usually use your lamp from 8:00 to 10:00. Would you like to create a routine for that?".
I am beginner to data science and machine learning and I would like to ask what kind of machine learning algorithm or method should I use for this problem. Someone recommended me Facebook Prophet, but as I can see that library can only predict future occurences not the periodicity of time-series data. Do you know any library or algorithm?
In the image below you can see my problem detailed. There is hourly on/off state of the lamp daily in green and in blue the result that I would like to get: the periods that the lamp is usually on.


texts: Clustering algorithms based on lists of most similar items?

texts: I'm a beginner to machine learning. I have a large dataset of items, each with a limited set of most similar items in the dataset, which I would like to create clusters of, preferably each having the same size, although this isn't necessary. Could someone point me in the right direction? Thanks.


texts: Big array and MemoryError: Unable to allocate memory (in Python)

texts:
I am trying to create a predictive model using linear regression with a dataset that has 157673 entries.
The data (in a csv file) is in such format:
Timestamp,Signal_1,Signal_2,Signal_3,Signal_4,Signal_5
2021-04-13 11:03:13+02:00,3,3,3,12,12

My current code:
filename = 'test.csv'
df = pd.read_csv(filename , parse_dates=['Timestamp'], header=0)
df['Timestamp'] = pd.to_numeric(pd.to_datetime(df['Timestamp']))
u, v, w, x, y, z = df.values.T

X = np.asarray([v, w, x, y, z])
Y = np.asarray([u, u, u, u, u])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
print('Score: ', lineReg.score(X_test, y_test))
print('Weights: ', lineReg.coef_)

When printing out the shape of both X and Y it is (5, 157673) (after putting u 4 more times in the Y array, since it would otherwise give the error ValueError: Found input variables with inconsistent numbers of samples: [5, 1]).
However now I am running into the error MemoryError: Unable to allocate 185. GiB for an array with shape (157673, 157673) and data type float64.
Why is that? There must be a mistake somewhere and if not, why is it suddenly in the shape of (157673, 157673) instead of (6, 157673) ?


texts:
I have sparse vectors and found that cosine similarity is very efficient to to measure the similarity. Now I want to cluster these vectors based on similarity. Hence, can someone please suggest/recommend clustering algorithms that make use of cosine similarity?

P.S.: I do not have a predefined number of clusters beforehand and want the clustering algorithm itself to decide it.


texts: In CNN (Convolutional Neural Network), does the combination of previous layer's filters make next layer's filters?

texts: I know that the first layer uses a low-level filter to see the edge information. As the layer gets deeper, it will represent high-level (abstract) information. Is it because the combinations of filters used in the previous layer are used as filters in the next layer? ("Does the combination of the previous layer's filters make the next layer's filters?) If so, are the combinations determined in advance?


texts: Predicting Bitcoin price with LSTM (and other RNNs)

texts:
In order to try what i learned about LSTM, i downloaded a simple bitcoin price dataset, and tried to make a network which predicts the bitcoin price by feeding a sequence of price history.
Looking for this topic on Google, it seems a lot of people have tried it, and they get the same results as I do : the LSTM seems to learn the identity function. Indeed i remarked that when predicting the prices with LSTM, the predicition seems to be the original price curve, but shifted to the right. (basically the LSTM just repeat the last known price)
And this is kind of logical, if you were to try to predict the current price of bitcoin given the previous prices, just by repeating the last known price you would be close to truth a lot of times.
I have tried changing my loss function in order to avoid the network to learn the identity function, but it didn't work. Has any work been done on that before ? Is it a known issue ? All the blog article which were using LSTM didn't mention it, tho it was obvious from their graphs... (e.g. http://www.quantatrisk.com/2020/04/01/how-to-predict-bitcoin-price-deep-learning-lstm-network-python/ prediction at the end)


texts: How to convert photo to a vector drawing

texts:
I am using the following code to convert photo to a drawing:
import cv2
import numpy as np

img = cv2.imread("adventure.jpeg")

# 1) Edges
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
gray = cv2.medianBlur(gray, 5)
edges = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 9, 9)

# 2) Color
color = cv2.bilateralFilter(img, 9, 300, 300)

# 3) Cartoon
cartoon = cv2.bitwise_and(color, color, mask=edges)

cv2.imshow("Image", img)
cv2.imshow("Cartoon", cartoon)
cv2.imshow("color", color)
cv2.imshow("edges", edges)
cv2.waitKey(0)
cv2.destroyAllWindows()

the result is as follows:

However I do not want the above result, what I want is this:

How do I do it using opencv / deep learning ?


texts: Can neural network represent a relation that is a derivative?

texts: Let's say I measure people hiking uphill, and it happens that their hiking speed is related to the slope so they hike slower where the hill is steeper. My input variable is the hill's elevation and my output variable is their hiking speed. I fit a neural net to this data, so that I can later predict the hiking speed given new elevation measurements. Can a neural net represent the input's first derivative and so do a good job capturing this relation? What about representing a second derivative, i.e. if hiking speed was related to the curvature of the hill?


texts: How to use AutoEncoders or VAE for initializing custom CNN architecture?

texts:
I would like to use AutoEncoder or VAE in order to learn set of features which I can use to initialize training procedure of a custom CNN architecture that I've build. Here is the code:
class DeepCNN:
  @staticmethod
  def conv_block(input, filters, pool, reg):
    output = tf.keras.layers.SeparableConv2D(filters, 3, activation="relu",
                                             padding="same", kernel_regularizer=l2(reg))(input)
    output = tf.keras.layers.BatchNormalization()(output)
    if pool == True:
      output = tf.keras.layers.MaxPool2D((2,2))(output)

    return output

  @staticmethod
  def dense_block(input, units, dropout_rate, reg):
    output = tf.keras.layers.Dense(units, activation="relu", kernel_regularizer=l2(reg))(input)
    output = tf.keras.layers.BatchNormalization()(output)
    output = tf.keras.layers.Dropout(dropout_rate)(output)

    return output

  @staticmethod
  def build(width, height, depth, classes, filter_bank = None, dense_units = None, reg=0.0002, include_top=True):
     assert filter_bank is not None, "Filter bank should not be empty ..."
     assert dense_units is not None, "Top of the network architecture must contain atleast 1 Dense layer"

     inputShape = (height, width, depth)
     chanDim = -1

     if K.image_data_format() == "channels_first":
            inputShape = (depth, height, width)
            chanDim = 1

     inputs = Input(shape=inputShape)

     output = DeepCNN.conv_block(inputs, filter_bank[0][0], filter_bank[0][1], reg=reg)
     for idx in range(1, len(filter_bank)):
       output = DeepCNN.conv_block(output, filter_bank[idx][0], filter_bank[idx][1], reg=reg)

     output = tf.keras.layers.Dropout(0.2)(output)

     if include_top == True:
          output = tf.keras.layers.Flatten()(output)
          for idx in range(0, len(dense_units)):
              output = DeepCNN.dense_block(output, dense_units[idx][0], dense_units[idx][1], reg=reg)
          output = tf.keras.layers.Dense(classes, activation="softmax", name="classification")(output)

     model = Model(inputs, output, name="deep_cnn")
     return model

model = DeepCNN.build(150, 150, 1, 2,
                      filter_bank=[(32, True), (64, True), (128, True), (256, True)],
                      dense_units=[(512, 0.4)], reg=0.0005)

I am confused how I should configure AutoEncoder or VAE in order to generate such weights that are compatible for custom CNN architectures. Any help would be appreciated. Thanks in advance!


texts: Reverse of Yeo Johnson variable transformation done by preprocess function in caret package to get original Target and predicted values

texts:
I have done a Yeo Johnson transformation by using preprocess from caret package. I have predicted the target variable using linear regression. Now, I would like to reverse the transformation for both the target and the predicted values.
I know this question has been answered before, but I still could not solve my problem. This question was answered here and here.
I could not modify it for Yeo Johnson.


texts: Multi-Class Text Classification: Doc2Vec performing very bad compared to Hashing Vector

texts:
I have a multi-class text classification problem in hand this is similar to product category mapping where we map products to its correct Category based on the text content provided.

I first created a solution with Hashing Vector and SGD classifier with actually gave around ~84% accuracy.

After going through many online content I found that Doc2Vec is the current cutting-edge representation of Document or paragraph in numerical format.

So I changed my solution to use Doc2Vec method for Feature Engineering but the accuracy got from this is only ~54%.

Code:

Reading and Cleansing Data

import logging
import datetime
import re
import string
import codecs
import pandas as pd
import numpy as np

from gensim.models.doc2vec import Doc2Vec,TaggedDocument
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.utils.class_weight import compute_class_weight
from sklearn.metrics import accuracy_score
from sklearn.linear_model import SGDClassifier

import warnings
warnings.filterwarnings("ignore")

#Reading the input/ dataset
data_file = "Consolidated_input_dataset.txt"
data = pd.read_csv(data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

#Cleansing the input dataset removing non alphabets
data['cleansed_desc'] = data.COMMODITY_DESC.str.lower().str.replace('[^a-z]',' ').str.replace('\s+',' ')

#Spliting to list for traing Doc2Vec
data['cleansed_desc_split'] = data.cleansed_desc.str.split()
train_data, test_data, train_label,  test_label = train_test_split(data[["cleansed_desc", "cleansed_desc_split"]],
                                                                   data[["Label"]],
                                                                   test_size=0.3, random_state=100, stratify=data.Label)

Hashing Vector:

sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

sgd_model.fit(train_data.Doc2Vec.tolist(), train_label.Label)

#Predict Output
output_node1_predict = sgd_model.predict(test_data.Doc2Vec.tolist())

print(accuracy_score(test_label.Label, output_node1_predict))

#Train Model
sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

vectorizer = HashingVectorizer(n_features=90000, ngram_range=(1,3))

vectorizer.fit(train_data.cleansed_desc)
data_features = vectorizer.transform(train_data.cleansed_desc)
sgd_model.fit(data_features, train_label.Label)

#Predict Output
test_features = vectorizer.transform(test_data.cleansed_desc)
output_node1_predict = sgd_model.predict(test_features)

print(accuracy_score(test_label.Label, output_node1_predict))

Output:

84%

Doc2Vec:

#Creating Doc2Vec

data_tagged = train_data.apply(
    lambda r: TaggedDocument(words=r['cleansed_desc_split'], tags=[train_label.loc[r.name].Label]), axis=1)

doc2vec_test = Doc2Vec(dm=0, vector_size=100, negative=5, hs=0, min_count=2, sample=0, epochs=5, workers=8)
doc2vec_test.build_vocab(data_tagged)
doc2vec_test.train(data_tagged, total_examples=doc2vec_test.corpus_count, epochs=doc2vec_test.iter)

train_data["Doc2Vec"] = train_data.cleansed_desc_split.apply(lambda x : doc2vec_test.infer_vector(x))
test_data["Doc2Vec"] = test_data.cleansed_desc_split.apply(lambda x : doc2vec_test.infer_vector(x))

#Train Model
sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

sgd_model.fit(train_data.Doc2Vec.tolist(), train_label.Label)

#Predict Output
output_node1_predict = sgd_model.predict(test_data.Doc2Vec.tolist())

print(accuracy_score(test_label.Label, output_node1_predict))

output:

54%

Parameters for Doc2 Vec:

Vector Size =100
Window = 10
Epoch=5
min_count=2
Negative=5

Total number of Documents = 3450000+
Vocabulary Size = 46000+


texts:
Looking at the Tensor-flow object detection API, and walking through the "How to train your own Object Detector" for raccoons, which is a handy guide to get you up and running.

What I would like to do is append the SSD_Mobile model to include my own images along with the old pre-trained data.

Is there any guide out there that shows how this can be done?


texts: Deep learning detect reference boundary in text (or number of references in text)

texts:
I have several documents that either contain or don't an X number of references. I would like to build a model that can detect the number of references if any in a text.
I've been thinking for training to generate a bunch of random text and generate a variety of styles of references for different articles. Generating this dataset is fairly straightforward.
I am not sure how to craft the data for CNN. Word2vec does not seem like a good idea since punctuation is part of what makes references different than regular text. I could just do tf-idf vectors but then not sure what to represent as my Y. Should I put the boundary (index position, start and end) of where the reference is? What loss function do I use for a vector Y variable? Most guide show how to do numeric, binary and multiclass. Any advise or resources are much appreciated.


texts: How to create an ensemble that gives precedence to a specific classifier

texts:
Suppose that in a binary classification task, I have separate classifiers A, B, and C. If I use A alone, I will get a high precision, but low recall. In other words, the number of true positives are very high, but it also incorrectly tags the rest of the labels as False. B, and C have much lower precision, but when used separately, they may (or may not) result in better recall. How can I define an ensemble classifier that gives precedence to classifier A only in cases where it labels the data as True and give more weight to the predictions of other classifiers when A predicts the label as False.

The idea is, A is already outperforming others in catching true positives and I only want to improve the recall without hurting precision.


texts: I have a small (~100 samples) dataset with roughly 20 features which are mostly binary, and a few are numeric (~5). I wanted to use methods for augmenting the training set and see if I can get better test accuracy. What methods/code can I use for augmenting binary datasets?


texts: Does Batch Normalization make sense for a ReLU activation function?

texts: Batch Normalization is described in this paper as a normalization of the input to an activation function with scale and shift variables $\gamma$ and $\beta$. This paper mainly describes using the sigmoid activation function, which makes sense. However, it seems to me that feeding an input from the normalized distribution produced by the batch normalization into a ReLU activation function of $max(0,x)$ is risky if $\beta$ does not learn to shift most of the inputs past 0 such that the ReLU isn't losing input information. I.e. if the input to the ReLU were just standard normalized, we would lose a lot of our information below 0. Is there any guarantee or initialization of $\beta$ that will guarantee that we don't lose this information? Am I missing something with how the operation of BN and ReLU work?


texts: In a pandas dataframe can I convert my column values into numbers?

texts:
So I am new to all this. I was wondering in pandas can I convert my column values into numbers?
I'll try and give an example to explain what I mean
So say for example I have a column called, 'animals', in this column I have six different animals but I want to convert them to numerical values so just as simple as 1,2,3,4,5,6 for each of the different animals. How would I go about doing this??


texts:
I am training model to classify fruit images belonging to 60 classes. I have this result:

Validation accuracy is greater than training accuracy. Does this mean underfitting? If yes, can I fix this by adding more layers to the neural network or by increasing the number of neurons?


texts: Cosine similarity or logistic regression for spam filtering

texts:
I'm trying to choose an algorithm for filtering spam.
I found two options:

Create word dictionary for spam and not spam data.
Calculate average TF-IDF for each word and use cosine similarity for filtering.
Or use word dictionary for training logistic regression model.

Could you suggest what fit the best for my goal.
Maybe I should use some another algorithm.


texts: Text classification for data with multiple labels per observation

texts:
I have a dataset of tweets that has been labeled by multiple people. So the columns look something like:

Tweet_ID, Coder_1_Classification, Coder_2_Classification, etc.

The idea is to build a tweet classifier based on the labels. How should I input this data into the classifier? I was thinking of taking the plurality response, but is there a way I can use all of the data?

Also, there is partial coverage, so not every labeler labeled every tweet.


texts: which model for ordinal scaled dependent variable

texts:
I want to use several models to find a relation btw. life satisfaction and several independent variables like children, unemployment, marriage and so on.

My dependent var has an ordinal scale [1:10] and the independent vars have nominal and ordinal scales but no metric ones.

I would set up an ordered model like an ordered probit, but I do not know what else I could use. Could you give me some hints?


texts: Training, Testing and Validation Dataset

texts:
I'm training a Unet model for tumor segmentation.
I have a dataset of 400 patients for that.
The used images are CT scans (3D images) that I divide into 2D images (for a total of 30k 2D images).
I am actually splitting the dataset into:
10% test data, 18% validation data, 72% actual training data.
I'm dividing the test and training data over patients (i.e. the patients used for testing are not the same as the one for training).
Afterwards, I shuffle the 2D images and split in training/testing dataset (i.e. the same patients can be found in training dataset and validation dataset but not same stack images).
I have two questions:

Should I split the train/validation dataset according to patients too ?
Are the division percentages in train/test/validation adapted for my problem ?


texts:
I'm training a very simple LSTM in PyTorch.  It's a single layer, and I'm using it for multi-label classification with a BCEWithLogitsLoss.  My batches are shuffled, I'm not using regularization yet, I'm using a constant learning rate, but every once in a while the loss makes a big jump up, and takes a long time to come back down again.

X axis is epoch, Y axis is log-loss.


texts: turn off parts of features in a neural network?

texts:
Suppose I have a neural network which accepts two sets of features as inputs and generates corresponding outputs, for instance, generate average final grade from: 1. working hours for N students in a class, 2. mid-term grade for N students in the same class.  During training, the neural network is trained with many different classes. What I would like to during inference is to turn off one set of the features (i.e. feed only working hours as inputs) and predict outputs using the same neural network.  Obviously, setting the mid-term grades to be all 0s would not be a good option. I wonder if anyone know what would be a good way to do this?

Thanks!