texts: Interpreting Decision Tree in context of feature importances

texts:
I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the tree and the list of feature importances. What I don't understand is how the feature importance is determined in the context of the tree. For example, here is my list of feature importances:

Feature ranking:
1. FeatureA (0.300237)

FeatureB (0.166800)
FeatureC (0.092472)
FeatureD (0.075009)
FeatureE (0.068310)
FeatureF (0.067118)
FeatureG (0.066510)
FeatureH (0.043502)
FeatureI (0.040281)
FeatureJ (0.039006)
FeatureK (0.032618)
FeatureL (0.008136)
FeatureM (0.000000)

However, when I look at the top of the tree, it looks like this:

In fact, some of the features that are ranked "most important" don't appear until much further down the tree, and the top of the tree is FeatureJ which is one of the lowest ranked features. My naive assumption would be that the most important features would be ranked near the top of the tree to have the greatest impact. If that's incorrect, then what is it that makes a feature "important"?


texts: What exactly is the linear layer in the transformer model?

texts:
Please see this image:

There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here:

Are these linear layers simply dense or fully connected layers? Let's consider the weight matrix Wi Q. Does this represent a dense layer with "Q" nodes? As they are using matrices as input rather than 1D vectors, I am getting a little confused.


texts: LSTM validation loss always locked into zero

texts:
I have been working on an LSTM problem based on this Kaggle solution:
https://www.kaggle.com/karanjakhar/simple-and-easy-aprroach-using-lstm
And my dataset preperation is pretty much the same. The only difference being that I look at validation sets too. And have a deeper model.
For reference, this is my dataset preperation:
dataset=dataset.drop("item_price",axis=1)
dataset['date']=pd.to_datetime(dataset['date'],format='%d.%m.%Y')
dataset=dataset.groupby(['date_block_num','shop_id','item_id'],as_index=False)["item_cnt_day"].sum() #getting sums of sales from days
dataset=dataset.pivot_table(values="item_cnt_day",index=['shop_id','item_id'],columns=["date_block_num"]) #pivoting to group based on which item at which shop
dataset.reset_index(inplace = True) #getting rid of the double layer thing
dataset=dataset.fillna(0)
test_data=pd.read_csv(filepath2) # The dataset is read
dataset=pd.merge(test_data,dataset, on=['item_id','shop_id'],how='left')
dataset.fillna(0,inplace=True)
dataset.drop(['shop_id','item_id','ID'],inplace = True, axis = 1)
dataset.head()

That gives me a dataset that matches one in the recommended solution as well
    0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33
0   0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 2.0 2.0 0.0 0.0 0.0 1.0 1.0 1.0 3.0 1.0 0.0
1   0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2   0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 2.0 0.0 1.0 3.0 1.0
3   0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0
4   0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

dataset is finally prepared and trained as follows:
X_train = np.expand_dims(dataset.values[:,:-1],axis = 2)
# the last column is our label
y_train = dataset.values[:,-1:]

# for test we keep all the columns execpt the first one
X_test = np.expand_dims(dataset.values[:,1:],axis = 2)
print(X_train.shape,y_train.shape,X_test.shape)

opt = tf.keras.optimizers.Adam(learning_rate=0.001)#changed from 0.001
my_model.compile(loss = 'mse',optimizer = opt, metrics = ['mean_squared_error'])
history=my_model.fit(X_train,y_train,batch_size = 64,epochs = 20,validation_split=0.33,shuffle=True)  ## Removing the shuffle cause shuffle seems to undo the CV

But I have tried a bunch of models and combinations and no matter what I do, my test losses seem almost locked into zero like values. Even after trying K-Fold cross validation and different model depths dropout etc. Any help is appreciate.
My guess is the problem comes from the dataset preperation or how I am splitting into train and test sets


texts: Interpreting a curve val_loss and loss in keras after training a model

texts:
I am having trouble understanding the curve val_loss and loss in keras after training my model.

Can anyone help me understand it? Also, is my model overfitting or underfitting?


texts: Is there any way to collect categorical features quickly in Julia DataFrames?

texts:
I'm using Julia 0.6.3 with Dataframes.jl

I was wondering if there was any way to get categorial features easily in Julia?

For large datasets it can be impossible to enter everything by hand.

My workaround is to rely on strings and usually low cardinality but it's not fool-proof.

My workaround so far :

cat_cols = []
for col in cols
    if contains(string(typeof(X_train[col])),"String") == true
        push!(cat_cols,col)
    end
end

But it seems kind of ugly and I don't catch label encoded values because they are integers.

I could also try to rely on low unique counts but then sparse features would be taken in aswell.

Any idea? Thanks!


texts: Micro Average vs Macro average Performance in a Multiclass classification setting

texts:
I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the data falling in class label 1, 11.99% data in class 2, and remaining in class 3)

I am training a multiclass classifier on this dataset and I am getting the following performance:

                    Precision           Recall           F1-Score
Micro Average       0.731               0.731            0.731
Macro Average       0.679               0.529            0.565

I am not sure why all Micro average performances are equal and also Macro average performances are low compared to Micro average.


texts: How should multiclass classifier performance be measured when one type of error is preferred over another?

texts:
Sorry if this question has been asked before--I am having trouble searching this topic since I'm not sure of my wording.

Say you have a classification problem where there are more than two labels which are discrete but roughly correspond to an increase in some quality--call these labels A, B, and C. Also say in this problem it would be preferrable to over-estimate that quality, rather than to underestimate. Is there a type of metric that captures this skew and penalizes a predicted A on an actual B more than it penalizes a predicted C on an actual B? Or is this preference better handled in a different part of data science methodology?


texts: How do we make data Obfuscate or "De-identificate" to make it anonymous and share it publicly?

texts:
Right now, I am working on preparing a small dataset for release to the public by getting rid of sensitive information.  While working on it, I wondered... what are the best practices of dealing private or sensitive polynomial attributes in a dataset?(*) I have heard to create anonymity or to permute is achieved by De-identification, obscurification,  anonymization.

However, I would like to learn more about this topic in data science/analysis.   I am particularly interested in the packages and concepts that can be use in R.

(*)Well besides the obvious of solutions of completely removing the sensitive attribute or using hashcode encryption.  I am a little bit familiar with how this problem can be complicated by the ability to correlate attributes


texts:
I have a database with a lot of comments, each comment has a vote, a vote can be positive or negative. ex : -2, -5, -90, +45, +20...

So based on this training dataset I want to predict votes of future comments. How is it possible ? Thank you in advance


texts: How to change learning rate of MomentumOptimizer in tensorflow

texts:
I am trying to implement VGG-16 architecture in TensorFlow.
As mentioned in the paper, they changed the learning rate 3 time during their 74 epochs of training.
def optimizer(lr):
    return tf.train.MomentumOptimizer(learning_rate = lr)

But when I tried to change the learning rate in tensorflow,
lr=lr/10
opt_momentum = optimizer(lr)
opt = opt_momentum.minimize(cost)

I get the following error
FailedPreconditionError (see above for traceback): Attempting to use
uninitialized value fully_connected_2/biases/Momentum_1

I use builder = tf.saved_model.builder.SavedModelBuilder(export_dir) to save the model.
However. I can't understand how to load the trained weights to initialize the layers after changing the learning_rate?
For changing the learning, I am calling the function optimizer with a new learning rate as argument, which is return me a new op but the error says that the layers are uninitialized.


texts: Equally sized topics in Latent Dirichlet allocation

texts:
I'm using the topicmodels package for R to cluster a big set of short texts (between 10-75 words) into topics. After manually reviewing a few models it seems like there are 20 realtivly stable topics. However, what I find really weird is that they are all roughly the same size! Each topic catches around 5% of tokens and 5% of texts. In terms of the tokens, the smallest topic is 4.5% the largest 5.5%.

Can anybody suggest if this a 'normal' behaviour? This is the code I'm using:

ldafitted <- LDA(sentences.tm, k = K, method = "Gibbs",
             control = list(alpha = 0.1, # default is 50/k which would be 2.5.  a lower alpha value places more weight on having each document composed of only a few dominant topics
                            delta = 0.1, # default 0.1 is suggested in Griffiths and Steyvers (2004).
                            estimate.beta = TRUE,
                            verbose = 50, # print every 50th draw to screen
                            seed = 5926696,
                            save = 0,    # can save model every xth iteration
                            iter = 5000,
                            burnin = 500,
                            thin = 5000, #  every thin iteration is returned for iter iterations. Standard is same as iter
                            best = TRUE)) #only the best draw is returned

In short: My question is if there are circumstances under which it is reasonable that Latent Dirichlet allocation will cluster text in topics of equal size? Or is it something I should be worried if it happens?


texts: Having trouble scaling scores of logistic regression

texts:
I am constructing a credit scorecard using logistic regression, similar to the one shown here. However, when trying to convert the coefficients of logistic regression into score representation (by scaling the values using the provided formula) I am getting numbers that dont make much sense.
Formula used for calculating scores:
Score_i= (βi × WoE_i + α/n) × Factor + Offset/n

where βi is the coefficient of the logistic regression (of variable i),

WoE_i is the weight of evidence of corresponding variable,

α is the intercept of the logistic regression,

Factor is calculated as PDO / ln(2),

Offset is calculated as target_points - (factor * ln(target_odds))

n is number of variables used in regression

In my case PDO = 50, target_odds = 2, target_points = 500, n = 81, and intercept is -0.12686514.
Here is an example of one of my features:

As it can be seen, both WoE and the coefficients increase in value as the "revenue" variable increases. The score however does not act in this way. Initially I just assumed the scores acted in inverse proportion to those two values so I added a negative sign in front of the formula:
Score_i= -(βi × WoE_i + α/n) × Factor + Offset/n
But for some other features, the score is proportional to the other values:

And adding the negative sign would now make this feature have a score that does not make much sense.
How can I keep the scaling consistent? What am I doing incorrectly here?


texts: Hypothesis vs Hyperplane in Machine Learning

texts:
I am finding it hard to understand the clear difference between Hypothesis and Hyperplane.
I know that Hypothesis is a candidate model that maps inputs to outputs after training . And , Hyperplane is the decision boundary in a classification algorithm .
But , I can't seem to understand how the two are differentiated in equations .
Can someone help me understand their differences in  equations with some visualizations ?


texts: Subsequence parttern matching for time series

texts: I have a set of time series data (just like voice sequence data) with the pattern as shown in the first figure (theoratical data). The measured data is given as presented in the second figure. What I want to do is localizing/finding the subsequece pattern as shown in the red squares. Is there any algorithm to solve this problem? It looks like the classification/regression problem in machine learning, but I have no idea on how to start it.


texts: How can I get an algorithm to have an evalutation metric based on aggregate predictions?

texts:
Let's say I have a model that makes a prediction per individual. An example data set is below. Normally, evaluation metrics (for example within the XGBoost algorthim), are used at the individual observation. However, I don't care whether individual predictions are accurate, I just want predictions in aggregate to be accurate.

In other words, using the sample data set below, and aggregating predictions by Gender, I want to minimize the RMSE between (1000+3000) and (900+3100) for males, and (2000+4000) and (1900+4100) for females; this is as opposed to minimizing the RMSE in the normal sense per each observation.

Is there a phrase for this type of evaluation metric? Is there a way to imlement this into popular algorithms like XGBoost?

Note: There is no reasonable way to aggregate the data prior to training, and ultimately I do need predictions per individual.


texts: How to standardise/normalise csv data in a tensorflow 2 dataset

texts:
Does anyone know how to apply standardisation (or normalisation) on the numerical features of a dataset loaded in batch via tf.data.experimental.make_csv_dataset in TF 2?

The tutorial at https://www.tensorflow.org/tutorials/load_data/csv has the funny remark "The mean based normalization used here requires knowing the means of each column ahead of time" which is of course rubbish. The whole point of loading a CSV using the make_csv_dataset method is because you don't want to load it in full, because it is too large for memory (10 GB in my case).

I've looked at keras batch normalisation, as a layer in the model, but I'm building an anomaly detection model which is symmetrical, so I can't denormalise my data at the end to see how good the output values match the input values.

In my pandas dataframe test code (with a smaller CSV), where I load the entire dataset in memory, I can easily use sklearn.preprocessing.StandardScaler, but not with the datasets that are loaded in batch when the model is training.

I would be happy with a normaliser that works on the batches of my dataset, where I accept it is not fully correct, but close.

Is there a way? TIA


texts: Tensorflow - How to set gradient of an external process (py_func)?

texts:
I'm computing and optimization of some Variables that are used on an external process, but I get the error "No Gradient".

A heavily simplified (not tested) version of the code, but you can get the idea:

def external_process (myvar):
    subprocess.call("process.sh", myvar)
    with open('result.json', 'r') as f:
        result = json.load(data, f)
    return np.array(result["result"])

myvar = tf.Variable(1.0, dtype = 'float32', trainable = True)

loss = tf.reduce_sum( tf.py_func(external_process, [myvar], [tf.float32])[0] )

optimizer = tf.train.AdamOptimizer(0.05)
train_step = optimizer.minimize(loss)
sess.run(train_step)

I saw this discussion but I don't fully understand it: https://github.com/tensorflow/tensorflow/issues/1095

Thanks!


texts: Show the percent change week on week Data Studio

texts:
In Data studio I'm trying to get the percentage change week on week of a metric so that weekly reporting on changes is automated - right now I have to calculate the differences manually every week.

Visually this is the idea, but any type of chart / visualisation would be fine. Is this possible in Data studio?


texts: Find Cluster Diameter and Associated Cluster Points with KMeans Clustering (scikit learn)

texts:
I have done clustering using Kmeans using sklearn. While it has a method to print the centroids, I am finding it rather bizarre that scikit-learn doesn't have a method to find out the cluster diameter  (or that I have not seen it so far). Is there a neat way to obtain this for each cluster together with points associated with a cluster?

I currently have this rather kludgy code to do it

import numpy as np
from sklearn.cluster import KMeans
from sklearn import datasets

iris = datasets.load_iris()
X = iris.data
y = iris.target

estimator = KMeans(n_clusters=3)
estimator.fit(X)
print({i: np.where(estimator.labels_ == i)[0] for i in range(estimator.n_clusters)}) #get the indices of points for each cluster


texts: Inferring Relational Hierarchies of Words

texts:
I am new to natural language processing and I have not heard of a problem similar to mine yet. I was wondering if anyone could refer me to a method for solving my problem, or tell me how this problem is referred to in the academic literature, so that I can look for resources online.

Here is the problem :
From some text (wikipedia articles, for example), I would like to extract the hierarchy of different concepts that can be found in it. By hierarchy I mean a tree wherein A is a descendant of B if A or one of A's parents (transitive) is defined by B. For instance, normal distribution would be a descendant of probability (since normal distribution is defined using probabilities) and probability would be a descendant (or child) of mathematics. Since it is transitive, normal distribution would also be a child of mathematics.

One way I thought about solving this is by looking at the number of times a word A is used alone (called A), the words A and B are used together (called A AND B, 'together' could be, for instance, in the same article or in the same paragraph, or in the same sentence), and the number of times the word B is used alone (called B). Let A be mathematics and B be probability. Then, if the ratios (A AND B)/A and (A AND B)/B are low, then it could imply that there is no direct link between A and B (but a link could exist through transitivity). Conversely, if A is bigger than B, A is a bigger concept than B. If A and B are almost the same then they are probably siblings (children of the same parent).

Let's take 3 examples:

Mathematics (A) and carrot (B). A AND B is really low compared to A and B, so there is no direct link between them (or only an indirect link by transitivity).
Mathematics (A) and probabilities (B). A AND B is quite high compared to B, and A is much bigger than B, so B should be a child of A (probabilities is a child of mathematics).
Topology (A) and Probabilities (B). A AND B is relativaly high (the texts that present the different areas of mathematics will likely speak about the 2), A and B are about the same order of magnitude, so A and B should be the children of a same parent. Indeed, Topology and Probabilities are the children of Mathematics.

This way of solving the problem is far from perfect, for instance 'the' (A) and 'probability' (B) would probably end up saying probability is a child of the (because A AND B is huge and A is much bigger than B).

If anyone knows some papers on this or has any ideas on how I might solve this problem, I would appreciate some direction. Also, does my solution seem viable? How could it be improved?


texts:
We are currently using XGBoost model with Tweedie loss for solving a regression problem which works very good, now I wanted to move our model to Keras and experience with neural networks, do anybody know how I can implement Tweedie loss for Keras? I only care about the instance when p=1.5 which gives us the best result in XGBoost.

Thanks


texts: Can I dynamically change the hyper-parameters of a model?

texts:
Question

Can I apply different hyper-parameters for different training sets?
I can see the point of using the shared parameters but I cannot see the point of using shared hyper-parameters. The advantage of using the shared parameters is introduced in the following question: Recurrent NNs: what's the point of parameter sharing? Doesn't padding do the trick anyway?

More Details
1) Explanations on My Model
As my model is a stock trading model, I will explain to you how I trade stocks. Please bear with me for explaining how I trade stocks. It is not a long explanation.

I am using Bollinger bands to trade stocks. (All the stock data in this example is daily).
In short, I calculate simple moving average (SMA) and standard deviation of N days long stock prices. (Yes, I assume that the stock prices follow the Gaussian distribution.) The upper band is SMA + k*stdev, while the lower band is SMA - k*stdev.
I buy the stock when the stock price is above the upper band ('Too_High_Buy') or below the lower band ('Too_Low_Buy'). For the 'Too_High_Buy' case, I sell the stock when the stock price goes below the SMA. For the 'Too_Low_Buy' case, I sell the stock when the stock price goes above the SMA.

The parameters of my model is SMA and Stdev, while the hyper-parameters are N and k.

N: It decides how smooth SMA (the yellow line) will be.
k: It decides how far the upper and lower bands will be located from the SMA.

As different values of N and K show different characteristics, we should search which values of N and K are good for stock price data.

2) How to decide the hyper-parameters(N & k)

I use 'sliding steps' to decide the appropriate number for two hyper-parameters, N and K.
Sliding steps use the fixed amount of training data to decide the hyper parameter and check the performance of these hyper-parameters on the validation dataset, which immediately follows the training dataset.

I thought sliding steps is a good cross-validation tool to apply on stock data, because statistical properties of stocks can change while time goes by. For example, 30 years ago, McDonald and Coca-cola shows the similar price movements because they were sold together. However, nowadays, Coca-cola focuses on healthy drinks while McDonald stays as unhealthy food brands, they can show different price movements.

The hyper-parameter here can be several things, but for the sake of simplicity, let's say the hyper-parameters that we should decide is the duration of training dataset (N). The duration of dropped and forecasting is set to 1 day.

Using grid-search of different values of N and K, I calculate which values of N and K shows the best performance during the validation period in the training dataset.

3) My question

Can I use different k for different training sets? In other words, can I dynamically change which value of K to use based on the performance of sub-training dataset?

While performing the grid-search in the sliding step window method, we use the same value of K in all the training datasets to trade a stock during the validation period.

However, we can use different values of K based on the performance of sub-training data.

For example, let's say N is fixed to be 30 days. Then from the 1st Jan to 30th Jan, k=0.6 shows the best performance and we use this k=0.6 for the 31st Jan. Then from the 2nd Jan to 31st Jan, k=1.5 shows the best performance, then we use this K value for the 1st Feb, and so on.

Why should we use the shared hyper-parameter K all across the model? For parameters, it makes sense because it allows reduction of the parameters that the model has to learn. (source: Recurrent NNs: what's the point of parameter sharing? Doesn't padding do the trick anyway?)

But for using shared hyper-parameters, the amount of hyper-parameters is the same whether I use the shared hyper-parameters or not. It is only 1 hyper-parameter, which is K. The amount of computations needed is the same as well.


texts: Multiple regression with non-normal data in minitab - help

texts:
I am aiming to assess the effect of BMI (continuous) on certain biomarkers (also continuous) whilst adjusting for several relevant variables (mixed categorical and continuous) using multiple regression. My data is non-normal which I believe violates one of the key assumptions of multiple linear regression. Whilst I think it can still be performed I think it affects significance testing which is an issue for me. I think I can transform the data and then perform regression but I'm not sure and also have some questions regarding the implications of this. I have tried Box-cox transformation but MiniTab is unable to do this as some of the values are zero. I have performed Johnson transformation (which I think is a variation of Yeo-Johnson transformation) and now have columns of normal looking transformed data but - A) is this the right thing to do, B) if so do I need to do this for all non-normal variables or just the outcome variable, C) how will this effect the beta coefficient in terms of calculating the final quantitative effect BMI has on the variable given that I’m using transformed data and not the original?
I could potentially change my outcome variable to a categorical variety and use multiple logistic regression but not sure if/how this helps.
I am not thinking about non-linear regression at this point as this seems more complex (perhaps too much for me) and I’m hoping to solve this issues without in the first instance. Any help would be much appreciated.
Kind regards


texts: LSTM multivariate time series prediction is performancing very poor

texts:
I have been working on Multivariate time series in Python. The method I have used is LSTM. But the prediction is looking very bad. even my RMSE and MAE value is so high. RMSE=18.089, MAE=14.907. It is no good for prediction. I want to predict such a way that the error should remain in 0 to 1. The model I have applied to predict is-
from keras.models import Sequential
from keras.layers import Dense,LSTM,Lambda
from keras.layers import Dropout
#create LSTM model
model=keras.models.Sequential([

    keras.layers.LSTM(75,return_sequences=True,input_shape=(xtrain.shape[1],xtrain.shape[2])),
    keras.layers.LSTM(30,return_sequences=True),
    keras.layers.LSTM(10,return_sequences=True),
    keras.layers.LSTM(10),
    keras.layers.Dropout(0.25),
    keras.layers.Dense(1)

])

model.compile(loss='mae',optimizer='adam')
history=model.fit(xtrain, ytrain,epochs=100,validation_data=(xtest, ytest),batch_size=64)

But the overall prediction is not good. I have split the train and test data for 60% and 40% respectively.

How I can improve my prediction?


texts:
Many expert said "Batch has more local optimal possibility than SGD". But, I don't know the reason..

How SGD could avoid local optimal better than Batch?
(Some people tell me about over shooting as the reason. But, I don't also know there is more over shooting in SGD)

If you explain this reason by using Geometric, Mathmatics or other intuition concept, It's gonna be great!


texts:
I had a face to face interview for a data scientist job a few days ago. One of the questions I was asked was: in the case of classifier predicting the brand of TV from some features (price, size, specs, ...) out of 4 possible brands, how do you encode the brand variable? My answer was one hot encoding, it was accepted but then they asked me to do it explicitly and I sketched something like:

brand A -> [1,0,0,0]
brand B -> [0,1,0,0]
brand C -> [0,0,1,0]
brand D -> [0,0,0,1]

And then, I was corrected under the reason that these columns were not independent. And that the solution should have been three binary columns instead.

Later it hit me that I do not know why independence is required, and also that three binary variables are not independent. Two would be.

Can someone provide some explanation to help with my confusion?


texts:
I am looking at a problem where I have two documents, both are textual documents. The first one is a long (few pages long) textual document, while the second one contains about 10 short texts, each being a one liner. The problem is to automatically understand the 1st document and "map" key ideas from here to one or more short texts in the second document. I am not being able to get a good grip on the problem as to how to proceed. I tried the following approach:

first collected as much similar documents as possible which are relevant; I have obtained 136 such documents, each few pages long. Then build a document level embedding using each sentence as a document (doc2vec). Then for each short sentence in the second document, inferred an embedding based on the document model I built. Then tried to find the most similar sentences for each of these short texts from the 2nd document. However the results are not good. Only about 30% match is what I am getting and the  corresponding sentences are not much related to the target sentence.

I am wondering if this is the right approach or are there any other approach available which I am not aware of.

Your advice would be greatly appreciated. It may be the case that my explanation above is not adequate. Please let me know and I will try to improve it.

Your input would be greatly appreciated.

Thanks


texts: Predict buying behavior under the condition that a customer is advertised or not

texts:
I'm trying to predict response of customers on a marketing campaign.
As of now, I have data from one Marketing campaign and rfm data of my customers.

Some proportion of customers say 60%, received adverts. Roughly 10% responded

From the response data, i.e. whether purchased or not, on this marketing campaign I built a random forest using scikit-learn.

The model performs on a hold out set really well. but the most influential variable is the boolean: CustomerHasBeenAdvertised

I want to use this model, to select customers for a future marketing campaign.
To obtain the "purchasing probabilities" of the customers under the condition of similar advertisement, I set the variable CustomerHasBeenAdvertised to 1.

However, on a data set with this side condition, all the forecasts are above 0.5.

Is this extra ordinary high value due to the variable importance? Or are there other explanations?

Is setting the variable CustomerHasBeenAdvertised to 1 the false approach?

If so, how could one handle the case: Customer bought without beeing  advertised?

Should one simply neglect the information whether advertisement took place or not?

Thanks in advance


texts: SVM, which range to use when normalizing

texts:
I am using the SVM classifier from Scikit Learn.
I was wondering is there is a know-best-practice when it comes to normalization. I'm using different normalization tecniques, but all my normalized data are now between 0 and 1.

Using GridSearchCV, I always end up picking the linear kernel for the SVC. Would it be better if I were to normalize having my values between -1 and 1?

I feel like the answer is yes, thinking about situations like this one:

But I wanted to know if there is an official answer. Thanks a lot


texts: How is Stochastic Gradient Descent used like Mini Batch gradient descent?

texts:
As I know, Gradient Descent has three variants which are:
1- Batch Gradient Descent: processes all the training examples for each iteration of gradient descent.
2- Stochastic Gradient Descent: processes one training example per iteration. Hence, the parameters are being updated even after one iteration in which only a single example has been processed.
3- Mini Batch gradient descent: which works faster than both batch gradient descent and stochastic gradient descent. Here, b examples where b < m are processed per iteration.
But in some cases they use Stochastic Gradient Descent and they define a batch size for training which is what I am confused about.
Also, what about Adam, AdaDelta & AdaGrad, are they all mini-batch gradient descent or not?


texts: Missing outputs in multiple-output neural net

texts:
I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was to use a neural network (e.g. adding multiple outputs to a pre-trained covnet) and I would try keras first (having used it before). I assume it would be useful to try to do this all in a single covnet (especially based on a TWiML talk on this paper, where the author suggested that in general one prediction target will help to improve features for another and given that e.g. how one would recognise gender will differ by breed etc.).

My question arises, because for some training/validation/test images some (but not all) of the multiple prediction targets may be missing. Sometimes we will just know the breed, for some images we will not know the gender and the exact age may oftne be unavailable (but only that the animal is adult versus younger than about 4 months, which a human can easily label by just looking at the animal).

If I had to guess, I would expect the missingess to be missing at random (but likely not missing completely at random). For example, for some breeds and ages it is harder to tell the gender by just looking at the animal (perhaps the neural network will manage to do it). So, someone who just takes a photo of an animal may not know the gender and humans are no good at labelling gender from photos. Another example: age might be unavailable, because someone just took a photo of an animal they do not have detailed information on or because someone bought an animal secondhand and never found out.

Is there some standard way of handling this? My intuition is that even records with one or two missing prediction target still should provide useful information for the other ones. Thus, listwise deletion of all records with any missing target information would be extremely inefficient (and possibly even otherwise problematic?!).


texts: On design of the training set: conceptual question

texts:
I am curious to know how training data should be constructed so that it scales to examples that are not a part of the training data. For example, the problem that I am facing right now is in the application of identifying or distinguishing the frequency response of time series that are generated from different distributions. So I constructed $p$ number of examples each from Gaussian, Uniform, Poisson and a kind or colored noise say pink. The White noise examples (Gaussian, Uniform and Poisson) are labelled as 1 and colored noise as 0. Using Neural Network the classification works fine. Now I wanted to do sensitivity analysis by checking if the trained network can classify white noise from another distribution and also colored noise say red. Both the tests failed. NN failed to classify them. But, as soon as I included the red and the new kind of white noise in the training data and tested on a different trail (time series), the NN could classify it.

QUESTION: This behavior makes me wonder if machine learning algorithm sare incapable of distinguishing examples from different systems eventhough the examples in testing have similar properties as those used in training. In this case, eventhough white noise appears similar but since they are generated from different distribution or say systems the training data must include examples from all the generating mechanism or systems, otherwise in testing the ML model fails to recognize it. Is this the usual behavior?


texts: Should I rescale losses before combining them for multitask learning?

texts:
I have a multitask network taking one input and trying to achieve two tasks (with several shared layers, and then separate layers).

One task is multiclass classification using the CrossEntropy loss, the other is sequence recognition using the CTC loss.

I want to use a combination of the two losses as criterion, something like Loss = λCE + (1-λ)CTC. The thing is that my CE loss starts around 2 while the CTC loss is in the 400s.

Should I rescale the losses at each epoch with a Max(L₁)/L₁ factor, where Max(L₁) is the maximal loss at epoch 1 and L₁ is each “sub-loss” at epoch 1. That is we scale the loss so that at the first epoch they have the same magnitude and then we keep scaling using those factors.

Is there a better approach? How do I ensure that my two losses have the same influence on the backpropagation with respect to λ?


texts: Correcting for one of multiple strong batch effects in a dataset

texts:
I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it originates from one variable, without taking off the potential batch effect from other variables.

If this is unclear, taking a short example is probably the best way to go to explain my problem:

Imagine that we have 10 persons taking part in an experiment. The experiment is as follows:

each person is given a set of a 1 000 numbered tennis balls, all of which we know the physical properties (for instance, weight, diameter and colour of the ball)
each person is then asked to throw the 1 000 tennis balls one by one as far as they can, and to record their results

Following the experiment, we will know for all 10 000 tennis ball:

the distance at which the ball was thrown
who threw the ball
the ball's weight
the ball's diameter
the ball's colour

Now, because not everyone has the same capabilities for throwing tennis balls (be it in terms of muscle strength or something else), we can expect to see some strong batch effects within the data (for instance, we could observe that a ball thrown by the first person will, on average, have been launched farther than a ball with the same weight and diameter when it's been thrown by the second participant, etc...).

Correcting for this kind of batch effect can be done in multiple ways if everyone had been given the same set of balls (in a setting with normal distributions, standardisation would probably work fine). Now, imagine that when organising the experiment, we did not pay enough attention and ended up giving some people heavier tennis balls, some others smaller tennis balls, etc...

At the end of the experiment, we realise indeed using a Chi2 test (or, say, Kruskal-Wallis H test) that everyone was not given a set of balls coming from a random sampling of all 10 000 balls.

How can we then correct for who threw the ball, without taking off the batch effects originating from the fact that the set of balls were different?

The main problem is that by correcting for batch effect using regular standardisation (for instance), we will probably end up removing the effect due to the fact that some people were given heavier or larger balls.

Or, in other words using the example, how could we account for the difference in terms of strength between the first and the second participant while not correcting for the fact that the first participant had in average heavier tennis balls than the second?

At first, I was thinking of running a Generalized Linear Model with the dependent variable being the distance at which the balls were thrown, and all the other variables as regressors, and then subtracting to the dependent variable only the effect of the variable for who threw the ball. I am however unsure of whether this would or not make statistical sense, which is why I ask if other techniques can be used, or if this one would work.


texts: Feature matrix for email classification:

texts:
This is in continuation of my earlier post.

In my previous model, I used just two features which couldn't fare well and gave 71% prediction (accuracy) score. Now, I'm trying to consider another important field "subject" from the data set. I'm planning to use NLTK package along with Tf-Idf.

I generated the below using NLTK (FreqDist) on the subject field -

Here is my idea:
I'll tokenize the subject field, apply stopwords, then Tf-Idf and get the top N most frequent words and use them as features. The Tf-Idf score will be my values. In total, I'll have (N+2) features.

But, one downside is - the feature matrix will be huge (say 200+ features) and sparse. Is that a concern? Does this design make sense?

Your thoughts please.


texts: Which distance method will account for positions of observations in arrays?

texts:
I am trying to figure out the distance method suited for my problem. I have different arrays of the same length. Each position in the array matters.
Let's say ...
each array is representing an image, and the value at the last position of the array will tell us that the observation point was found in a far end of an image, etc. And the value in that position of the array is the intensity of the color. What I am trying to do is calculate the similarity distance from one image to another. What distance method would be best suited for this problem?


texts: Confidence interval for class membership probabilities

texts:
I would like to calculate confidence intervals for predicted probabilities of a class membership obtained with randomForest.
I know I could use predict.all in randomForest(), which gives me the predictions of all the trees and hence a way of calculating the variance of the average prediction. However, from reading the following paper by Wager, Hastie and Effron, the variance obtained this way is biased upwards, https://arxiv.org/pdf/1311.4555.pdf
This paper presents a bias-corrected estimator of variance and states on page 8 that it has been implemented in randomForest.
I have checked the documentation but can't find any reference to it. https://www.rdocumentation.org/packages/randomForest/versions/4.6-14
Can anyone shed light on this?


texts: Fast Python implementation of the gradient descent

texts:
I'm looking for fast Python implémentations of gradient descent optimization algorithm. I have a convex problem , with no constraint, so for now I'm using the BFGS algorithm implemented in scikit-learn ( minimize ).

Is there anything faster / scalable on multi-core systems ?


texts:
I'm trying to get a better grasp of model interpretability using many different kinds of models for a binary classification problem.

Quick note: By interpretability in this case, what I mean is understanding which features have the largest effect on the target. (Not necessarily understanding how the model works under the hood, although I do understand this fairly well for many models.)

For Logistic Regression, I know that you can inspect the model coefficients. Assuming that the features were normalized before training the model, the magnitude of each coefficient is proportional to its importance, and also shows the direction of the influence. (Correct?)

For Adaboost (in Python's scikit-learn package), you have access to model.feature_importances() which shows the importance of the feature, but not the direction. By using partial dependence plots, you can also get the direction of the influence.

The main question: Am I correct in assuming that using feature importance with partial dependence plots for Adaboost gives a fairly good view of how to understand which features influence the target, similar to using model coefficients for Logistic Regression? Is there a better way that people usually do this?

Another quick note: I know that for right now I'm leaving aside the question of model accuracy. I'm assuming in this case that we want to build a model for the sake of interpreting relationships, not necessarily for predictive power.


texts: What happens if at leaf node both classes have same number of samples?

texts: I analyzed a small dataset which had three features, so I kept max_depth of decision tree to be 3, in doing so I found it something intresting, there was a leaf node which had number of samples of both classes to be equal and decision tree choose one class, now I am intrested to know how class is decided in such scenario, is it random or some other criteria, I have attached image to explain my scenario


texts:
I have a question related to an alterative Q-Learning approach. I'd like to know if this already exists and I am not aware of it, or it doesn't exist because there are theoretical problems behind it.

Traditional Q-Learning
In traditional Q-learning, the update of the Q-value happens at every iteration. The agent is in state s, performs action a, reaches state s' and obtains reward r. The Q-value for that pair state-action is updated according to the Bellman equation. As an example, let's consider an agent exploring a grid-world. One specific cell in the world gives a positive reward. Until the agent finds the cell leading to a positive reward, the agent gets small or no rewards. Then, after reaching the cell with the high reward, all the previous cells gets updated iteratively with positive Q-values when they are explored. This means that we need many iterations, and especially if the path leading to the positive reward is very long or requires a specific set of actions, it can take a significant amount of episodes (or never be fully discovered, even if it was explored previously).

Alternative approach
My question is: why can't we update all the Q-values related to the agent at the end of the episode? That means, information is cached during iterations and when the agent terminates the episode, the Q-values gets updated for all the cells that led to that reward (or did not lead to a reward). In this case, the agent doesn't need tons of episodes to update the Q-values related to that path.
Is this approach legit? Which problem do we face following this?
The only drawback I can think of is lack of exploration, that can be solved choosing random actions according to a decreasing epsilon value.

EDIT

My problem is not computing efficiency, but it's mainly that I am using a descending learning rate, equal to $$\alpha = \frac{1}{1+number\_visits}$$
For this reason, my intuition says that updating the Q-values at the end of the episode should work better. Indeed, since the learning rate depends on the number of visits, the contribution of the early visits is way higher. Updating as soon as possible should reduce the negative impact of this aspect.


texts: How to programmatically suggest changes to a text to match some other text?

texts:
I'm currently thinking of the system that would compare 2 texts (source, target) and suggest changes to the source text to match more the style of the target. It should not convert it to the target but more suggest to rephrase some sentences, changes words, add some key words and so on.
Any ideas how this could be done?
Thanks a lot for any suggestions!


texts: Binary classification: how to transform features in real numbers?

texts:
I want to train a binary classification algorithm for spam detection using labeled data set. The dataset has the following features:
Email address, text message (split into subject and corpus), date

An example of data is:
Email | Subject | Corpus | Date
Gg12_school@gmail.com | Example | this is just an example of my dataset  | 2020/08/20

What I would like is to transform data features in real numbers and binarize email addresses.
As algorithm I was thinking of
SVM and/or Naïve Bayes.
My difficulties are, however, in how transform data features in real numbers in order to get more parameters in my classifier.
I am using Python.
Could you please give me an example of how to do it?


texts:
For experimenting we'd like to use the Emoji embedded in many Tweets as a ground truth/training data for simple quantitative senitment analysis. Tweets are usually too unstructured for NLP to work well.

Anyway, there are 722 Emoji in Unicode 6.0, and probably another 250 will be added in Unicode 7.0.

Is there a database (like e.g. SentiWordNet) that contains sentiment annotations for them?

(Note that SentiWordNet does allow for ambiguous meanings, too. Consider e.g. funny, which is not just positive: "this tastes funny" is probably not positive... same will hold for ;-) for example. But I don't think this is harder for Emoji than it is for regular words...)

Also, if you have experience with using them for sentiment analysis, I'd be interested to hear.


texts: BERT :dropout(): argument 'input' (position 1) must be Tensor, not str

texts:
I am new to NLP and would like to build a BERT model for sentiment analysis so I am following this tutorial.
However, I am getting the error below:
F.softmax(model(input_ids, attention_mask), dim = 1)

When I would like to execute this cell I get the error:
 dropout(): argument 'input' (position 1) must be Tensor, not str


texts: Can we use a predictive model of a device to another device of similar specification?

texts:
Let's say, we have a model M1 that is trained on a particular device D1 (let's say the device is a Car), with historical data over 6 months of D1.
M1's purpose is to predict the behavior of the car. Now we have another device D2 (another car), with same specification but new and yet to be run.

Can we use M1 to predict the behavior of D2, since we do not have any data on D2?


texts:
This is the first time I attempt to use machine learning with Keras. In contrast to others I need to use one of the disadvantage of such algorithms.
I need a function that accepts an angle and distance to an object and output a new angle and power (imagine aiming for an object with a bow for example and the algorithm tells me how far up my arm should go and the bow's power).
There's nothing predictive in this configuration. I will generate a large set of 4D (input,output) data with every possible case. I want the AI to "evaluate" some inputs and return the corresponding outputs for that set of inputs, in other words to remember the data and output the same numbers.
I need an AI for this task because I need smooth values between inputs values it has never seen (limited interpolation)
I have used two models:
model = Sequential()
model.add(Dense(12, input_dim=2, activation='relu'))
model.add(Dense(24, activation='relu'))
model.add(Dense(24, activation='relu'))
model.add(Dense(24, activation='sigmoid'))
model.add(Dense(2, activation='linear'))

Which now I know is incorrect because Sigmoid is used for binary classification. Still, it works! I end up with a mse of 4. I did not manage the get the same loss with all-ReLu layers with the same # of epochs.
model = Sequential()
model.add(Dense(12, input_dim=2, activation='relu'))

model.add(Dense(24, activation='linear'))
model.add(LeakyReLU(alpha=0.1))

model.add(Dense(24, activation='linear'))
model.add(LeakyReLU(alpha=0.1))

model.add(Dense(2, activation='linear'))

This models has a loss of 5.43 after 500 epochs and seems to stabillize here.
Note:

I am forced to retrain the model because I generate the training data very fast. I need to stay with a model that will keep going reduce its loss.
Normalization is worse than anything I have seen.
Because my model should be sensitive to different very close inputs, I tested with batch values from 5 to 2.
My dataset is currently 1100 of lines
This model will be trained very closely to the data I give it. It should not matter how many lines I feed it because I don't want prediction or generalization. I want the AI to output what it has seen for a corresponding set of inputs. That would mean overfitting the AI to the maximum until it reaches a very low loss. Then I can test it for exactly those values it was trained on.

Should I continue with the first model? Does it make sense to use the Sigmoid layer? How can the second model be improved?
Sample of my data:
theta[-90,90], distance [0,40],  theta_output[-90,90] power[0,1,2]
0.0,8.696802,0.25688815116882324,1
-1.990075945854187,5.455038,11.56562614440918,1
-56.81309127807617,3.1364963,-53.07550048828125,1
-38.21211242675781,4.718147,-32.30286407470703,1
-33.828956604003906,5.163292,-35.61191940307617,0
-27.64937973022461,6.182574,-25.107540130615234,1
2.8613548278808594,13.922726,-2.3708770275115967,2
-8.812483787536621,14.951225,-3.919188976287842,2
0.0,21.448895,-3.9320743083953857,2


texts: Can an algorithm tested only on artificial data be accepted in a high rank conference?

texts:
I devised a classification algorithm which is useful for specific complicated distribution of classes of data. The method works good on the artificial data which cannot be solved by typical algorithms in the literature. But yet i couldn't find any real data to evaluate the algorithm on real problems.
Also there is mathematical support for the basis of the approach and the way it works.
So i was wondering if an algorithm with good mathematical support but no evaluation on real data has any chance to get accepted in computer science conferences like SDM or CVPR?


texts: Multi-ouput Classification + Sentiment Analysis

texts:
I am facing some issues with a text classification problem and need your help to understand the best way to approach it.

The problem statement is as follows:
Assume I have a set of sentences describing a certain product (assume they are furniture items). Each of the sentences needs to be tagged with a class coming from a roster of 10 possible classes. The 10 classes refer to possible attributes of the item, such as softness, brightness etc.
On top of this, sentences contain sentiment information on each item (good, bad, neutral). Example: "I am satisfied with the performance of this couch, it is really soft".

The first question is: how would you approach the problem of labeling each sentence with both classes AND sentiment?

Now, to make things even more complex, assume each sentence can describe up to 3 items, with relative sentiment.
How would you approach the multi-ouput sentiment+class problem in this case?

Thanks a lot in advance for any help!


texts: Reducing the GPU memory usage when the model is already small enough

texts:
I trained a model and froze it into a PB (protocol buffer) file and a directory of some variables, and the total size is about 31M. We deployed it using a GPU card and followed this answer and set the per_process_gpu_memory_fraction to a very little number to make the memory to be about 40M. The program performs very well but when we check the GPU usage by nvidia-smi which shows that the memory usage is about 500M.
Then my question is how can I justify this gap? How can we reduce that? Can we do something like quantization to decrease the 500M? We want to deploy it into an edge device so the 500M is too large.


texts: Is it Worth Installing Cesium on Windows?

texts:
I am using Python(Anaconda package). I know Cesium requires a Cython and a C/C++ complier because I have a windows computer. I was looking at the installation instructions and it appears to be complicated.
Is it possible that by installing Cesium something goes wrong and I won't be able to undo it? I am wondering if working on time series projects requires these tools?


texts: How can you adjust a prediction based on features in the future being different than predicted?

texts:
I have a model that takes mostly cumulative data, and makes a prediction. It's not baseball, but I'm using this as a pretty accurate analogy. You put in all the totals so far, and it make a prediction of where a target will be in the future.

games_played|runs|runs_against|hits|hits_against|games_won
          20| 100|         120| 200|         240|       10

Expected games won after 5 more games: 12

What I'd like to do is answer questions like "How many runs are we expected to have in N games? And if we had more runs than that, how many more wins would that get us?"

             |games_played|runs|runs_against|hits|hits_against|games_won
          now|          20| 100|         120| 200|         240|       10
    predicted|          25| 120|         150| 240|         300|       12
user_adjusted|          25| 130|         140| 260|         280|       13

The only way I can think of to do this right now is to have many models, one for each feature, and put all the predicted features into one other model. Then users can update individual features. That feels like excess work and prone to errors, but I could be totally wrong. I can't just linearly forecast the features in the future because the relationship is more complex.

What is the algorithm/method/tool that's best suited to this? I'm using scikit-learn right now.


texts: Group_by field is not showing in the summarise output in R

texts:
In R,using dplyr package, I tried the function "summarise" and I expect the result to show along with the groupby field. However, all of a sudden I see summarized output but without the groupby filed which makes the results meaningless. Any one any idea?

For example if data has ages and salaries for members belonging to a team and I try to view the Average Age and Salary by Team the code below returns just the averages without the team names

Code:
df_summ = df %>% group_by(Team)%>% summarise(AvgAge= mean(Age),AvgSalary= mean(Salary))

So without the groupby column "Team" the result is meaningless

What Am I missing here?


texts: How to approach deep learning CV/Resume parser using Convolutions?

texts:
I'm currently looking to invest some time on how to make a resume parser using deep learning.

I need some initial ideas (or) approaches on how to put together things at first, where to start. If you have any good approaches on handling resume parsing for multiple type resume structures please let me know.

Also to start here's my approach I was thinking, do let me know if you think this works!

1) First I will take a 100 resume as a image(i.e., jpg) file.
2) I will create bounding box around projects/education area and create my training data.
3) After training I'll try to predict it on test data.

Is this an ideal approach? Are there any good approaches you can suggest which works based on any paper?

Much appreciated. Thanks.


texts: How to structure data and model for multiclass classification in SVM?

texts:
I am trying to predict a categorial variable given a set of input variables, which are also categorical. Both the target and features variables only take the class values [below mean, mean, above mean].

I have used one-hot encoding on both the target and feature vectors, but I am not sure if this is correct.

If I have 2 feature vectors I get 2x3 columns after one-hot encoding. For example, the first instance may be:

X[0] = [0,1,0,    1,0,0]

indicating the first feature has mean value ([0,1,0]) and the second feature is below mean ([1,0,0]), with the corresponding target

y[0] = [1,0,0]

indicating the target is below mean.

With the encoded X and y, I have tried to do this:

svm_clf = SVC()
svm_clf.fit(X_train, y_train)
y_pred = svm_clf.predict(X_test)

but it gives me 'ValueError("bad input shape {0}".format(shape))'

I have given the input and output below for the real situation, where the input has 9 features (so 9x3 columns after encoding). The error message indicates there is a problem with the 'y' vector.

Should I be one-hot encoding the target?

How can I tell the classifier that the output can only take a "single value" i.e. the 3 columns in the target are not independent as they all belong to the one variable. For example, the output for a given instance cannot be [1,1,0] as this indicates it is both "below mean" and "mean" which is not possible.

I have also tried a random forest classifier which ran OK, but the results were not plausible, so I assume I am doing something wrong,

Input features:

X_train type:  and shape: (872, 27) and head(5):

[[ 0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]
 [ 0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]
 [ 0.  1.  0.  0.  1.  0.  0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]
 [ 1.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  1.  0.  0.  1.  0.  0.  1.  1.  0.  0.]
 [ 1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  0.  0.  1.  0.  1.  0.  1.  0.  0.]]

Target:

y_train type:  and shape: (872, 3) and head(5):

[[ 0.  0.  1.]
 [ 0.  0.  1.]
 [ 1.  0.  0.]
 [ 0.  0.  1.]
 [ 0.  1.  0.]]

Error message:

  File "analyse.py", line 287, in dummy_test
    svm_clf.fit(X_train, y_train)
  File "/usr/lib64/python3.6/site-packages/sklearn/svm/base.py", line 149, in fit
    X, y = check_X_y(X, y, dtype=np.float64, order='C', accept_sparse='csr')
  File "/usr/lib64/python3.6/site-packages/sklearn/utils/validation.py", line 578, in check_X_y
    y = column_or_1d(y, warn=True)
  File "/usr/lib64/python3.6/site-packages/sklearn/utils/validation.py", line 614, in column_or_1d
    raise ValueError("bad input shape {0}".format(shape))
  ValueError: bad input shape (872, 3)


texts: What is the difference between step size and learning rate in machine learning?

texts:
I am using TensorFlow to implement some basic ML code in Python. I was wondering if anyone could give me a short explanation of the meaning of and difference between step size and learning rate in the following functions.

I used tf.train.GradientDescentOptimizer() to set the parameter learning rate and linear_regressor.train() to set the number of steps. I've been looking through the documentation on tensorflow.org for these functions but I still do not have a complete grasp of the meaning of these parameters.

Thank you and let me know if there is any more info I can provide.

(I posted this on Stack Overflow before I knew there was a Data Science forum board too, sorry)


texts: How to visualize data skewness using Google Data Studio?

texts: My dataset is stored in BigQuery and I like to use Google Data Studio to visualize data distribution/skewness of all the columns (in dataset) using graphical representation.


texts: Support Vector Machines: How can you generate higher dimensional data from lower ones?

texts:
I recently read a post describing the Support Vector Machine theory and brought about some idea that says it throws data from n dimensions to some higher dimensional space.

SVM description

I wanna know how can we generate more dimensional features from given n features?

Whats the basis of these extra dimensions? Won't these be dependent on the previous n dimensions, and if so how can we say that these are independent "dimensions"??


texts:
how can i plot my results from gridsearch csv?
clf = GridSearchCV(pipeline, parameters, cv=3,return_train_score=True)
clf.fit(x, y)
df = pd.DataFrame(clf.cv_results_)

i'm trying to get a similar plot to what is here: https://matthewbilyeu.com/blog/2019-02-05/validation-curve-plot-from-gridsearchcv-results , but this uses the grid search object and i have tried and failed at trying to get the same using just the gridsearch df (from above).
can anybody help in how i go about this?


texts: Real world performance evaluation on problem similar to malware detection

texts: I have a question on how to evaluate the performance of models similar to malware detection in the real world. For example, I have created a malware detection model to predict whether the engine will be down. And the model seems to do a decent job so I push it to production. However, in real-world, when the model predicts the engine is going to be down, I will send someone to fix it. The tricky part is I am doing the prevention by sending someone to fix the engine before it has been down. Are there any good methods to evaluate the performance of the model? Thanks!


texts:
I am trying to score each product based off a couple of features. I was thinking I can use the sales rate(7 days prior) as the target column and predict the sales rate using some ensemble technique and attest the score based off the sales rate value.

I feel there are many other ways to do it, but some guidance here will really be helpful. I am trying to look for a good place to start. Also, maybe if its just scoring, are there other statistical techniques I can use instead of machine learning?

End Goal: Try to attest a score to each product, for example: if target has a total of 10000 products, each of these products should have a score based off the input features.


texts: what indicators can be used to classify a group of stocks?

texts:
I am working with a dataset contains the daily return time series of 50 different stocks, I want to divide these stocks into several different groups.
My idea was to make a new dataset contains some indicators of these 50 stocks, and use a cluster algorithm to classify them, for example Hierarchical Clustering Method and plot dendrogram to decide how many groups make most sense. The indicators I am using are: Mean, Standard Deviation, Skewness and Sharpe ratio.
First of all, my economic background is weak and I was wondering if there are any other well used indicators could be added to this dataset to make the classification make more sense.
Secondly, too many indicators can make it difficult to interpret the stocks being classified.
Can someone share some ideas?


texts: Why do probabilities sum to one and how can I set optimal threshold level?

texts:
I am working on a text classification use case.
The training data has two classes, so the XBBoostClassifier  and onevsrest model is classifying the test data into either of the two classes.
But my requirement is to classify either into given the classes or if no match found, then set it as 'undetermined" so that i can manually classify the data.

I tried the following OneVsRest Classifier

pl = Pipeline([
        ('vec', CountVectorizer(token_pattern = tks)),
        ('clf', OneVsRestClassifier(LogisticRegression()))
    ])
pl.fit(x,y)
predictions = pl.predict_proba(test.comment_text)

But the sum of the probabilities is one and moreover the probability is above 90 for the class to which the data belongs to.

Please clarify the following points
1. Why the probability is always one? whether it means that the data is mutually exclusive?
2. The probabilities are like this

CLASS 1           :CLASS 2
0.892993358265023 : 0.106808845640795
0.999999742528922 : 2.57685096542208E-07

Does that mean that in first case, probability is only 90% for class 1 and hence the classifier is not able to classify the data properly. However in other case, there is clear cut difference as the probability is around 99%

Can I set the threshold, like 90%, and conclude if the probability is less than 90% , let the users manually classify the data?

Please provide your inputs


texts:
I was read and paper for machine learning, and i found this term "multi-hot encoding" without explanation.
Can you help me please?
the paper: https://arxiv.org/abs/2001.06917


texts: Convert Out-of-bag (OOB) estimate to quad weighted kappa score

texts: Is there a way to directly calculate an approximate quad weighted kappa measure from an OOB estimate, obtained from a gradient boosting model with subsampling without going through cross validation?


texts: Resizing images for training with Mobilenets

texts:
I have a script to download images, but the images are of different resolutions so I have written a script to shrink the image. I have two options:

size=(224,224)

with cv2

cv2.resize(img,size,interpolation=cv2.INTER_AREA)

with PIL

img.thumbnail(size,Image.ANTIALIAS)

after saving them I see cv2 doesn't maintain the original ratio where as PIL maintains the ratio.

My question :

Maintaining aspect ratio is important or not.
If yes (224,224) is a good choice or should I set it to higher
resolution.

Sorry if the question is naive, I am new to image processing.


texts: Classifying variable types on a list of variables

texts:
I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the variable type. I wanted to see if I can use some form of unsupervised learning to aid in this task. Would appreciate any advice/suggestions.

Let me elaborate on what I'm working with.

When I mention variable type, they're all numeric, but I'm trying to classify them into categories like a dollar-amount, age, number of something based off of the name of the variable as the rules for invalid flag differ by those categories.

Because of this, I'd like to classify my variables into things like:

Dollar Amount
Age
Number of items
etc...

Here's an example of what invalid values look like:

Invalid values for a variable of type "Number of items":

6,7,8,9

Invalid values for a variable of type "Dollar amounts":

99996, 99997, 99999

Some additional points:

These variables have maximums, like 9 for number of things. But isn't the most reliable to filter by, as it could definitely affect other types like $ amounts.
The variable names can sometimes be telling of the type of variable

1) It could have a keyword in the variable name like "N_" to indicate that variable is number of items.

2) Sometimes the rule may not be so simple and can be confounded by other keywords, eg: N_ITEMS_PCT_50. This is actually a number of items with a percentage over 50% rather than a percentage value.

Some of the features I collected to help with measuring similarity:

1) The variable names, of course

2) Maximum values of each variable

3) # of times that an invalid flag (for all types) would appear for each variable. I would do this by calculating how many observations fall in the range of invalid values. So if I did this for "number of items", I would count number of observations ranging from 6 to 9. I would calculate another column to look for invalid dollar amounts by counting number of observations between 99996 to 99999.

I'm interested to see if this could be a viable approach as I'd try have my work cut out for me rather than making this a very manual process for 700 variables. Would appreciate any insight.

Thanks


texts: predicted bounding boxes that stretch beyond grid cell (Andrew NG CNN course)?

texts:
I was following Pr. Andrew Ng course on Course about Convolutional neural network and I have a doubt regarding one of the points he mentions in the Yolo algorithm.

In one of the slides he mentioned two key points:

1) For each grid in our $3 \times 3$ grid image will have 2 predicted bounding

2) And each of these bounding boxes will be bigger than the size of the grid.

I couldn't get why there will be $2$ predicted bounding boxes? Is it because we consider two anchor boxes?

Also, how can the bounding boxes bigger than the size of the grid? Because we know that each object can belong to one grid only based on the midpoint


texts: Tools for project management Data Science

texts:
I'm in charge of a small data science team (3 data scientists, me included). We do our projects with at least one business person (PM) per project ( we have 5 of these).
We managed everything with meetings and emails, but as the number of projects and people keeps increasing, I find it necessary to have a proper management tool.
I would like to have something were we could, per project, add business needs (requirements). These requirements could translate into coding actions. One could estimate the time needed for each action.
It should be possible to then assign these tasks to DSs, and organise the priority to be given to each task. This way the whole team could see what need to be done in each project, what each DS is going to do after, and when each task is going to be completed.
I'm looking for a free or very cheap solution. I looked at Jira, but I'm not sure if it's a good solution.


texts:
How the neural networks are overfitted for regression.Either it tries

to equal individual observation values or equals to the sum of all observations


texts: How to do Named Entity Recognition in Tables?

texts: What are approaches to do Named Entity Recognition on Tables? I am referring to tables which have a column header and the corresponding information in the cells below the header, or the respective with line headers, not tables which contain full sentences. Since these NER lack the context of a sentence and are sort of determined by a header, the approaches I found (most depend on context) do not really work.


texts: Differences between a Statistician and a Data Analyst in industry

texts:
What is the Difference in the job of a Statistician and a Data Analyst In Industry? My take is that although both analyse data, a Statistician deals with the more theoretical aspects of data such as using mathematics to analyse data and to create mathematical models of data. So their work is more towards the mathematics side of things. Whereas the job of a Data Analyst deals more with the programming and side of things and their job is more practical in nature and they focus more on the implementation side.

Im more of a theoretical person and im more mathematically orientated and i wanted to do a job that involves lots of math, so is a statistician job more suited for me rather than a Data Analyst?


texts: Sklearn SVM - how to get a list of the wrong predictions?

texts:
I am not an expert user. I know that I can obtain the confusion matrix, but I would like to obtain a list of the rows that have been classified in a wrong way in order to study them after classification.
On stackoverflow I found this Can I get a list of wrong predictions in SVM score function in scikit-learn but I am not sure to have understood everything.
This is an example code.
# importing necessary libraries
from sklearn import datasets
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split

# loading the iris dataset
iris = datasets.load_iris()

# X -> features, y -> label
X = iris.data
y = iris.target

# dividing X, y into train and test data
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

# training a linear SVM classifier
from sklearn.svm import SVC
svm_model_linear = SVC(kernel = 'linear', C = 1).fit(X_train, y_train)
svm_predictions = svm_model_linear.predict(X_test)

# model accuracy for X_test
accuracy = svm_model_linear.score(X_test, y_test)

# creating a confusion matrix
cm = confusion_matrix(y_test, svm_predictions)

To iterate through the rows and to find the wrong ones, the proposed solution is:
predictions = clf.predict(inputs)
for input, prediction, label in zip(inputs, predictions, labels):
  if prediction != label:
    print(input, 'has been classified as ', prediction, 'and should be ', label)

I didn't understand what is "input"/"inputs". If I adapt this code to my code, like this:
for input, prediction, label in zip (X_test, svm_predictions, y_test):
  if prediction != label:
    print(input, 'has been classified as ', prediction, 'and should be ', label)

I obtain:
[6.  2.7 5.1 1.6] has been classified as  2 and should be  1

Is the row 6 the wrong row? What are the numbers after the 6.? I am asking this because I am using the same code on a dataset that is bigger than this one, so I would like to be sure that I am doing the right things.
I am not posting the other dataset because unfortunately I can't, but the problem there is that I obtained something like this:
  (0, 253)  0.5339655767137572
  (0, 601)  0.27665553856928027
  (0, 1107) 0.7989633757962163 has been classified as  7 and should be  3
  (0, 885)  0.3034934766501018
  (0, 1295) 0.6432561790864061
  (0, 1871) 0.7029318585026516 has been classified as  7 and should be  6
  (0, 1020) 1.0 has been classified as  3 and should be  8

When I count every line of this last output, I obtain the double of the lines of the test set... So I am not sure that I am analysing exactly the wrong list of predicted results…
I hope to have been enough clear.


texts: Trying to use CBOW for tweet classification

texts:
I'm trying to use the Continuous Bag Of Words method for word embedding on a corpus of 7503 tweets.
In particular, I'm trying to use CBOW on this Kaggle competition, which involves classifying tweets depending on whether they refer to disasters.
I followed the instructions in this article, and then trained a linear classifier on the average vector of the words in a tweet. The classifier performs very poorly, even on the training set (it only classifies less than 60% of tweets correctly).
There are two reasons I think this might be happening:

Training time. So far, I have trained the CBOW network for 150 epochs, which took about five hours with GPU acceleration. I don't know how much training is generally required, so I have no idea if this is enough. It is currently in the process of running for 500 epochs, but I'm asking here as well, in case there is something else that I'm missing.
Data size. The dataset is very small; in applications that I'm familiar with, neural networks need much more data to train well. Is CBOW a viable model for this problem, or do I need to use pre-trained embeddings? To be clear, I know that in practice one would use pre-trained, but I want to do as much as I can from scratch for my own learning.

I can't find any accessible literature online about best practices for either of these. In particular, I have no idea how much data is usually needed for training (since the neural network is so shallow, I suspect that the usual rules don't apply), or how much training is necessary.
Any help would be very much appreciated!


texts:
I found textacy Python library, built on top of Spacy, a useful tool to experiment with direct quotation extraction. Unfortunately, textacy that does not support extraction of indirect quotation (according to this issue). So far, I have used rule-based approach that looks for reporting verb and its open clausal complement. However, this is quite custom and home-made solution.
Is there a library, preferably in Python, for such a task?


texts: Use the output of 2 hidden neurons in the last hidden layer of a NN to visualize the result of a 4-class classification task

texts:
I am working on a 4-class classification task using an Artificial Neural Network. My aim is to visualize how well those 4 classes can be separated, moreover how consistent the data coming from each class is and how far they are "away" from one another.

My approach is to use 2 neurons in the last hidden layer and plot the output of the neurons in a scatterplot. However my current output usually looks like this where the different colors stand for the actual classes the data is coming from:

Obviously this is not very useful. First of all I was hoping that the points would not all be all along the same line and second of all I am not sure if this can even be interpreted in the way that I want.

I am using the neuralnet package in R to train the network using linear output. I am guessing that if this approach can work the choice of activation function would be crucial.

I am new to Neural Networks and I am not sure whether or not something like this can be done or not and I am hoping to get some comments.


texts: Using Neural Networks to extract multiple parameters from images

texts:
I want to extract parameters from an image using a neural network.

Example:

Given an image of a brick wall the NN should extract the width and height of the bricks, the color and the roughness.

I can generate images for given parameters to train the NN and want to use it to extract the parameters from an actual image.

I've looked into CNNs. Can I perform this task with them? Do I need special learning algorithms to extract multiple parameters instead of classification? Are there any NNs that are designed for such tasks?


texts: Stochastic gradient descent and its variations

texts:
As I understood, SGDW and SGD + momentum is two different optimizer techniques and SGDWR is SGDW + scheduler in a form of cosine annealing with warm restart. Am I right? If not, please correct me.
So, another question would be, can SGDW and SGD + momentum be merged together or it will be pointless in terms of results?
And which one out of three show better results (even if it is individual for each model, but asking in average) in terms of better generalization, peak accuracy and times it get to this peak?
Thanks in advance!


texts: How should I sample my validation set if I randomly sample training data?

texts:
I have:
training dataset of size 150k.
validation dataset of size 19k.
At each epoch I randomly sample without replacement 10k datapoints for training because I get Out of Mem Errors.
I need to downsample my validation set too. Which of the following methods seem most appropriate:

Randomly sampling validation set which is x% of 10k and use the same set across every epoch.
Randomly sampling validation set which is x% of 10k at every epoch.


texts: How to handle missing data in a logistic regression?

texts:
I am building a model to solve a binary classification task. So far, the input is low dimensional (10 dimensions at most). I need to face the occurrences of missing input. It is my first time at trying to handle that. I am thinking of doing some feature engineering where I add a binary variable for each dimension taking values 1 if missing and 0 if not.
This is a data based solution. Are there canonical ways or known better of handling this in the literature? What are the limitations or weaknesses of my train of thoughts?


texts: I got the following error : 'DataFrame' object has no attribute 'data' can you help please

texts:
I do have the following error: AttributeError: 'DataFrame' object has no attribute 'feature_names'

appreciate your input

from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn import tree
from sklearn.datasets import load_wine
from IPython.display import SVG
from graphviz import Source
from IPython.display import display
from ipywidgets import interactive
import pandas as pd

df1 = pd.read_csv("/Users/dean/Desktop/AI TECNOMATIX EXCEL/cartest1.csv")
#print(df1)
inputs = df1.drop('StatNumOut',axis='columns')
target = df1['StatNumOut']
from sklearn.preprocessing import LabelEncoder
le_company = LabelEncoder()
le_job = LabelEncoder()
le_degree = LabelEncoder()
inputs['name_n'] = le_company.fit_transform(inputs['name'])
inputs['MachineAvailability_n'] = le_company.fit_transform(inputs['MachineAvailability'])
inputs['MachineExitLocked_n'] = le_job.fit_transform(
    inputs['MachineExitLocked'])
inputs['MachineEntranceLocked_n'] = le_degree.fit_transform(
    inputs['MachineEntranceLocked'])
inputs['MachineMTTR_n'] = le_degree.fit_transform(inputs['MachineMTTR'])
inputs['StatNumIn_n'] = le_degree.fit_transform(inputs['StatNumIn'])
inputs_n = inputs.drop(['name',
                        'MachineAvailability',
                        'MachineEntranceLocked',
                        'MachineExitLocked',
                        'MachineMTTR',
                        'StatNumIn'],
                       axis='columns')
inputs_n
target

model.score(inputs_n,target)

labels = df1.feature_names
def plot_tree(crit, split, depth, min_split, min_leaf=0.2):
    estimator = DecisionTreeClassifier(random_state = 0
       , criterion = crit
       , splitter = split
       , max_depth = depth
       , min_samples_split=min_split
       , min_samples_leaf=min_leaf)
    estimator.fit(inputs_n, target)
    graph = Source(tree.export_graphviz(estimator
       , out_file=None
       , feature_names=labels
       , class_names=['0', '1', '2']
       , filled = True))

    display(SVG(graph.pipe(format='svg')))
    return estimator

inter=interactive(plot_tree,
                  crit = ["gini", "entropy"],
                  split = ["best", "random"],
                  depth=[1,2,3,4],
                  min_split=(0.1,1),
                  min_leaf=(0.1,0.5))
display(inter)
```


texts: Encoding Data and huge loss during ANN training

texts:
I just started to learn on ANN and tried to experiment on my own on a Linear Regression. I got a dataset which had housing prices for a city. Tried going through this but my model gives me a huge loss. Am stuck at two issues and how do I resolve them?
Issue 1 - The column location has Location data. How do I go about Normalizing it? I tried using OneHotEncoder but get a huge array with many indices.
Issue 2 - How do I resolve my loss? Do I need to increase the number of hidden layers or type of activation function?
The code can be found here. The dataset is also visible in the gist.


texts: How to generate custom image dataset for object-detection?

texts:
I have to build a custom logo detector from e-commerce images.
I am aware of object detection techniques like YOLO, SSD etc and could find many resources on how to annotate a custom object detection dataset and train a object detector, for example https://medium.com/swlh/train-your-custom-object-detector-with-tensorflow-object-detector-api-65d38dcdf08c. However, in each of such articles, the images already had the concerned object and the process started from annotation step.
However in my case, I have a logo image and unlimited images without logo on which I would like to detect the logo to catch copyright infringement.
My questions are:

Which techniques can I use to place the logo automatically on the images to create a robust training dataset for my logo detector?

What pre-processing techniques should I use before annotating them using some tool?

Will it make any difference if I randomize the location of the logo on the training images versus if I put logos at the top left corner of each image for training data preparation?

For Single Shot Detectron (SSD), what other kinds of transformations are redundant in the training data? Like resizing the logo, changing its aspect ratio, etc. Are these useful variations or they won't make any difference?


texts: Math of Logistic regression cost function

texts:
In the current scikit-learn documentation for binary Logistic regression there is the minimization of the following cost function:
$$\min_{w, c} \frac{1}{2}w^T w + C \sum_{i=1}^n \log(\exp(- y_i (X_i^T w + c)) + 1)$$
Questions:

what is the $c$ term? It is not explained in the documentation
What is the cost function minimized when LogisticRegression(multiclass=multinomial) is used instead?


texts: Adjusting Variables in Multiple Linear Regression

texts:
Suppose I have $10$ exposure variables $a_{1}...a_{10}$ and one dependent outcome $y$.I suspected $a_{9}$  and $a_{10}$ as a possible confounding variables. So at first I performed the multiple linear regression with the $a_{1}...a_{8}$. And got $3$ exposure variables significant$(p<0.05)$. However, again after performing the MLR with all the variables including $a_{9}$ and $a_{10}$. And this time no variables are significant. So my ques is what can I conclude from this result and is the adjustment correct? my best guess is

Those 3 variables have showed relationship with $y$ at first but after adjusting no direct relationship with any exposure variables.
Those 3 variables have showed most important while predicting $y$. However, no variables found more significantly important after adjustment. All are acting same to predict the outcome.
Or any other interpretation..... The real results are below:

For 1st regression Model the Outcome is:

For 2nd regression Model the Outcome is:


texts: Learning AI, Machine Learning, Deep Learning

texts: I am a recent college graduate with a bachelors in computer engineering. I am very interested in deep learning however someone had mentioned to me that I should learn machine learning before I learn deep learning. Is that true ? And if so which online course would you recommend I start at ?


texts: I am doing a visualization project on convolutional neural nets to aid learning and need a simple to display but complex enough pretrained CNN model so I can visualize feature maps for each layer. I ideally would like 5 conv layers but not sure where to find the best model? I am aware of AlexNet but that is it.. thanks!


texts: Fitting Lasso model repeatedly on same train dataset

texts: I am using lasso for feature selection. I have selected lasso parameters from grid search. Now, I used the the best model which I got to fit on training data set. Thus, out of 70 variables I got 66. I observed if I fit lasso again on the same training data set, the variables get dropped again (by drop I mean their coefficients are zero). My doubt is should I go ahead with first iteration where I got 66 variables or should I fit again and again when there is no further variable drop?


texts: how should I measure performance if there is no test data?

texts:
I have 'practice' data set which I can split into training, validation, and test set and I will play with data to make a machine learning model. But in real situation, I will be given a very small data set which I will split into training and validation sets. There is no enough observations to make a separate test set. Then how can I estimate the 'fair' performance of the model on the real data?

The only idea I have is to find a relation between performance on the validation set and test set on the practice data (a regression line etc) and then apply the formula to the performance on the validation set of the real data to find that on the test set of the real data (which actually does not exist.)

Is there any other good idea?


texts:
I am trying to figure out the appropriate way to build a pipeline to train a model which includes using the SMOTENC algorithm:

Given that the N-Nearest Neighbors algorithm and Euclidian distance are used, should the data by normalized (Scale input vectors individually to unit norm). Prior to applying SMOTENC in the pipeline?

Can the algorithm handle missing values? If data imputation and outlier removal based on median and percentiles values are performed prior to SMOTENC rather than after it, wouldn’t this bias the imputation/percentiles?

Can SMOTENC be applied after one-hot encoding and defining the numerical binary columns as categorical features?

When the pipeline is included in a cross validation schema, will the data balancing only be applied to the imbalanced training fold or also for the test fold?

Here is how my pipeline currently looks like:
from imblearn.pipeline import Pipeline as Pipeline_imb
from imblearn.over_sampling import SMOTENC

categorical_features_bool = [True, True, ……. False, False]
smt = SMOTENC(categorical_features =categorical_features_bool,
                random_state=RANDOM_STATE_GRID,
                k_neighbors=10
                ,n_jobs=-1
                     )

preprocess_pipeline = ColumnTransformer(
        transformers=[
            ('Winsorize', FunctionTransformer(winsorize, validate=False,
                                              kw_args={'limits':[0, 0.02],'inplace':False,'axis':0}),
             ['feat_1,'Feat_2']),

            ('num_impute', SimpleImputer(strategy='median', add_indicator=True) ,
             ['feat_10,'Feat_15']),
        ], remainder='passthrough', #passthough features not listed
        n_jobs=-1,
        verbose = False
    )

Model = LogisticRegression()

model_pipeline = Pipeline_imb([
            ('preprocessing', preprocess_pipeline),
            ('smt', smt),
            ('Std', StandardScaler()),
            ('classifier', Model)
            ])


texts:
I am working on image classification project. I have trained model on car dataset. So it gives good accuracy but when I predict BMW car with my model it gives below results.
BMW :- 98%
Audi :- 91%
Why Audi labels shows 91% it should be low. How to solve this issue? I am using resnet 50 pre trained model.


texts: Creating a user-based recommendation engine?

texts:
I have a data set of some questions and answers that users have completed by choices. I'm trying to build a recommendation engine to find similar users based on their answers on top of this.

So for each user I have a data like this:

user_1, {"question_id": "choice_id", ...}
user_2, {"question_id": "choice_id", ...}

I've found clustering and similarity algorithms will be helpful and I've chosen Apache Spark's RowMatrix columnSimilarities to be very efficient.

from pyspark.mllib.linalg.distributed import RowMatrix

rows = sc.parallelize([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])

mat = RowMatrix(rows)

mat.columnSimilarities(threshold)

My question is how should I transform my data to get an input like this?
I've found scikit-learn's DictVectorizer may be helpful but the problem is users complete questions randomly so I can't use questions as features.


texts: How to sort list by parameter in python?

texts:
I have a list of employee records. Each tuple of the list represent a person's record, which includes his name, ID, and age.

For example,

emp_records = [('Karim',100, 45), ('Rahim',10, 30),('Salim', 300,60),('Abu',50,35)]

Now, I want to sort emp_records based on ID.


texts:
Hello everyone I'm having a weird problem.
I having problem with one of two models I've been using. Models take an image data as input and outputs joystick and keyboard information.

A simple CNN, implemented using Keras to use with MNIST data, in which I changed the input size;
The inception_v3 (which is way more complex than the first one).

While training the first model, it throws the following error:
> 2018-05-07 23:03:22.080446: W T:\src\github\tensorflow\tensorflow\core\framework\allocator.cc:101] Allocation of 1037238272 exceeds 10% of system memory.

I have tried changing the batch_size parameter to a smaller batch size without any results.


texts: What is the intuition behind using 2 consecutive convolutional filters in a Convolutional Neural Network?

texts:
I understand the purpose of Convolutional filters (or kernels). I visualize them as learnable feature extractors. E.g. Extract vertical edges or horizontal edges, etc.

Could somebody kindly explain to me the intuition behind stacking 2 or more consecutive convolution filters? Why couldn't the 2 filters be merged into 1?
Picture from Andrew Ng's video

Video URL
Here is the link to Andrew Ng's lecture on Machine learning as a reference


texts: k nearest neighbors method, temporal trend in error

texts:
I have this set of data that looks like this

I was asked o build a $k$-nearest neighbors algorithm for it which I just finished building. I have this question in regards to the data that I do not understand: Do you notice any spatial or temporal trends in error?
I am not sure how to proceed in answering that question. Any suggestions would be appreciated.


texts: What happens when we train a linear SVM on non-linearly separable data?

texts: What happens when we train a basic support vector machine (linear kernel and no soft-margin) on non-linearly separable data? The optimisation problem is not feasible, so what does the minimisation algorithm return?


texts: Proper feature selection methods for classifying signal into two category

texts:
I have a confusion to decide which feature selection method that I should employ in my research whose objective is to analyze which features that are significant in representing a certain condition of the human body,  lie on two categories: normal or not.
I used multiple sensors to determine some features and plan to characterize the signal through the feature respecting the patient condition.
I have explored so many articles and blog about the fittest method for doing feature selection in classifying data into two categories. and this one is pretty good to me:
https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/
It said that we can use ANOVA and kendall's rank,
but on another site, it is mentioned that RFE could be employed too in selecting features for classifying data,
another paper also said that we could employ mRmR, genetic algorithm, and Relief,
I am not sure about this, but is it mean I can employ all of those on feature selection without any further restriction according to my research objective?


texts: What is the rationale for discretization of continuous features and when should it be done?

texts:
Continous feature discretization usually leads to lose of information due to the binning process. However most of the Top solutions for Kaggle Titanic are based on discretization(age,fare).

When should continuous features be discretized ? Is there any criteria and pros and cons on accuracy.


texts: Classify phrases as biomedical or non-biomedical

texts: Words like myotonic are biomedical, but words like new appear in regular English texts. I can imagine a few ways to classify words into biomedical and non-biomedical. Has this problem been tackled before? Does anybody know any ready-to-use solutions for this problem?


texts: Clustering algorithm which does not require to tell the number of clusters

texts:
I have a dataframe with 2 columns of numerical values. I want to apply a clustering algorithm to put all the entries into the same group, which have a relatively small distance to the other entries. But which clustering algorithm can I use, although I do not know how many groups will be formed? It would be ideal if there is a parameter to determine the maximum distance allowed. And if there isn't such an algorithm, maybe it would be really helpful to come up with some intuitions, how such an algorithm can be implemented by myself. Thanks a lot!! :)

The data could look like this:

a,b
20,30
19,31
10,10
9,8
12,11
31,11
32,11


texts: What Math is required to learn Policy Gradients (Part of Reinforcement Learning)?

texts:
That seems a lot of Math... So, in order for me to understand it... What topics of math should I learn?
Or to summarize: What is the math prerequisite to learn Policy Gradients?


texts: Probability vs recall for time series classification task

texts: I am working on the time series classification task that focuses on predicting a fault. I framed the problem as a multi-step forecasting problem, where my goal is to predict to the class at y(t+1 hour) = f(X_lagged). My target y is either 1, or 0, and the problem is essentially a binary classification problem. In the papers I have read, recall/precision/f1 metrics are used as a model evaluation metric. However, if I want to state the problem as "What is the probability that the fault will happen in 1 hour?", what metric do I need to use?