texts:
So my current project involves using a neural network to try and predict the probability of a player getting a kill in a first-person shooter.

I've recorded a number of features that should be relevant (positions, proximity to other players, amount of health, ect) and recorded the data for an amount of time (10 seconds) prior to one of the players trying to get a kill. This is to say, the training data is a number of sequences of the state of the game world in discrete time steps at the end of which one of the players will achieve a kill. The data does include some overlap, which should be fine since the outputs are supposed to represent the probability that each player will achieve a kill, not an actual prediction. The system is implemented so that the data is pre-processed into smaller 5 second sequences using a sliding window, which are used to train the network.

For the model I've attempted to use a convolutional neural network for multivariate time series classification (which I believe is the correct classification for this problem?) as detailed in this paper. The only modification I've made is the addition of a couple of fully-connected layers to the beginning of the model (the layers are only applied to one time step in the sequence at a time, like a temporal shared layer) in order to allow the model to learn a representation of the game state, since the convolution layer only convolves through time and a not insignificant chunk of the features are just one-hot encodings of data.

Though this brings me to the main issue. When training this network after a few iterations the network tends to get stuck predicting almost identical probabilities for every sequence, regardless of difference in input. After many epochs the network might change to predict a different label (and occasionally it will predict a second label for a very small subset of the training data, although this always disappears a couple epochs later), but it will still remain very insensitive to changes in input data. This lack of variation in the output, I believe, is related to the low variations in the input data.

The data for each frame can be split into 11 sections: the first is 8 features representing one-hot encodings of the map information, and then the next 10 are just 10 repeats of 67 features representing information about each of the 10 players. Of these 67 features, 36 are also one-hot encodings. Performing data analysis on the remaining 31 features I have found that they, aside from a couple of features, tend to vary by less than 0.1 within each sequence (bearing in mind that the data has been normalised).

I'm fairly new to developing models and have never encountered this problem before, so any insight on how to deal with it would be greatly appreciated.

I will gladly supply additional information on my model or the data if needed.

Thank you of any help you can offer!


texts: How to compare 10000 data frames in Python?

texts:
I have 10000 data frames like this:
Each data frame corresponds to a different card game (and so different numbers in the table) and I want to compare these data frames all together. For example, I want to compare the heat maps of these data frames. Is there anyway to do this in Python? Is there any toolset so that I can compare all of them in one diagram or something like that? Because I want to see a trend in all of these 10000 data frames.


texts:
I have a dataset which contains ~100,000 samples of 50 classes. I have been using SVM with an RBF kernel to train and predict new data. The problem though is the dataset is skewed towards different classes.

For example, Class 1 - 30 (~3% each), Class 31 - 45 (~0.6% each), Class 46 - 50 (~0.2% each)

I see that the model tends to very rarely predict the classes which occur less frequent in the training set, even though the test set has the same class distribution as the training set.

I am aware that there are technique such as 'undersampling' where the majority class is scaled down to the minor class. However, is this applicable here where there are so many different classes? Are there other methods to help handle this case?


texts: Kernelized Probabilistic Matrix Factorization - Implementation?

texts:
I am trying to implement a kernelized probabilistic matrix factorization which is mentioned in this paper. KPMF Paper. I have coded two update functions as two separate methods. I don't know if this is a correct implementation or not. Can somebody help me understand if the implementation is correct? Below are the two functions:

\begin{equation}
\frac{\partial E}{\partial U_{n,d}} = -\frac{1}{\sigma^2}\sum_{m=1}^{M}\delta_{n,m}(R_{n,m}-U_{n,:}V^T_{m,:})V_{d,m} + e^T_nS_UU_{:,d}
\end{equation}

\begin{equation}
\frac{\partial E}{\partial U_{n,d}} = -\frac{1}{\sigma^2}\sum_{n=1}^{N}\delta_{n,m}(R_{n,m}-U_{n,:}V^T_{m,:})U_{d,n} + e^T_mS_VV_{:,d}
\end{equation}

 def run_U(self,U,V,R,en,su):
    Uhat = np.zeros(U.shape)
    for i in range(U.shape[0]):
        en.fill(0)
        for z in range(U.shape[1]):
            fty = 0
            en[0,i] = 1
            gty = (en.dot(su)).dot(U[:,z])
            for k in range(len(V)):
                if R[i,k] != 0:
                    fty += (R[i,k] - (U[i,:].dot((V[k,:]).T)))*(V[k,z].T)
                else:
                    fty = 0
            Uhat[i,z] = gty+fty
    return Uhat

def run_V(self,U,V,R,em,sv):
    Vhat = np.zeros(V.shape)
    for i in range(V.shape[0]):
        em.fill(0)
        for z in range(V.shape[1]):
            fty = 0
            em[0,i] = 1
            gty = (em.dot(sv)).dot(V[:,z])
            for k in range(len(U)):
                if R[k,i] != 0:
                    fty += (R[k,i] - (U[k,:].dot((V[i,:]).T)))*(U[k,z].T)
                else:
                    fty = 0
            Vhat[i,z] = gty + fty
    return Vhat


texts: Loading data directly from BigQuery to Orange

texts:
I would like to ask you if there is possibility in Orange to load to it data directly eg. from BigQuery? I added block "Python script" to the flow and my script looks like that:
import os
import sys
import Orange

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = r'path toapplication_default_credentials.json'

from google.cloud import storage
from google.cloud import bigquery
from google.cloud import secretmanager
bigquery_client = google.cloud.bigquery.Client(project='my project')
secret_client = secretmanager.SecretManagerServiceClient()
query ="""

my query

"""

query_job = bigquery_client.query(query)

out_data = query_job

I was trying to put also Orange.data.Table on query_job but it is not working. How can I load data directly from Python to orange?


texts: Why is this equation converted to matrix form in this way? Is it possible to multiply an inverse matrix with a vector?

texts:
I have been banging my head on wall for days trying to decode this equation. please help me out with this...
Below is the equation (consider $x$ as $\Delta x$, and $y$ as $\Delta y$):
$x = - \eta(Id-\eta^2D^2_{xy}fD^2_{yx}g)^{-1}(\nabla_xf-\eta D^2_{xy}f\nabla_yg)$
$y = - \eta(Id-\eta^2D^2_{yx}gD^2_{xy}f)^{-1}(\nabla_yg-\eta D^2_{yx}g\nabla_xf)$
The authors of the paper have converted it into matrix form as:

I am failing to see how is this possible, I mean how am I supposed to multiply inverse of a matrix with a vector.
Thanks in advance for helping me out, and forgive my poor english. I have been struggling in this equation for days, it is from the paper competitive gradient descent (for someone who might be interested equation number 3 and 4) https://arxiv.org/abs/1905.12103?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%253A+arxiv%252FQSXk+%2528ExcitingAds%2521+cs+updates+on+arXiv.org%2529.


texts: How could I find the number of instances per fold to plot a histogram, in my dataset?

texts:
I am currently trying to learn machine learning with Python,
and I have been given an .npz file containing the data. I have explored the dataset, and after exploration noted that it has 5 datatypes:
cell_data = np.load("C:/Users/alexs/Documents/DataMining/cell-data.npz")
cell_data.files

Giving this at the output:
['images', 'counts', 'folds', 'compressed', 'allow_pickle']

As well as the image attached.
I am promised the dataset itself has 3 folds. The count is an Nx6 matrix with each row
corresponding to a single image patch and each column corresponding to the 6 cell types (called T1,T2, … , T6). The folds seems to be an 1xN matrix, however I am not sure, it consists of values ranging {0,2}.
How would I find out the number of instances per fold, and if this were possible, how would I find out the ranges of the folds, i.e. which instances belong to which folds (or group the instances into their own seperate array that respresents each fold like, fold1 = x, fold2 = x_2 etc.)?
Once again, I apologise for the simple question, but I am quite new to programming and I am trying to learn.
Any help would be appreciated, thank you


texts: Train using the raw pixel images or training using facial embeddings

texts: I want to create a model that can classify faces based on these face shapes(oval, square and heart). When creating my CNN model should I train it using the images as they are or should i generate the facial embeddings of the images then train the classifier(DNN or CNN) using the facial embeddings. Which approach would be the best


texts: Unsupervised Classification for documents

texts:
I'm trying to create a classifier in which there is less "manual" work for the user. For less manual work I mean that there won't be an initial phase of manual labeling of a training set, like in Machine Learning (supervised)

My dataset is composed by instances that are really different by class. They are documents in which there are orders for specified products for different clients. And every client got its template.

For example I got:

[Client A]
Image
Date: xxx  Order:
Products:
Table

[Client B]
Date: xxx
Order
Image
Products:
table
Image

Now I'm doing the classification doing a simple check on every document, for the presence of a specified feature, that's manually identified by a user (by area and using edit distance)

The classes are really different (in some cases), and trying an unsupervised classifier like an agglomerative clustering the classes are split really well. After that, using measures like TF/ICF often the features (in my case I use tokenized and normalized text as features) which got the greater values are the ones that are used in my manual classification.

The criterions that I use for stop the clustering iteration are different ( I got different configuration) like max distance, or max number of clusters.

After that, I thought that when the clusters will be created an user at the end will label each cluster identifying the class by the best TF/ICF (term frequency, inverse cluster frequency) features found in each cluster. And after that the clusters will be used like "classifier". I know that this approach will lead to worse classification, but It's not a problem.

The problem is that when two classes are really similar (I got classes in which the difference is only the customer code, for example) they are really difficult to split.

Any idea on how approach this problem?
And, there is a way in which my algorithm can find out if there is a "new class" in the flux?


texts: Why is there no end user Application, yet?

texts:
machine learning is being hyped since the deep neural networks.

It seems to me, that you have to program in order to do machine learning.
But is the process of training data and labeling data is the same of every problem. Why isn't there an Excel like application that enables thousands of non experts to do machine learning ?

Disclaimer : I am not a data scientist .


texts:
I am working on a project with two data sets. A time vs. speed data set (let's call it traffic), and a time vs. weather data set (called weather).

I am looking to find a correlation between these two sets using Pig. However the traffic data set has the time field, D/M/Y hr:min:sec, and the weather data set has the time field, D/M/Y.

Due to this I would like to average the speed per day and put it into a single D/M/Y value inside the traffic file.

I then plan to use:

data = JOIN speed BY day, JOIN weather BY day with 'merge'

I will then find the correlation using: (I am borrowing this code from elsewhere)

set = LOAD 'data.txt' AS (speed:double, weather:double)
rel = GROUP set ALL
cor = FOREACH rel GENERATE COR(set.speed, set.weather)
dump cor;

This is my first experience with Pig (I've never even used SQL), so I would like to know a few things:

1. How can I merge the rows of my traffic file (ie. average D/M/Y hr:min:sec into D/M/Y)?
2. Is there a better way to find a correlation between the fields of different datasets?
3. Are the JOIN BY and the COR() functions used appropriately in my above code?


texts:
I have a Python Pandas dataframe with a column named "codes". This column is very messy and contains the following values:
codes                       output
1234A, C (1AAA1), D, E      1234A 1234C 1234D 1234E
34240 & 1 & 3               34240 34241 34243
PS1A;D;E                    PS1A PS1D PS1E
CE11A, C & 2452D            CE11A CD11C 2352D
EOPSDCE11A, C & 2452D       EOPSDCE11A EOPSDCD11C 2352D

Basically, what we need is to complete the single letter/number (i.e. C) with the full code excluding the last char (i.e. 1234 in the first line), remove the phrase in the parenthesis, keep the code if it's a full code (length between 2-10), and use space as delimiter.
I have tried to use str.replace but was not able to manage to write the regex correctly.
Thanks for the help in advance!


texts:
I am training a binary neural network so I can obtain its weights, and use them on another network where I add a few things. For example, this this one layer of the network:
model = Sequential()
model.add(Input(shape=(nb_channel, img_rows, img_cols)))
model.add(XnorConv2D(8, kernel_size=(2, 2),strides=(1,1), H=H, padding='valid', use_bias=use_bias, name='conv1'))
model.add(BatchNormalization(epsilon=epsilon, momentum=momentum, axis=1, name='bn1'))
model.add(Activation('relu', name='act1'))
model.add(Flatten())
model.add(XnorDense(nb_classes, H=H, kernel_lr_multiplier=kernel_lr_multiplier, use_bias=use_bias, name='dense1'))

There is no issue with the training phase. I extracted the weights from the model above and recreated the layers, just for inference. I just a little bit unsure how to implement the batch normalization layer. From the keras documentation, batch normalization works differently during inference. I can obviously implement my own normalization:, (x - mean) / variance, but I'd rather use a built-in method. Any suggestions on how I can do this such that the result I get is same as the result I get if I use the model.predict method from keras ?


texts: What stopping metric to chose to optimize 'sensitivity' for a GBM in H2O?

texts:
I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set.

What stopping metric can be used to optimize the sensitivity score on validation set?

I have about 400 observations. The response variable is binary (0/1) and I have 40 predictor variables.

My current set-up uses AUC as stopping metric.

df <- as.h2o(df)
split <- h2o.splitFrame(data=df, ratios=c(0.6, 0.2)) # split 60, 20, 20%
train <- h2o.assign(split[[1]], "train.hex") # 60%
valid <- h2o.assign(split[[2]], "valid.hex") # 20%
test <- h2o.assign(split[[3]], "test.hex") # 20%

x <- setdiff(names(df), "disease")
y <- "disease"

gbm <- h2o.gbm(
  x = x,
  y = y,
  training_frame = train,
  validation_frame = valid,
  ntrees = 10000,
  learn_rate=0.01,
  # Stopping parameters
  stopping_rounds = 5, stopping_tolerance = 1e-4, stopping_metric = "AUC",
  sample_rate = 0.8,
  col_sample_rate = 0.8,
  seed = 1234,
  nfolds = 50,
  score_tree_interval = 10
)
h2o.auc(h2o.performance(gbm, valid = TRUE))


texts: Applying an algorithm on it's own training data for descriptive purposes?

texts:
Good Day, I am newer to data science so I am not confident in this. To set up the question I will describe my data and approach.
Data
I don't want to share specific data examples as I want to try and keep some anonymity.  I have data for events between 2016 and 2019.  Each event has ~14 features (categorical and numerical) as well as a binary label of success or failure.  Each of these is run through different transformers (normalizing and one-hot encoding).
Approach
What I am interested in doing is knowing how likely events are to success (not necessarily predict if they will succeed).
I played around with train/test splits to find an algorithm that worked best. I ran a stratified k-fold cross-validation on the data in a grid search to tune the hyperparameter using logloss as my measurement of choice.  I am implementing this in Python so Scikit-Learn as my tool of choice.  The algorithm I settled on was GradientBoostingClassifier.
Problem
What I am interested in doing now is as I look at events in 2020, I am curious on how likely they are to succeed.  When I look at my 2020 data I have the same 14 features (transform them the same way) and as well I instantly know if they succeeded or failed.  So predicting Success/Failure is not interesting nor what I want to do.  I can easily generate probabilities on this 2020 data, using my trained model, with predict_proba in sklearn.
Question
Now my partner wants me to apply this model on the same training data from 2016-2019.  Initially this feels like a big no-go. You never predict your own training data as it will be heavily biased.  But I am not predicting.  These feels more like a traditional statistics descriptive problem where I look at the nature of known data and see how it behaves.
So again, I am not interesting in knowing/predicting if something WILL succeed or fail (we know this instantaneously).  I am more interested in knowing if it succeeded (or failed) vs how likely it was to do so.  A success with a 5% chance vs succeeding with 40% chance is way more interesting in this problem.
So my question, per the title is, can you apply a trained algorithm / model on its own training data if my interest is not in predicting forward but evaluating success vs likelihood of success (and still yield useful information)?


texts: Nested features with one to many relationships

texts:
I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level.

For example, consider predicting machine failure based on its last repair report.

A machine has many (many) different parts.

Every part could have been damaged or not.

If damaged, it could have been inspected, repaired or replaced etc.

These operations have associated costs and the part itself has a money value and other attributes (like the time it took to repair it...).

If I want to make predictions on the level of a machine, I need to aggregate all this information. I could one-hot encode all the parts indicating whether they have been damaged or not. But this still leaves me with the other attributes on the lower part level, like price and performed operation. I could probably further expand them into individual columns by considering all the combinations like part_X-repaired-..-price, part_X-replaced-..-price, ... part_Z-replaced-..-price but this seems to get out of hand.

Is there a better way to handle this type of data? I was thinking maybe some clustering technique but when I try to set it up I run into the same problem.

It is somewhat a reversed hierarchical model structure (in hierarchical (linear) models the outcome variable is on the lowest level if I'm not mistaking).


texts:
I looking at football data and trying to predict whether a goal will occur using xgboost with objective binary: logistic.
My data is 1:10 unbalanced with no goals being more dominant. I have used smote or task.over in mlr package to oversample (with a factor of 4).
I train the model, tune and cross validate but the predictions seems reasonable (low aucpr of 30% but high in other stats).
However when I look at the probabilities predicted by the model, it  much larger than the actual average. Is there anything that could cause this?
What are the probabilities? i.e class 1 probs are the probabilities that they are in class 1, so may not truly represent the probability to score a goal.
Thanks in advance


texts: I understand that by filtering out the instances with labels that Random Forest trees are uncertain upon with their decisions, and model these with another classifier could give a better overall result. My question is, how can I "combine" two (or more) classifiers' classification on a single unlabeled dataset?


texts:
I have a dataset of choices (between A,B and C) done by certain users, and I want to train a neural network to predict the choices. I divide in train and test sets.
An instance is formed by: [UserId, some features, choice]. Choice is the target variable.
Same user can have multiple instances (with different features and choice).
My assumption is that the same user makes often the same choice, so I want a feature describing the "frequency" of a choice (e.g. User 1 choose A 60% of times).
From which dataset can I compute this frequency?

I cannot use the training set, otherwise is overfitting (with a frequency=0, the model is sure it was never chosen in the training-set).
I do not have an additional dataset of choices in a different period (this would be ideal).
Can I compute the frequency from the test-set? In that way, I'm actually training a model (on training set), based on the frequencies in a different period (the test set), which is what I want. Does it overfit the performances in the test-set? I assume not, since the model is trained on the training set. My results confirm similar performances in the two sets.


texts: When I add regularization like L1,L2 , do I need more epochs to properly train my model?

texts:
When I add regularization techniques in my model like L1 or L2 do i need more epochs to properly converge my model.
for r in (None,"L1","L2"):
        for max_iter in (30,45,60):
            classifier=SGDClassifier(loss="log",penalty=r,max_iter=max_iter,learning_rate="constant",eta0=0.01,random_state=42)
            print("max_iter={}".format(max_iter))
            classifier.fit(X_train,Y_train)
            acc=classifier.score(X_test,Y_test)
            print("accuracy when r={} is {}".format(r,acc*100))

When r = None:

max_iter = 30/45 it says ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.
max_iter = 60 no warning.

When r = L1:

max_iter= 30 same warning.
max_iter = 45/60 no warning.

When r= L2:

max_iter = 30/45/60 same warning

Does it matter or this is random?


texts: Why the estimated Lasso coefficients of almost all variables are equal to zero?

texts:
I would greatly appreciate if you could guide me. In fact, I used "Bayesian Optimization " to tune hyper-parameters  of Lasso but the estimated Lasso coefficients of almost all variables are equal to zero.

 X = np.array(pd.read_csv('my_X_table1-1c.csv',header=None).values)
 y = np.array(pd.read_csv('my_y_table1-1c.csv',header=None).values.ravel())

 ln=X.shape

names = ["x%s" % i for i in range(1,ln[1]+1)]

# split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

def Lassocv(alpha):
    return cross_val_score(Lasso(alpha=float(alpha), random_state=42),
                           X_train, y_train, scoring='neg_mean_absolute_error', cv=5).mean()

if __name__ == "__main__":

    LassoBO = BayesianOptimization(Lassocv, {'alpha': (0, 8)})
    LassoBO.maximize(init_points=2, n_iter = 30)
    print('Final Results')
    print('Lasso: %f' % LassoBO.res['max']['max_val'])
    alpha=LassoBO.res['max']['max_params']['alpha']

lasso = Lasso(alpha=alpha)
lasso.fit(X_train, y_train)
print('train Score:', lasso.score(X_train, y_train))
print('test Score:', lasso.score(X_test, y_test))
print('train MSE:', mean_squared_error(y_train, lasso.predict(X_train)))
print('test MSE:', mean_squared_error(y_test, lasso.predict(X_test)))
print('train MAE:', mean_absolute_error(y_train, lasso.predict(X_train)))
print('test MAE:', mean_absolute_error(y_test, lasso.predict(X_test)))

print("Features sorted by their score:")

# A helper method for pretty-printing linear models
def pretty_print_linear(coefs, names=None, sort=False):
    if names == None:
        names = ["X%s" % x for x in range(len(coefs))]
    lst = zip(coefs, names)
    if sort:
        lst = sorted(lst, key=lambda x: -np.abs(x[0]))
    return " + ".join("%s * %s" % (round(coef, 3), name)
                      for coef, name in lst)

print("Lasso model:", pretty_print_linear(lasso.coef_,names=names,sort=True))

output:

Final Results
Lasso: -0.786422
train Score: 0.476819501615
test Score: 0.459836314561
train MSE: 1.10511096023
test MSE: 1.02388727356
train MAE: 0.681164633717
test MAE: 0.663119930613
Features sorted by their score:
Lasso model: 0.001 * x25 + 0.0 * x54 + 0.0 * x48 + -0.0 * x17 + 0.0 * x31 + 0.0 * x12 + -0.0 * x15 + -0.0 * x34 + 0.0 * x1 + 0.0 * x2 + -0.0 * x3 + -0.0 * x4 + 0.0 * x5 + 0.0 * x6 + 0.0 * x7 + 0.0 * x8 + 0.0 * x9 + 0.0 * x10 + 0.0 * x11 + 0.0 * x13 + -0.0 * x14 + 0.0 * x16 + 0.0 * x18 + -0.0 * x19 + 0.0 * x20 + -0.0 * x21 + 0.0 * x22 + 0.0 * x23 + -0.0 * x24 + -0.0 * x26 + 0.0 * x27 + -0.0 * x28 + -0.0 * x29 + 0.0 * x30 + 0.0 * x32 + 0.0 * x33 + 0.0 * x35 + 0.0 * x36 + -0.0 * x37 + 0.0 * x38 + 0.0 * x39 + -0.0 * x40 + -0.0 * x41 + -0.0 * x42 + 0.0 * x43 + 0.0 * x44 + 0.0 * x45 + 0.0 * x46 + -0.0 * x47 + -0.0 * x49 + -0.0 * x50 + -0.0 * x51 + 0.0 * x52 + 0.0 * x53 + 0.0 * x55 + 0.0 * x56 + 0.0 * x57 + 0.0 * x58 + 0.0 * x59 + 0.0 * x60


texts: Classification with heavily overlapping data points of different labels

texts: I am trying to build a classifier that will classify data points of different classes, but which significantly overlap, but I'm not sure how to best approach the problem. I have read that tree models can perform well in this situation, and that Support Vector Machines might as well, but I'm unclear what other approaches might be good for tackling this problem. Can anyone give me some direction?


texts:
I am new with Machine Learning and I started with some lessons in Kaggle. There, I learnt how to use DecisionTreeRegressor() and RandomForestRegressor() from sklearn.

However, I cannot really understand how I can verify that my explanatory variables do not overfit the model. For example, the lessons included evaluation with the use of Mean Absolute Error. MAE and MRSE can evaluate whether my Decision Tree depth is optimal or not, but not if my explanatory data are even relevant.

I come from Economics, so I am used to deal with such problems using diagnostics or $R^2$. Is there any equivalent to $R^2$ benchmark to determine whether my explanatory variables are overfitting my model or not?


texts: Is pattern recognition the same as unsupervised learning? Is machine learning the same as supervised learning?

texts:
Firstly, here is the definition of a well-posed learning problem:

  A computer program is said to learn from experience E with
  respect to some task T and some performance measure P, if its
  performance on T, as measured by P, improves with experience
  E.

My machine learning professor claims that supervised learning is machine learning and unsupervised learning is pattern recognition. Here are his definitions of machine learning and pattern recognition:

  Machine Learning is learning from experience. It’s also called
  supervised learning. E consists of features and labels, and P
  and T are well-defined.

And

  Pattern Recognition is finding patterns without experience. It’s
  also called unsupervised learning. E consists of only features,
  and P and T are defined in much broader terms of finding
  ‘interesting patterns'.

If I cross reference this with other sources on the internet, I get different definitions. They say that machine learning involves both supervised and unsupervised learning. Likewise, pattern recognition involves both supervised and unsupervised learning. Some say that pattern recognition is machine learning. Some even say the opposite, that machine learning is a form of pattern recognition.

So is my professor right, or is the internet right, or is the answer somewhere in between? Can we answer this question definitively?


texts: Pandas Groupby makes kernel die in Jupyter notebook/Python

texts:
I have a groupby in jupyter-notebook that takes ages to run and after 10 minutes of running it says 'kernel died...',

The groupby looks like this:

df1.groupby(['date', 'unit', 'company', 'city'])['col1',
'col2',
'col3',
'col4',
  ...
'col20'].mean()

All of the 'col' columns are float values. I am running everything locally. Any ideas?

UPDATE:

The shape of df1 is:

(1360, 24)

Memory and dtypes:

dtypes: category(3), datetime64[ns](2), float64(17), int64(2)
memory usage: 266.9 KB

The unique size of city, date, company, unit:

len(df1.date.unique()) = 789
len(df1.unit.unique()) = 76
len(df1.company.unique()) = 205
len(df1.city.unique()) = 237

I have 16GB of memory on MacBook Pro.

UPDATE 2:

It works only if I have date and unit inside the groupby columns as the only 2 columns. If I add either a company or city, it doesn't work anymore, it keeps running indefinitely.


texts: What to do about non-responses to demographic survey questions?

texts:
Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity,  gender, or other demographic data?

In studies (and datasets) examining diversity in various communities, I see a wide range of strategies. Some simply remove records where a person declines to specify a race/ethnicity or gender. Some include them and create an "unknown" category.  If looking at race/ethnicity, for example, some studies just state they assume all non-responders are white.

None of these solutions seem perfect. Are there any others? And if you remove responses from people who, to take one example, decline to specify a gender, aren't you potentially just erasing part of the population that does not identify as male or female?

A number of surveys now allow participants the option to "decline to state" a gender or "decline to state" a race/ethnicity or some similar option, or to just not answer the question at all, is there any developing consensus on what to do after that data is collected?  Does it depend on the population being studied?


texts: Wrong output multiple linear regression statsmodels

texts:
I recently moved to python for data analysis and apparently I am stuck on the basics. I am trying to regress the parameters of the following expression: z=20+x+3*y+noise, and I get the right intercept but the x and y parameters are clearly wrong. What am I doing missing? Code below:

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

# generate true values, and noise around them
np.random.seed(5)
x = np.arange(1, 101)
y = np.arange(1, 101)
z = 20 + x + 3* y + np.random.normal(0, 20, 100)

data = pd.DataFrame({'x':x, 'y':y, 'z': z})

lm = smf.ols(formula='z ~ x + y', data=data).fit()

# print the coefficients
lm.summary()

returns

where the x and y parameters are both 1.5, instead of being 1 and 3. What's wrong?


texts: Should I create a separate column for each Id value in a feature column or can I use the feature column as it is?

texts:
I am working on developing a model for predicting, revenue that a movie will make. One of the features in the training set contains id of the series that a movie belongs to.Say, Star Wars series has Id 1, then the corresponding value in that column for "Return Of Jedi" movie will be 1.

Should I create a separate column for each of the series and add value 1 if the film belongs to that series, or will just keeping the existing column containing ids will be fine?


texts: Online learning w/ feature weighting/adjusting

texts: Let's say I have a supervised learning problem with a sequence of features and labels. First, I learn on the training data and then I decide to stream in data, point by point and do online learning. Is it possible to update the weights or figure out the feature importances as each data point comes in? Also, what online learning algorithms would allow me to do this and can this be done in Python?


texts: Diminishing returns in language identification data set size?

texts:
Most problems have a curve whereby the results improve as data are added but level off at some point.

Are there research papers or industry results that discuss the correlation between data set size and prediction accuracy for natural language identification?


texts: Alterable Similarity Join for Time Series

texts:
I'm currently trying to implement the MPdist (matrix profile distance) algorithm for time-series data, but I've developed a new distance metric that I'd like to use in place of the Euclidean metric. I've implemented an algorithm that techincally works, however I used brute force to compute the distance between every subsequence of every time series that I'm using.
As such, it's horribly inefficient. I want to use a similiarity join algorithm to get complexity down. Still, they're really not my area of expertise and most algorithms I've seen are not very transparent and make critical use of Euclidean distance for calculating the 1NN similarity join.
Could someone point me to an article which is perhaps a little more transparent and is such that I could more easily replace the Euclidean metric with another? Or a github repo or something. I only need 1NN


texts: Learning rate in the Perceptron Proof and Convergence

texts:
Every perceptron convergence proof i've looked at implicitly uses a learning rate = 1.

However, the book I'm using ("Machine learning with Python") suggests to use a small learning rate for convergence reason, without giving a proof.

Can someone explain how the learning rate influences the perceptron convergence and what value of learning rate should be used in practice?


texts: SMOTE vs SMOTE-NC for binary classifier with categorical and numeric data

texts:
I am using Xgboost for classification. My y is 0 or 1 (true or false). I have categorical and numeric features, so theoretically, I need to use SMOTE-NC instead of SMOTE. However, I get better results with SMOTE.
Could anyone explain why this is happening?
Also, if I use some encoder (BinaryEncoder, one hot, etc.) for categorical data, do I need to use SMOTE-NC after encoding, or before?
I copied my example code (x and y is after cleaning, include BinaryEncoder).
_train, X_val, y_train, y_val = train_test_split(x, y, test_size=0.2, random_state=1)

smt = SMOTE()
X_resampled, y_resampled = smt.fit_resample(X_train, y_train)

params_model1 = {
    'booster': ['dart', 'gbtree', 'gblinear'],
    'learning_rate': [0.001, 0.01, 0.05, 0.1],
    'min_child_weight': [1, 5, 10, 15, 20],
    'gamma': [0, 0.5, 1, 1.5, 2, 5],
    'subsample': [0.6, 0.8, 1.0],
    'colsample_bytree': [0.6, 0.8, 1.0],
    'max_depth': [3, 4, 5, 6, 7, 8],
    'max_delta_step': [0, 1, 2, 3, 5, 10],
    'base_score': [0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65],
    'reg_alpha': [0, 0.5, 1, 1.5, 2],
    'reg_lambda': [0, 0.5, 1, 1.5, 2],
    'n_estimators': [100, 200, 500]
}

skf = StratifiedKFold(n_splits=3, shuffle=True, random_state=1001)

xgb = XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
                    colsample_bynode=1, colsample_bytree=0.3, gamma=1,
                    learning_rate=0.1, max_delta_step=0, max_depth=10,
                    min_child_weight=5, missing=None, n_estimators=1000, n_jobs=1,
                    nthread=None, objective='binary:logistic', random_state=0,
                    reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
                    silent=None, subsample=0.8, verbosity=1)

scoring = 'f1'
rs_xgb = RandomizedSearchCV(xgb, param_distributions=params_model1, n_iter=1,
                            scoring=scoring,  n_jobs=4, cv=skf.split(X_resampled, y_resampled), verbose=3,
                            random_state=1001)

rs_xgb.fit(X_resampled, y_resampled)

refit = rs_xgb.best_estimator_

joblib.dump(refit, 'validator1.pkl')

loaded_xgb = joblib.load('validator1.pkl')

y_predict = loaded_xgb.predict(X_val.as_matrix())

print(confusion_matrix(y_val, y_predict))

print("Final result " + str(f1_score(y_val, y_predict)))


texts: How to choose the optimal k in k-protoypes?

texts:
To analyze a dataset from banking I have both numerical and categorical values. I transform them to analyze with k-prototypes.

The original dataset:

The modified dataset:

E.g.: Job (for 1 to 12 'cos there are 12 levels)

Should I scale the dataset before doing the k-prototypes?

How could I determine the optimal "k" to choose (coding)?

I thought to execute:

library(clustMixType)

lbd <- lambdaest(BPor)

kpres <- kproto(BPor, 5, lambda = lbd) #Change '5' for every possible value of k.

print(kpres)

And then, calculate the sum of within cluster error (choosing the little one).


texts: Is there standard parameters that characterize how fast neural network learns to achieve specific error rate?

texts: I want to estimate how fast my model learn. Mb how many learn steps my model need to do to achieve specific error rate(precision/ recall/ MSE/ etc.). Any standard approach for measuring this?


texts: Is cross-entropy a good cost function if I'm interested in the probabilities of a sample belonging to a certain class?

texts:
I'm training a neural network that, for each of six classes, tries to predict the probability that a sample belongs to it. After that, I want to use these probabilities as fractions of the sample belonging to that class. My network gives softmax output and is trained using cross-entropy costs (in fact linear output which is then transformed to softmax by tf.nn.softmax_cross_entropy_with_logits)

Is this the right cost function to use when I want to train the network in getting all 6 probabilities correct instead of just classifying each sample as one of the 6 classes? I started hesitating because the tensorflow documentation says about tf.nn.softmax_cross_entropy_with_logits:

  Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).

UPDATE
Even though it sound strange, the cross-entropy cost function seems to work best here. In my understanding this is because the cross-entropy for discrete probability distributions (wikipedia) is

  $H(p, q) = -\sum_x p(x)\, \log q(x).$

This function is minimal when $p(x) = q(x)$ for all $x$. This explains why minimizing the cross-entropy error forces the output distribution $q$ to the target distribution $p$, even though I'm abusing tf.nn.softmax_cross_entropy_with_logits by instead of feeding it labels I feed it a discrete probability distribution.

Why it works better than MSE probably is because the magnitudes of the 6 class probabilities are very different. Class 1 may have a probability of 80% while class 2 has a probability of 0.5%, so MSE pays more attention to getting the 80% class right.

Does this mean that CE is the best option still, or is there a way to scale the outputs or weigh them such that the network pays attention to getting each class correct?


texts: Create new column in Orange using Python Scripting Widget

texts:
This is a fairly basic question. How do you create a new column in Orange - based upon an existing column?

For example:

NewCol = 2 * ExistingColumn

or

NewCol = len(ExistingColumn)

I would like to use the Python Script Widget - not the feature widget.


texts: Is it wrong to use Glorot Initialization with ReLu Activation?

texts:
I'm reading that keras' default initialization is glorot_uniform.

However, all of the tutorials I see are using relu activation as the go-to for hidden layers, yet I do not see them specifying initialization for those layers as he.

Would it be better for these relu layers to use he instead of glorot?

As seen in OReilly's Hands-On Machine Learning with Scikit-Learn & Tensorflow:

| initialization | activation                    |
+----------------+-------------------------------+
| glorot         | none, tanh, logistic, softmax |
| he             | relu & variants               |
| lecun          | selu                          |


texts: What are possible approaches to deal with unseparable data even after PCA?

texts:
Greetings data scientists,

I am dealing with a complex classification/prediction problem and I am finding very difficult to separate the classes at all.
Even after PCA, my data (over the two PCs), look like this:

Given this dataset, what approach could be best to accurately predict the classes?

Background: I am trying to classifying modelled problems via the best approach/algorithm to solve them.

Many thanks,


texts: Visual/intuitive comparisons between various pairwise loss functions for learning-to-rank tasks

texts: In learning-to-rank, There are various pairwise loss functions such as Bayesian Personalised Ranking (BPR) or Weighted Approximate-Rank Pairwise (WARP), etc. I know that these aim at different optimization purposes in ranking, but it will be very useful if somebody can share a visual/clear comparison of them. For example, how the results will be different if we apply the loss functions on the same ranking task.


texts: Not sure how to manage 1 hot encoded data in RNN

texts:
Context
I am trying to build a plain RNN using Tensorflow for letter-level language detection. I realize that my approach isn't the ideal one, but I'm trying to learn by doing the more basic version first.
I am training the RNN on a sequence of letters, and for now I am using 1 hot encodings - each vector corresponding to a letter.
What I've done so far:
Here are the various placeholders and variables defined in the graph. I'm putting all of them here for reference (Note that truncated_backprop_length is just the length of the letter sequence that I train with.):
batchX_placeholder = tf.placeholder(tf.float32,
    [batch_size, truncated_backprop_length, encoding_size])
batchY_placeholder = tf.placeholder(tf.int32,
    [batch_size, num_classes])
init_state = tf.placeholder(tf.float32, [batch_size * encoding_size, state_size])

W = tf.Variable(np.random.rand(state_size + 1, state_size), dtype=tf.float32)
b = tf.Variable(np.zeros((1, state_size)), dtype=tf.float32)

W2 = tf.Variable(np.random.rand(batch_size, encoding_size * batch_size), dtype=tf.float32)
b2 = tf.Variable(np.zeros((1, state_size)), dtype=tf.float32)

W3 = tf.Variable(np.random.rand(state_size, num_classes), dtype=tf.float32)
b3 = tf.Variable(np.zeros((1, num_classes)), dtype=tf.float32)

I unstack the training samples in batchX_placeholder so I can iterate through them. The loop that iterates through each training example of shape [batch_size, encoding_size] looks like this:
for current_input in inputs_series:
    current_input = tf.reshape(current_input, [batch_size * encoding_size, 1])

    input_and_state_concatenated = tf.concat([current_input, current_state], 1)

    next_state = tf.tanh(tf.matmul(input_and_state_concatenated, W) + b)

    current_state = next_state

I take the final state from this loop and do the following:
logits = tf.matmul(tf.matmul(W2, final_state) + b2, W3) + b3

loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits,
                                               labels=batchY_placeholder)

Results
I'm assuming that I'm running my training loop correctly and will leave that out of the question (for now) for simplicity. With the above setup, the loss does not converge during training.
Pieces I'm questioning

I have reshaped current_input so it can be stacked against the state to get input_and_state_concatenated. So the i^{th} group of elements (there are encoding_size of them) in current_input now correspond to the i^{th} batch.

I was not sure how to get the final state, of shape [encoding_size * batch_size, state_size] down to [batch_size, num_classes], so I kindof made something up. See the line starting with logits = .....

Now that I think about it, these two pieces are inverses of one another - the encoding itself seemed weird, and it led to weirdness in decoding it as well.
Question
Where does my approach to passing a sequence of 1 hot encoded vectors diverge from what is typically done?


texts: Using standard deviation as a metric for model selection

texts:
I'm really getting stuck with overfitting and I'm trying all I can to reduce it.

I want't to write a metric to help score models in a cv loop. I'm using 10x5 folds and still getting out of sample accuracy scores +/- 5-8% from the mean of my CV score. (i.e when i upload to kaggle)

Does the validation score's standard deviation have a correlation with overfitting / error and can this be used in my scoring? currently the stdev is around 5% that seems ALOT.

I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?

Does any one have any experience with this?

Any help welcome.


texts: When inputting image rgb values to MLP, should I divide by 255?

texts:
I have an MLP with 3072 input nodes which are for 1024 rgb pixels. My datasets is in an array with each row representing one image and looking like this:

[red_pix1, red_pix2, ..., red_pix1024, green_pix1, green_pix2, ..., green_pix1024, blue_pix1, blue_pix2, ..., blue_pix1024]

Each array value is an integer between 0 and 255.

My question is, before training the network, should I "normalize" my dataset by dividing each element by 255? That way, each input element would have values between 0 and 1. Is this better than having values between 0 and 255?


texts: (Not a programming question) Is there a common word to indicate predict or forecast

texts:
The general understanding is:

Predict: past value - could be data the model has seen (data from train set) or past data model has not seen (data from test set) (together in-sample data)
Forecast: future value - data which neither model or user has seen (out-of-sample data)

Related question: https://stats.stackexchange.com/questions/65287/difference-between-forecast-and-prediction
Question: Is there a common word that can mean either? This will help in talking or writing or explaining to non-technical people.


texts: How prevalent is `C/C++` in machine learning development?

texts:
I am currently a data scientist mostly doing NLP, and I do most of my work inPython. Since I didn't get a CS degree in undergrad, I've been limited to very high level languages; Java, Python, and R. I somehow even took Data Structures and Algorithms avoiding C or C++.

I'm intending to go to graduate school to study more Natural Language Processing, and I'm wondering how much C/C++ I need to know. Deep-learning frameworks like PyTorch or Tensorflow are written in C++, and CUDA is only available in C. I'm not going to be writing Cython libraries, but I would like to do research and build new models (i.e. like "inventing" CNN's, seq2seq models, transformers).

I don't know how much C/C++ is used, and I'm unsure if it's worth learning the language-specific complexities that may be channeled into learning something else; hopefully somebody can let me know how prevalent the use of C/C++ is?


texts:
Is there any methodology called Recursive Transfer Learning? For example, let's consider a situation that we have a lack of data while training a convolution neural network (CNN) for object detection from scratch. Let's assume hypothetically the CNN is to be trained for recognizing (detecting) the dogs of the beagle-bone breed. And we also demand a very high accuracy result of over 90% for 0.9 IOU threshold from our model. The problem is that our dataset is very small with only 500 labeled images of various breeds of dogs, with only 30-40 images per dog breed. And we are interested in detecting the dogs of only beagle bone breed (but with very high accuracy).

We figure out two ways of solving the problem, conventional transfer learning, and recursive transfer learning.

In Recursive Transfer Learning, multiple iterations of transfer learning are performed on the same model. We start by training the model on a generalized dataset and fine-tune it as we go towards a more specific dataset. Moving forward with our example to implement recursive transfer learning, at first, we initialize our CNN with the weight configurations of another CNN trained on a very huge dataset of Image-net COCO. It is the first time we use transfer learning for our model. We train this pre-trained CNN on our smaller dataset of 500 images to recognize a general class of Dog. Instead of having separate labels for dog breeds, we label all 500 images as one general dog class and train our model. This general model successfully detects dogs but does not know its breed. Then we again fine-tune this model for the second time by training it over a smaller and specific dataset having only 30-40 images to recognize only beagle-bone dogs. Now, this final model attains very high accuracy in detecting only beagle-bone dogs (as we wanted). And we can repeat this by moving from general tasks to even more specific tasks.

On the other side, in the conventional approach, we train the model to detect all classes or breeds of dogs at once. This involves only one iteration of transfer learning, using a pre-trained image-net coco model and train it over a dataset of 500 images to detect various breeds of dogs separately. As compared to the recursive transfer learning approach, this model attains lower accuracy.

In my experiments, the strategy of recursive transfer learning performs better than the conventional strategy. Is this strategy already known by a different name? Is it common to use such strategies in Deep Learning projects? Can it solve the problem of lack of data for Deep Learning models like CNNs?? Is it effective?


texts: Time series forecasting with RNN(stateful LSTM) produces constant values

texts:
I have a time series daily data for about 6 years(1.8k data points). I am trying to forecast the next t+30 values, Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values.
I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.
Is this the correct way to forecast time series with LSTMs?
How can this behaviour be explained?
Will this be the case even for a time series with great seasonality?
Is this behaviour expected even for a very large time series data?
Should I rather try to train the network with Y matrix having 31st day to 60th day values for the same X?
My train data looks something like this:
array([[-0.35811423, -0.22393472, -0.39437897, ..., -0.36718042, -0.37080689, -0.35267452], [-0.22393472, -0.39437897, -0.13327289, ..., -0.37080689, -0.35267452, -0.2030825 ], [-0.39437897, -0.13327289, -0.1532185 , ..., -0.35267452, -0.2030825 , -0.25294651]

Architecture: Input LSTM layer (20 neurons) 1 hidden lstm layer 20 neurons 1 output dense layer, batch size as 1. Stateful lstm, I reset model states after each epoch.


texts: Tuning parameters for gradient boosting/xgboost

texts: In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these parameters?


texts: How to perform data scaling/standardization on dataset containing grouped values?

texts:
So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example:

| Problem_instance | Problem_Size | Used_Solver | Cost |
| P1               |           50 | A           |   75 |
| P1               |           50 | B           |  125 |
| P1               |           50 | C           |  225 |
| P1               |           50 | D           |  100 |
| P2               |          150 | A           |  165 |
| P2               |          150 | B           |  360 |
| P2               |          150 | C           |  275 |
| P2               |          150 | D           |   45 |
| P3               |           25 | A           |   35 |
| P3               |           25 | B           |   65 |
| ...              |          ... | ...         |  ... |

I'm trying to use machine learning to predict the best performing Solver for a given problem instance. In data processing stage, I need to standardize or scale my data, but I'm not sure how to this best.

Firstly, I'm not sure which sklearn's Scaler to use (StandardScalar/ MinMaxScaler/..).

Secondly, I'm confused how to handle the different records for each instance. When I group the data first based on problem_instance and then use a MinMaxScaler, the record with Cost = 0 would be the best solution for this problem and Cost=1 the worst. But if I use the same strategy to scale the Problem_Size this would be equal to 0 everywhere. On the other hand if I use a global scaling, the information about which Solver is the best for each instance is lost.

Can someone help me how to handle the data preprocessing for this problem?


texts: In what way is recurrent neural network state "hidden"?

texts: Recurrent neural networks have hidden state denoted $h$. Why is the state considered "hidden"? It's clear to me what purpose the state itself serves, but I can't figure out why it is "hidden". To me, the state seem perfectly "visible" - not "hidden".


texts:
I'm doing a basic correlation analysis but for some reason pandas corr() is deleting columns, not sure why.

import pandas as pd
data = pd.read_csv("data.csv")
print(len(data.columns))
print(len(data.corr().columns))

Output:

100
64


texts: Keras shape error in applications Inception Resnet v2

texts:
I'm using Keras 2.1.3 and I'm trying to fine tune a Inception Resnetv2 with Keras application.

So I load the pretrained model from keras.applications

input_tensor = Input(shape=(299,299,3))
model = applications.inception_resnet_v2.InceptionResNetV2(weights='imagenet',
                                                        include_top=False,
                                                        input_tensor=input_tensor,
                                                        input_shape=(299, 299,3))

I create the bottleneck for my problem :

top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(40, activation='softmax'))

And finally create a new model to concatenate the two parts :

new_model = Sequential()
for l in model.layers:
      new_model.add(l)

At this step, I got an error

  ValueError: Input 0 is incompatible with layer conv2d_7: expected axis -1 of input shape to have value 192 but got shape (None, 35, 35, 64)

So I printed each layer shape and I have

  Layer n-1 : Input : (None, 35, 35, 64), Output : (None, 35, 35, 64)

  Layer n : Input : (None, 35, 35, 192), Output : (None, 35, 35, 48)

As you can see shapes dismatch and it seems weird that come from Keras.

EDIT:

My exact error is

  ValueError: Input 0 is incompatible with layer conv2d_7: expected axis -1 of input shape to have value 192 but got shape (None, 35, 35, 64)

model.summary() (model is model from Keras.applications.inception_resnet_v2.InceptionResNetV2) give me for the conv2d_7 layer

  max_pooling2d_2 (MaxPooling2D)  (None, 35, 35, 192)  0           activation_5[0][0]

  activation_9 (Activation)       (None, 35, 35, 64)   0           batch_normalization_9[0][0]

  conv2d_7 (Conv2D)               (None, 35, 35, 48)   9216        max_pooling2d_2[0][0]

  conv2d_10 (Conv2D)              (None, 35, 35, 96)   55296       activation_9[0][0]


texts:
I wish to predict whether the difference in Value and Growth returns is positive or negative for the next month.

To do this, I have collected data of a few features(to be specific, Macroeconomic Indicators), and the difference in Value and Growth returns for about 200 previous months. I wish the prediction to depend on the order of the previous months. Decision trees and others don't care about the order so I looked upon the net to find about LSTM. Is it correct to use them here?


texts: Model-independent measures for feature importance given highly correlated features

texts:
I am currently working on a research project where the central question is which features drive the prediction of different models. The main issue is, that there is high (multi-)collinearity among those features.

Imagine a setting with about 200 different features that are all potential candidates for helping predict the same dependent variable. In the past, the relevance of small sub-sets of like 5 of these features has simply been analyzed by throwing them into a linear regression model and analyzing R2 and significance of coefficients. In contrast, I want to evaluate feature importance for the complete set of 200 while also getting rid of the forced linear relationship in these regression models, i.e. I do not want to dictate any specific functional relationship ex ante. Thus, I train artificial neural networks and want to compare both their overall out-of-sample prediction performance and individual feature importance with the results from a multiple linear regression.

I am aware of model-independent measures for feature importance that allow comparing importance across different types of models (e.g. permutation importance). However, the issue comes from these measures being quite unreliable when there is high multicollinearity among features. For example, with two highly correlated features, the neural network will likely rely mainly on only one of the two predictors, even though both carry relevant predictive power for the given task. The same issues would of course also be present in the linear model. When you are just interested in your overall model performance, this would not really be an issue. However, my specific research question specifically deals with understanding which features carry relevant information for predicting the dependent variable (when not dictating any functional form ex-ante).

The only possible solution I have come up with so far, is specifying a feature's importance by taking maximum of its importance or any of the products of its correlation with another feature times that features importance (e.g. take the larger of the following two: feature importance of feature i vs. feature importance of feature j * correlation between i and j). But while I am not even sure if this is a "scientific" appropriate approach, it does also only adress the problem of pairwise correlation.

What would your go-to-approach be for creating a reliable measure for feature importance in the case of high (multi-)collinearity be? Can you maybe point me in the right direction of previous research on this topic that I might have overlooked?

Thanks for your help!


texts: How to trust the labels generated using ML models?

texts:
I have a dataset of patient records. But I do not know whether he is +ve for a cancer or not. So, I do not have the labels in my dataset.

Now I can run a machine learning models like clustering to generate labels.

For ex: I can run clustering to group the two classes based on similarity and find out who all belong to +ve and -ve class.

Of course, we cannot sit and manual review the patients' data to know whether he is actually +ve for cancer or not.

So when we generate labels via machine learning models like clustering above, is it a recommended approach?

Is it used in industries/real time where people don't have ground truth and only rely on labels based on ML models?

How can we trust these labels generated?

If it's a human I know that it can be trusted. But how do we trust these labels.

Are things like this being used in Industries and how do they tackle the trust issue?


texts: Tactics to avoid feeling overwhelmed by machine learning

texts:
Short version: despite lots of reading, machine learning still feels like being a monkey in the dark. Any advice?
For background, I'm a researcher in computer science, in a field non-related to machine learning.
I have been trying to get more proficient in machine learning*, yet no matter how much I read and fiddle with code/toy datasets, when I try to go to a harder problem, I always feel overwhelmed by the choices I need to make:

I have to choose the algorithm: This is the part I typically  find the most straightforward;
For said algorithm, I have to choose the objective function :  usually, many are applicable, and I find it difficult to gain a good intuition of what makes an objective function adapted in some cases rather than others, apart from the very classical ones for linear or logistic regression
And then, I should devise the features: this still feels completely arcane to me, apart from using content-based features readily available in the data.

I am under the impression that I have to "create" the tailored algorithm and the data.
Concerning the algorithm, I have spent some time into studying gradient boosting and the math behind it, to the point that I have a reasonably solid comprehension of how it works, and an intuition of parameter tuning for simple datasets. However, that knowledge does not generalize.
How are these issues typically approached? Are there any resources that can help?
* By taking the Machine Learning  Coursera course and its more in-depth version, reading more XgBoost-specific material (on its internals and parameter tuning and intuition), as well as playing with the Titanic dataset, and a housing market dataset.


texts:
I'm learning about decision trees, and I feel like up till now I've understood them and the math behind them pretty well except for one thing: the gain ratio.

As I understand, the gain ratio is implemented in order to punish for features that may take on A LOT of possible values.

If a feature takes on a lot of possible values, it becomes plausible that if we split on that feature there may be values that only point to a single class, but simply because there are only 1 or 2 data points with that value for that feature anyways.

In other words, the only reason that we would get low entropy for splitting on that feature is because the feature could take on a lot of values, and therefore a lot of those values pointed specifically to a single label. So our decision tree algorithm would end up splitting up on something like "ID #", and wrongly calculate that we just had a HUGE information gain.

However, this only seems like a problem because "ID#" is a feature we shouldn't be splitting on to begin with. I mean, if we had another feature that also took on a lot of possible values, but each of those values actually DID imply some label for that datapoint, then wouldn't applying the gain ratio mean that we are actually messing up our decision tree by punishing what was actually a very good split with tons of information gain?

Isn't it better to just identify which feature will have nothing to do with our labeling BEFORE we feed in the training data to the algorithm?

IDK, I just don't see why the gain ratio would really be useful...


texts: This will probably be a basic question since I am starting with computer vision. I am trying to use resnet18 from pytorch and work with CIFAR-100 dataset. Single image has size 3x32x32 and the model cannot forward this throwing error. It can process arrays of size 224, 128, 64. How should I structure my images to process them with resnet18 specifically (not other arch).


texts: Can anyone explain the reasoning behind this post?

texts:
I am reading THIS blog post,
and I do not understand the logic behind this part:

Why is the relu here max(0, Xavg - X) ? And even so, it does not really explain the values obtained in the two situations where X1 = X2 = 0, X1 = 1 and X2 = 0, and X1 = X2 = 1.

Can someone please explain this in more detail?


texts: I have 3 graphs of a binary Logistic Regression that I want to understand better what is happening and learn of a strategy to make the model better

texts:
My problem is the following: I have a binary Logistic Regression model with a very imbalanced dataset that outputs the percentage of the prediction. As can be seen in the images, as the threshold is increased there's a certain point it stops predicting.
I am researching calibration techniques to try and make it work better but I thought maybe I could get some direction here.

I've tried giving weights to the classes but it didn't seem to get much better.

Is it a probability calibration problem?

The three graphs below are shown in no particular order.

Thanks in advance.


texts:
Similar questions have been asked in this context but this one seems a bit different.

[Aim]
We would like to find out what the probability is of a person purchasing a 4USD column D (see below) price given that he has purchased C-2, B-2, A

[Context]
We have a sizeable dataset of about 500,000 observations of this pattern. E.g. customer decides to buy A), bought B-1, then C-2, D-3.

[Issue]
We don't know how deep the events reach. E.g. it can be to column E, F, G. That's why we would like to auto-generate a probability tree in preferably SPSS but are open to other solutions as well.
How does this look like mathmatically? Can we just extend bayes conditional probability? If so, how does it look? Any potential issues?
Any other comments/ ideas are always welcome :)

Thanks!


texts: Predict output sequence one at a time with feedback

texts:
I would like to solve the following classification problem: Given an input sequence and zero or more initial terms of the true output, predict the next term in the output sequence.

For example, my training set is a large corpus of question and answer pairs. Now I would like to predict the response to "How are you?" one word at a time.

Some reasonable responses might be "I'm fine.", "OK.", "Very well, thanks." etc. So the predicted distribution over possible first words would include the first words in those examples, among others.

But now if I see the first word is "Very", I would like my prediction for the second word to reflect that "OK" is now less likely, because "Very OK" is not a common response.

And I would like to repeat this process of predicting the next word given the input sentence and every word so far in the output sentence.

One approach might be to train a stateful RNN on examples like How are you<END> I am fine<END>. But if I understand correctly this would also learn to predict words in the "How are you" part, which I think might detract from the goal.

Maybe separate layers for the input and partial output that are then merged into a decoder layer?


texts: XGBoost Classification Probabilities higher than RF or SVM?

texts:
I am using Random Forests, XGBoost and SVMs to classify whether the home team wins or the away team wins their bowl game (in college football). I trained the models on all the games during the season.
I've come across something that is a bit weird and can't explain. I calculated a prediction confidence by subtracting the class probabilities. The XGBoost confidence values are consistency higher than both Random Forests and SVM's. I've attached the image below.
I did some hyper-parameter tuning for all of my models and used the best parameters based on testing accuracy.

Random Forest:

700 trees
15 variables randomly sampled (mtries)
minimum split criteria of 5 rows.

XGBoost:

0.5, Learn rate
gbtree as my booster
max depth of 6

SVM:

RBF kernel
C (slack) of 1
0.01, Sigma

I wasn't clear with my question: Why exactly does XGBoost prefer one class greatly to the other? In comparison to these other methods. I'm trying to figure out why my prediction confidences of a class are so high for XGboost.


texts:
I'm a beginner and I'm trying to do a clustering of a multi-sentence text, but my results are horrible. Any tips for me to improve my result?

import pandas
import pprint
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
from sklearn.feature_extraction.text import TfidfVectorizer

dataset = pandas.read_csv('text.csv', encoding = 'utf-8')

comments = dataset['comments']

comments_list = comments.values.tolist()

vetorize = TfidfVectorizer()
X = vetorize.fit_transform(comments_list)

clusters_number = 6
model = KMeans(n_clusters = clusters_number, init = 'k-means++', max_iter = 300, n_init = 1)

model.fit(X)

centers = model.cluster_centers_
labels = model.labels_

clusters = {}
for verbatim, label in zip(verbatim_list, labels):
    try:
        clusters[str(label)].append(verbatim)
    except:
        clusters[str(label)] = [verbatim]
pprint.pprint(clusters)

#Top terms for cluster
print("Top termos par cluster:")
ordem_centroides = model.cluster_centers_.argsort()[:, ::-1]
termos = vetorizar.get_feature_names()
for i in range(clusters_number):
    print ("Cluster %d:" % i,)
    for ind in ordem_centroides[i, :10]:
        print (' %s' % termos[ind],)
        print()

I have many different themes present in various clusters. I pre-processed my data (stopwords, lowercase, I removed the punct...). But still I have 'like to cancel order' in one cluster and 'love cancel order' in another. When, in fact, the ideal is to join all 'cancel order' in a single cluster.


texts:
My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational database" should produce words like "mysql", "Oracle", "Sybase", "Microsoft SQL Server" etc...

My thinking is to treat the initial text as a set of words (after lemmatization and stop words removal) and predict words that should be in that set. I can then take all of my texts (of which I have a lot), remove a single word and learn to predict it.

My initial thought was to use word2vec and find words similar to my set. But this doesn't work very well, as I don't want similar words but words that go together in many sentences, which is sort of the task that word2vec is training on to create its embedding...

How would you model this problem? I don't think RNNs are relevant here because I want to use a set of words - they do not have any order, I'm not trying to predict the next word in any way. However, the size of the set could vary I think...

What kind of NN architecture would you use for this sort of problem?

UPDATE:

To further explain the purpose of the predictor and why word2vec doesn't work for me:

I did train a word2vec model on my data (which is very domain specific and I have lots of data on it). Now my problem is that it produces words that are similar to the ones I input to it... and not ones that could be in the context.

e.g: I input "computer manufacturer" to get most similar words and i get:

(u'equipment_manufacturer', 0.7463390827178955),
 (u'manufacture', 0.7410058975219727),
 (u'manufacturer_distributor', 0.7196526527404785),
 (u'distributor', 0.6950951814651489),
 (u'desktop_computer', 0.6632821559906006)

These are terms that have similar meaning...

But I'm not looking for similar meaning. I would be more interested in getting "Dell" for example as the output.
Another example is to associate "Apple_inc" with informatics, OS, hardware.. - not similar words, but related ones...

So my thinking now is to build a word2vec CBOW model, but not to use just the embeddings that are created but actually use the network to predict new words from context...


texts: Is it appropriate to run correlation on continuous y value with discrete x value?

texts:
I am dealing with temperature data that are measured in different distances away from the forest edge (0m, 10m,30m,50m,80m) and I would like to find out whether the temperature will drop as the distance away from the edge increases. I tried to use geom_point to plot the temperature with respect to the distance, however, the x axis value are not labelled as 0m, 10m,30m,50m and 80m, may I ask how to change it?

Also I would like to ask is it appropriate to run correlation using temperature and distance from the edge to show that they are negatively correlated in this situation? I saw correlation should be based on two continuous variable, but the distances from the edge are discrete variable, so should I just present the result using boxplot? Or is computing regression appropriate in this case?


texts: How does sklearn random forest decide feature threshold at node splitting exactly?

texts:
Thinking of the RandomForestClassifier function in sklearn.ensemble, I understand that at each non-terminal node the algorithm:

Randomly selects a subset of size max_features from the set of all features
Searches for the feature x and the threshold value x_0 such that when the node is split according to {x<=x_0} and {x>x_0}, the Gini impurity is minimized (I'm using the Gini setting)

My question is about how exactly this search is carried out and a value for x_0 chosen. The obvious method would be to just test all possibilities, taking into account that only the order of x_0 relative to the training set matters. That is, look at the x-values of all training points. If a is the largest of these values <= x_0 and b is the smallest of these values > x_0, then any other choice of x_0 within the half-open interval [a,b) would have produced exactly the same splitting. So it suffices to test one value of x_0 from each half-open interval and choose the one with the lowest Gini impurity.
To be specific then, my question is:

Does sklearn check all possibilities like above?
The value of x_0 makes no difference in the training step as long its nearest neighbors in the training set don't change. But it may make a difference in the testing step, when the tree is applied to new data points. So how does sklearn decide a specific value for x_0 in the training step?


texts: Neural Network Architecture for Identifying Image Copies

texts:
I have a large image collection and wish to identify the images within that collection that appear to copy other images from the collection.

To give you a sense of the kinds of image pairs that I wish to classify as matches, please consider these examples:

I have hand classified roughly .25M pairs of matching images, and now wish to use those hand labelled matches to train a neural network model. I am just not sure which architecture would be ideally suited for this task.

I originally thought a Siamese Network might be appropriate, as they have been used for similar tasks, but the output from those classifiers seems more ideally suited to finding different figurations of the same object (which is not what I want), rather than different printings of the same figuration (which is what I want).

If anyone can help recommend papers or architectures ideally suited to identifying images given the training data I have prepared, I would be tremendously grateful for any insights you can offer.


texts:
I want to use GAN for data augmentation but I am confuse what are the pros. and cons. of data augmentation using GAN or why we use data augmentation using GAN compared to other data augmentation technique like standard data augmentation?

Your answer will be a great help to me.
Thanks.


texts: Image similarity: Similarity of mixed vector

texts:
In order to identify the similarity between images (products) I want to use a neural network approach similar to TiefVision. This pre-trained neural network is basically translating the images into a feature vectors and then creating a similarity measure between the images using a distance measure between the vectors. To make it more tangible have a look at a 2D visual representation below.

I want to take it one step further: When a single user "likes" multiple images, I want to average their feature vectors. This will result in a new vector, which I want to use to calculate recommendations. My assumption is that images close to the combined feature vector will possess similar features towards all "liked" images together.

Now I wonder: Is my thinking flawed - maybe because averaging the vectors will simply lead to entirely different images, or will it actually produce images with similar features?

(source: indico.io)


texts: business ranking based on transaction data

texts:
I have some customer (say 10,000 customers) transaction data of many stores, what would be a good approach to rank these stores?

The transaction data is of entire year, and the transaction data including transaction amount, type of store, location of store ... etc.

And how would one evaluate such ranking algorithm? Thanks!


texts:
I often find myself writing code like the following (oversimplfied example)

df = read_csv('customer_data_export.csv')
df2 = df.query("date > '2017-01-10'")
data = df_filtered.groupby('transaction_id').sum()
plot_data = pivot_table(data, columns='weekday', rows='n_items')
# Etc etc

Basically the problem is that while it's relatively easy to come up with semantic names for columns (as random variables) it's hard for me to come up with meaningful names for each step of transformed dataframe. Additionally I prefer to have short names to make the code easier to type. (Working in Jupyter notebook, the tab-completion isn't the best).

What are some best practices that people follow with this kind of thing?


texts: Text Analysis : Recommendation to identify cause of loss from claim narrative documents

texts:
I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify cause of loss like : drunk and driving , negligence , bad weather , etc ?
I have used TF-IDF technique to rank order words per claim but it does really help to identify the exact cause of loss easily even if I concentrate on top 20% words. I am aware of word embedding but really not sure if they could help .
Also tried LDA topic modeling in the past but that gives topics for entire corpus rather than claim level.
Below is the code I used to calculate tf-idf scores for the unigrams.
library(dplyr)
library(tidytext)
library(janeaustenr)
library(tidyr) # for separate
library(qdapDictionaries)

#generate unigrams
loss_narrative_by_file_1 <- loss_narrative_by_file %>%
  unnest_tokens(unigram, whole_text_proxy, token = "ngrams", n = 1)

# removing punctuation and numbers ( removing punct / digits faster on unigrams than whole text)
loss_narrative_by_file_1$unigram <- gsub("[[:punct:]]|[[:digit:]]|^http:\\/\\/.*|^https:\\/\\/.*","",loss_narrative_by_file_1$unigram)

#retaining non blank unigrams
loss_narrative_by_file_1 <- loss_narrative_by_file_1[unigram!="",]

# retain meaningful words only
loss_narrative_by_file_1 <- loss_narrative_by_file_1[unigram %in% GradyAugmented]

# remving stopwords

unigrams_filtered <- loss_narrative_by_file_1 %>%
  filter(!unigram %in% stop_words$word)

#check coutns

unigrams_filtered %>%
  count(unigram,sort=TRUE)

# after cleansing
unigram       n
<chr>     <int>
  1 claim     56247
2 loss      55068
3 policy    50394
4 insured   47512
5 date      36879
6 coverage  32167
7 plaintiff 31358
8 vehicle   26156
9 umbrella  25055
10 company   24479

# tf- idf ###############################################

# concat claim & filename
unigrams_filtered$loss_file <- paste(unigrams_filtered$loss_Loss,unigrams_filtered$filename_Trimmed, sep = "_")

unigram_tf_idf <- unigrams_filtered %>%
  count(loss_file, unigram) %>%
  bind_tf_idf(unigram, loss_file, n) %>%
  arrange(desc(tf_idf))

### validating results of tf-idf shows top 30% words with highest tf-idf score tend to contain the words associated with cause of loss. How can I narrow down further ?enter code here


texts: Visualize performance, % of goal implementation

texts:
I want to visualize goal achievment progress.
This is my first idea:

use area chart to show progress in current metric
use horizontal band to show the goal value
colorize areas under/above the band into "positive" and "negative" colors

Is this approach informative enough? Are there better choises?

Additional info:

charts made in Tableau
two data sources: metric progress & goals


texts: ensuring that dependent variable decreases monotonically with independent variable

texts:
I have one key relationship between a numeric independent variable X and a numeric dependent variable Y, which is like a negative exponential function determined by 2 parameters. There are other independent variables. One numeric, the rest are categorical.

At the moment I am using quantile regression forests. They work pretty well. However, the aforementioned negative exponential relationship is not guaranteed.

Is there a way to achieve the above. For example, could this be modeled like a hierarchical model. Here the parameters of negative exponential function over X are adjusted depending on the other features. In fact, could one model this as a ANN (e.g. using KERAS) where the negative exponential function is the output layer - similar to the softmax function?

Any feedback would be very much appreciated!


texts: How to Split And Resample Imbalanced Dataset Into Train, Validation and Test

texts:
I want to understand how to split the imbalanced data set with a binary target variable where 87% of the samples are negative and 13% of the samples are positive. Now, I know that you should always split the data into train and test set before doing any kind of resampling to avoid info leakage but how what strategy gives better results - random train_test_split or StratifiedShuffleSplit? The general sentiment is that the latter seems to be outperforming the former in case of an imbalanced data set. Btw, how does StratifiedShuffleSplit work?

Also, how to factor in the cross-validation set and how to process it?

Should it also contain the same number of positive and negative samples as per the sampling was done on the training set?


texts: Will disparity in image format/quality between binary classifications affect training of Convolutional Neural Network?

texts:
I have an image dataset containing two classes. One of the classes has many images and they are all JPG images with the following format:
Horizontal/Vertical Resolution: 96 dpi
Bit depth: 24
Size: ~217 KB

The other class contains images which I (for the most part) have had to create myself. There are a few that are from the original dataset but the rest are created outside of it. These images have a format like so:
Horizontal/Vertical Resolution: 120 dpi
Bit depth: 32
Size: ~2.75 MB

Both image classifications are of type .jpg and have dimensions 1280 x 960
During training, all of these files will be reduced in size and converted to a Python numpy array using the following code:
import numpy as np
from skimage.io import imread
from skimage.transform import resize

np.array([resize(imread(str(fileName)), (width, height, channels)) for fileName in list_of_filenames])

This gives me one batch of images that have been read from a directory, reduces their size to the specified height/width, and puts them into a numpy array.
My question is, with the disparity in image size, bit depth, and dpi will my image classifier simply become a classifier for the image format itself rather than the content of the image? And how would the differences in the images affect the resize function?
Thank you


texts: Clustering time series based on monotonic similarity

texts:
Context

I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, but responding to the same exogenous variables. However, for each time series, the magnitude of the response is very different. For a time series of reference $X$, I would like to be grouped in the same cluster series that are alike $X^a$ for all $a > 0$.

Tryouts

So far, my interpretation of the problem is that I want to cluster time series sharing a strong monotonic relationship. My first tryouts used hierarchical agglomerative clustering by defining a distance based Kendall's tau rank coefficient since it measures the strength of a monotonic relationship. By visual interpretation, best results were obtained using Ward's linkage method. However, this approach seems unorthodox, non-robust, or doubtful for several reasons.

First, Scipy documentation mentions here that Ward's method is only correct when the Euclidean distance is used. Secondly, I couldn't find any detailed application of time series clustering based either on Spearman or Kendall's tau coefficient. Furthermore, I was very surprised that I couldn't find any paper or reference aiming at clustering based on a monotonic criterion.

I am willing to consider other approaches, though I cannot measure their benefits. For instance, rescaling all time-series to map them to a standardized gaussian distribution (e.g. Box-Cox) and then using the Euclidean distance. Another possibility is to turn the first difference of time series into a boolean vector (1 if $\Delta X >0$, $0$ otherwise) and then use the Euclidean distance or another distance metric.

Questions

Since I am new to time series clustering, I have some troubles to picture by myself what would be the best approach(es) (or the worse) for this specific purpose. Hence, I have two related questions:

Specifically, is using Hierarchical Clustering based on Kendall's tau and Ward's linkage method a wrong way to go and why?
Generally, what is the best way to cluster time series based on monotonic association?

Some references on the topic are also welcome.


texts: Is there a standardized way to do data analysis?

texts:
Is there a standard way to do data analysis?
So for example, something like this:
 1. Data mining
 2. Data cleaning
 3. xx
 4. Data and result interpretations

I ask this because I'd like to do it for a paper which analyses a model of Data Science and, therefore, a common method would be appreciated. For a Literature Review, its f.e. (Transfield et. al 2003).
I actually could not find any reliable source, neither an approach to do it standardized.


texts:
I am looking for sentiment analysis data, mostly customer product review.

I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date data and I am willing to pay.

Is there any vendor that sells aggregated sentiment data from retailers or e-commerce. Or is there any other avenues we can get the review data.


texts: how are split decisions for observations(not features) made in decision trees

texts:
I have read a lot of articles about decision trees, and every one of them only focused on telling how a feature/column is considered for split, based on criterion like gini index, entropy, chi-square and information gain. But, not one talked about the observation part.
Example: Let's say I have a dataset with 3 independent features and 1 discrete target variable, namely height_in_cm(like 130, 140), performance_in_class(like below average, average, very good), class(like 7th, 8th or 10th class) and plays_cricket(1 for yes & 0 for no) is the target variable. So, with target variable as the root node and then for splits, I may try all the features iteratively and settle with the one that I had most information gain with or with most pure nodes. ex, let's start with any variable, let's take the first one, height_in_metres and after two splits with two child nodes, like height < 120 in one child node and another child node with height > 120 and i will then calculate gini impurity and they turned out to be 0.45 and 0.49 respectively.
Questions:

just like i iteratively try all the feature combinations, do i also need to try all the combinations of a feature split, for above case, height < 100 & height > 100, then height < & height > 110 & height < 90 & height > 90 and so on. How to do this and what's the efficient way.
just like there are metrics like gini impurity and entropy to measure the quality of a feature split, are there any metrics to measure the quality of split based on observations?


texts: How to use unigram and bigram as an feature on SVM or logistic regression

texts:
How to use unigram and bigram as an feature to build an Natural Language Inference model on SVM or logistic regression?on my dataset i have premise, hypotesis and label column. I'm planning to use the unigram and bigram of the premis or hipotesis or both as one of the features on my training.
for example :
 premise                                      |hipotesis                         |hypothesis bigram
===============================================================================================
I am planning to use the unigram and bigram   |I am planning to use the unigram  |[(i, am), (am, planning), (planning, to), (to, use), (use, the), (the, unigram)]

the hypothesis bigram is a list of bigram(word), so i cant use it as input to my svm or logistic. can i convert the hypothesis bigram into vector?


texts: How to use logistic regression with decay

texts:
I have a dataset with binary outcomes. I use Logistic regression for making the prediction.

example of my data :

id    Col_A    ...    y
1       5      ...    1
2      10      ...    1
3     500      ...    0
4     705      ...    0
5       2      ...    1

I would like to apply a decay on my logistic regression function
i.e. the prediction result will depend on col_A: if col_A contains a high value the prediction must be converged to 0 (an exponential function). So that I will get probability value instead of binary

Is it possible to do that? and if it is possible how can I achieve it?

I use the followig Logistic regression source code:

class LogisticRegressionUsingGD:

    @staticmethod
    def sigmoid(x):
        return 1 / (1 + np.exp(-x))

    @staticmethod
    def logit(theta, x):
        return np.dot(x, theta)

    def p_hat(self, theta, x):
        return self.sigmoid(self.logit(theta, x))

    def cost_function(self, theta, x, y):
        m = x.shape[0]
        a = y * np.log(self.p_hat(theta, x))
        b = (1 - y) * np.log(1 - self.p_hat(theta, x))
        total_cost = -(1 / m) * np.sum(a + b)
        return total_cost

    def gradient(self, theta, x, y):
        m = x.shape[0]
        return (1 / m) * np.dot(x.T, self.sigmoid(self.logit(theta, x)) - y)

    def fit(self, x, y, theta):
        opt_weights = fmin_tnc(func=self.cost_function, x0=theta, fprime=self.gradient, args=(x, y.flatten()))
        self.w_ = opt_weights[0]
        return self

    def predict(self, x):
        theta = self.w_[:, np.newaxis]
        return self.p_hat(theta, x)


texts: Stuck implementing k means for big and small dogs - dodgy results

texts:
my algortithm isnt working. The code seems to make sense and everything but im just not convinced with the results. I fell like the centroids should be amongst the data more, sort of central, but they arent doing that no matter how many iterations i set the algorithm too. Could someone run my code and give me some pointers? Ive labeled the code so you can understand it.
Some main points:

k = 2; 2 centroids
centroid1 = Dark red, so datapoints in this cluster are light red
centroid 2 = Dark green, so datapoints in this cluster are light green
dogs are either 'big' or 'small' depending on their height and weight
The euclidean distance is used to decide on which cluster to belong to
Raw/main data stored in dictionary labeled 'x'; they are moved about into the dictionaries 'cluster1' and 'cluster2' depending on their distance to the centroids

code:

from matplotlib import pyplot as plt
import random, math

plt.title("Big Dogs and Small Dogs")
plt.ylabel("Height (cm)")
plt.xlabel("Weight (kg)")

def euclideandis(x, y, a, b):   # Formula for euclidean distance
    return math.sqrt((x-a)**2 + (x-a)**2)

x = {4.4:31,
     3.2:19,
     4.6:32,        # Data- key = weight, value = height
     4:25,
     4.1:29,
     2.90:17,
     2.9:11}

centroid1 = [[random.uniform(min(x.keys()), max(x.keys())), random.uniform(max(x.values()), min(x.values()))]]   # create random centroids that are between the datapoints
centroid2 = [[random.uniform(min(x.keys()), max(x.keys())), random.uniform(max(x.values()), min(x.values()))]]

for j in xrange(1):   # keep updating colours, position of centroids
    cluster1 = {}       # Everything in cluster 1 is closest to the dark red spot, therefore gets included in this dict, and gets scattered in light red
    cluster2 = {}       # Everything in cluster 2 is closest to the dark green spot, therefore gets included in this dict, and gets scattered in light green
    for key in x:
        temp1 = 0       # works out euclidean distance for all data points in x, then compares them
        temp2 = 0
        temp1 = euclideandis(key, x[key], centroid1[0][0], centroid1[0][1]) # Dis betw data point, red centroid
        temp2 = euclideandis(key, x[key], centroid2[0][0], centroid2[0][1]) # Dis betw data point, green centroid
        if temp1 < temp2:
            cluster1[key] = x[key]      # if the euclidean distance between datapoint and red spot,
                                        # smaller than the euclidean distance between datapoint and green spot,
                                        # add the point to the red cluster, else add to green cluster
        else:
            cluster2[key] = x[key]

    centroid1 = [[0, 0]]        # Centroids reset as they will be changed
    centroid2 = [[0, 0]]

    iterable = 0
    for key in cluster1:        # works out mean coordinates of each cluster and changes the centroids coordinates to this
        iterable = iterable + key
    centroid1[0][0] = iterable/len(cluster1)
    iterable = 0
    for key in cluster1:
        iterable = iterable + cluster1[key]
    centroid1[0][1] = iterable/len(cluster1)
    iterable = 0
    for key in cluster2:
        iterable = iterable + key
    centroid2[0][0] = iterable/len(cluster2)
    iterable = 0
    for key in cluster2:
        iterable = iterable + cluster2[key]
    centroid2[0][1] = iterable/len(cluster2)

plt.scatter(cluster1.keys(), cluster1.values(), color = "red")      # scatters everythning
plt.scatter(cluster2.keys(), cluster2.values(), color = "lime")
plt.scatter(centroid1[0][0], centroid1[0][1], color = "maroon")
plt.scatter(centroid2[0][0], centroid1[0][1], color = "green")
plt.show()

To sum up, there isn't a clear error with my program, only I'm not really convinced about the results

(above) why after 100k iterations arent the centroids in the middle of the datapoints? The red one is but the green one isnt


texts: Converting Text Entries into values for machine learning - KNN

texts:
Consider the following dataset.

| Area | Job Type | Complete |
-------------------------------
| AAA  | Install  |     N    |
| AAB  | Repair A |     Y    |
| OOC  | Repair C |     Y    |
| LCX  | Cease    |     N    |
-------------------------------

I am using JavaScript (Started getting into Tensorflow.js amongs other ML algorithms and theories.) and I am struggling to find a suitable ML method to process this. There could be over 100 areas and over 15 Job Types, but Complete can only be Y/N.

I was thinking about assigning a number to each case as follows

AAA -> 1    |   Install  -> 1   |    Y -> 1
AAB -> 2    |   Repair A -> 2   |    N -> 0
OOC -> 3    |   Repair C -> 3   |
LCX -> 4    |   Cease    -> 4   |
... -> x    |

Is this viable? would it work?

I want to give it another case and return the % chance of that case happening.
I have tried to use Naive Bayes classifier and has some success.

-------------------------------
| LCX  | Cease    |     10%    |
-------------------------------

This is just a small sample there are other x's that I want to include that are a  mix of floating points and other String Values.
the whole main data-set contains over 40 million entries and over 40 possible columns that could be a factor that effects 'completed' so plenty of training data to go around!

What would the best method be what approach would you recommend?


texts: How to select the significant cross-terms in logistic regression?

texts:
I have data where the number of feature vectors and the number of target classes are identical. I built a logistic regression model to learn the class of unseen data using training feature vectors including cross-terms whose coefficients are look like this.

Here the total number of classes and feature vectors both is 16. Total number of rows represent feature vectors which is 16+(16 choose 2) = 16+120 = 136.  So each row represents the logistic regression coefficients of the 16 classes. My question is how can I check the co-localization of the cross-terms. Colocalization I meant by any two feature vectors together can predict any class of the classifier. Similarly, certain cross-term does not look relevant so under what criteria they can be removed. If you have any suggestions it will help me to think more. Thanks.


texts: Relation between hyperparamters and training set for an object detection model

texts:
I have 2 instances of an object detection model. The only difference between these two models is the training data used:

The first model was trained with a small training set
The second model was trained on a larger training set than the first one

The first model was trained on the following hyperparameters:

Number of iterations: 250k
Batch Size: 10
Learning Rate: warms up to 0.001 and decreases to 0.0002 after 150k iterations

Since the second model has more training data, I assumed I need to change the hyperparameters a bit. So I tried training the second model on the following hyperparamters:

Number of iterations: 600k
Batch Size: 10
Learning Rate: warms up to 0.001 and decreases to 0.0002 after 400k iterations

When I measure the mAP for both models on a testing set, the first model vastly outperforms the second model.

model 1 mAP: 0.924

model 2 mAP: 0.776

This leads me to my question for you:

How would the hyperparameters (batch size, learning rate etc) change when the size of your training set increases? What factors need to be considered for this increase in training set size, in order to get the most optimal model possible?

Any and all responses will be greatly helpful. Thank you :)