texts: What is the difference between feature generation and feature extraction?

texts:
Can anybody tell me what the purpose of feature generation is? And why feature space enrichment is needed before classifying an image? Is it a necessary step?
Is there any method to enrich feature space?


texts: What are the tools to plot cluster results?

texts:
I am clustering based on my cosine similarity matrix. Now I want to plot/visualize my very large clusters. I am interested in using a tool that is better than sklearn.

Please recommend me.


texts: Hive Bring Table to Local Driver for Fast Debugging on CLI

texts:
I have a large Hive table on HDFS. Every time I query it, it runs a map reduce job which is slow.
For debugging my code on CLI, I want a fast query. Is it possible to sample rows of the table and bring them to the local machine. Then the hive queries would only run on the local machine without map reduce. If yes, how?
PS I know that map reduce runs for some commands only. I have nested structs which trigger map reduce.


texts: Why is the number of samples smaller than the number of values in my decision tree?

texts: I'm using scikit-learn RandomForestClassifier for a classification problem. When taking a closer look at one of the trees I noticed that the number of samples at the root was 662, but there were 507 instances of the first class and 545 of the second. What's going on or did I understand something wrong? Is the number of samples actually the number of unique samples and since I used bootstrap aggregation there are many samples that were chosen multiple times?


texts:
Why is it that the formula for each element in a convolution between an image $I$ and a $k \times k$ sized kernel $K$ is
$$ (I*K)_{ij}=\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}I_{(i-m),(j-n)}K_{mn}=\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}I_{(i+m),(j+n)}K_{-m,-n}$$
Seeing as the double sum formulas should be equal, we need to work on the same region of $I$ in both formulas. In the left formula we work on a region $I[i-(k-1):i, j-(k-1):j]$ whereas we in the right formula work on the region $I[i:i+(k-1), j:j+(k-1)]$.
Shouldn't the formula instead be
$$ (I*K)_{ij}=\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}I_{(i-m),(j-n)}K_{mn}=\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}I_{(i-p+m),(j-p+n)}K_{-m,-n}$$
So that the regions are the same if we use the padding $p=k-1$.


texts: How to train the predicting boxes in a YOLO network?

texts:
I have just finished this tutorial that explains how YOLO networks work. Instead of training the network's weights with a training set, the author loads pre-trained weights and uses them to test the network. I'm interested in how you would train this network.

I want to run the network on an image that is size 1000 x 1000 and can hold objects with 10 different classes. The network's forward function returns a tensor that has dimensions

[1, num_bounding_boxes * final_layer_grid_width * final_layer_grid_height, 15].

For each cell and bounding box in the final layer, the network predicts 15 values:

x_center, y_center, bbox_width, bbox_height, object_confidence_score, class_1_confidence_score, ..., class_10_confidence_score

I have a list of labels corresponding to the images. Each element in the list (corresponding to an image) looks like this:

[[313,567,47, 23, 4],
[398, 122, 57, 32, 6],
 ...,
[499, 993, 47, 19, 8]]

The first element indicates that the point (313, 567) of the original H x W image is the center of an object, that the corresponding bounding box is of width 47, of height 23, and that the class is 4.

My plan was to check to which grid cell in the final layer each object should be assigned based on its coordinates and the stride of last layer. Suppose that the last layer is 100 x 100 and the input image is 1000 x 1000 then the first element in the example label should be predicted by grid cell [31,56], because the stride is 10. I think that I can create a target Tensor based on this method and I think that I can train the
object confidence score and the 10 different class scores this way.

What I'm not quite sure of is how to train the x_center, y_center, bbox_width, bbox_height. What values should these parameters hold if the grid cell in the final layer should not predict a bounding box?

Question: How do I train the x_center, y_center, bbox_width, bbox_height parameters?


texts: Clustering categorical variable values based on continuous target values

texts:
Let's say I have $n$ data points with just one categorical feature $x$ and a continuous target variable $y$. I want to divide the possible values of $x$ into subsets such that the value of $y$ doesn't vary much within a subset.

As an example, suppose there are only $5$ possible values of $x$: $x_1,\ldots,x_5$. We observe that the values of $y$ don't vary much across $x_1$ and $x_2$. $y$ doesn't vary much across $x_3$ and $x_5$, but is different from the typical $y$ value for $x_1$ and $x_2$. For $x_4$, $y$ values are quite different from the above groups. So we can say that $(x_1,x_2), (x_3,x_5)$ and $(x_4)$ can be considered as "clusters".

Now what's a concrete way of saying that $y$ doesn't "vary much across $x_1$ and $x_2$"? One natural way to define that is for $y$ to have the same distribution for $x_1$ and $x_2$, i.e. the same mean and standard deviation of $y$ values for both $x_1$ and $x_2$.

Is there a better way to characterize the fact that $y$ doesn't "vary much across $x_1$ and $x_2$? Maybe the way I defined it above is too idealistic and I need a criterion more suited to a real-life dataset?
Are there any popular existing methods or library functions to solve such kind of a problem (i.e. clustering categorical feature values on the basis of continuous target values)?


texts: What would be the main and essential criteria for evaluating auto-sklearn library ?

texts: I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and sklearn has to be manually tuned. what could be the essential criteria to judge the performance between these two libraries


texts: How to find similarity between different factors in a dataset


texts:
I'm a beginner in Keras. I've loaded MNIST dataset in Keras and checked it's dimension.
The code is

from keras.datasets import mnist

# load data into train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
print("Shape: ", X_train[0].shape)

And the output is

(60000, 28, 28, 1)
(60000, 10, 2, 2, 2, 2)
(10000, 28, 28, 1)
(10000, 10, 2, 2)
Shape:  (28, 28, 1)

As X_train and X_test are already in the shape (#sample, width, height, #channel). Do we still need reshaping? Why?
The tutorial I'm following use the following reshaping code:

X_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype('float32')
X_test = X_test.reshape(X_test.shape[0], 28, 28, 1).astype('float32')

My second question is that why is .astype('float32') is used in code?

Lastly, I could not understand the output of print(y_train.shape) and print(y_test.shape).

Please suggest.
I've already read Reshaping of data for deep learning using Keras however still my doubts are unclear.


texts: Tensorflow v1 Dataset API AttributeError with ndim

texts:
I'd like to make pipeline for optimizing Gpu and Cpu.
Dataset
It's about 10000 datapoint and 4 description variables for the regression problem.
df = pd.read_csv("dataset")
X_train, X_test, y_train, y_test =
train_test_split(df.iloc[:, :-1].values, df.iloc[:, -1].values)
scaler = MinMaxScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
batch_size = 64
with tf.Session() as sess:
  dataset = tf.data.Dataset.from_tensor_slices((X_train_scaled, y_train))
  dataset = dataset.cache()
  dataset = dataset.shuffle(len(X_train_scaled))
  dataset = dataset.repeat()
  dataset = dataset.batch(batch_size)
  dataset = dataset.prefetch(batch_size*10)
  iterator = dataset.make_one_shot_iterator()
  print(sess.run(iterator.get_next()[0]),sess.run(iterator.get_next()[1]))
  his = model.fit(dataset, epochs=300, steps_per_epoch=1000, verbose=0)

[[0.54192635 0.36815166 0.37738184 0.13592493]  [0.31898017 0.33687204
0.59490225 0.59597855]  [0.2733711  0.26047393 0.42761693 0.99986595]  [0.77025496 0.98919431 0.45632269 0.66447721]  [0.64305949 0.50236967
0.53823311 0.56313673]]
[429.66 460.53 428.49 446.62 456.84]

In the fitting model, the following error
AttributeError: 'PrefetchDataset' object has no attribute 'ndim'

I saw some issues with this problem.
But it didn't work for me.
Software version:
Keras:2.2.4
Tensorflow:1.12.0


texts: Optimising for Brier objective function directly gives worse Brier score than optimising with custom objective - what does it tell me?

texts:
I am training an XGBoost model and as I care the most about resulting probabilities, not classification itself I have chosen Brier score as a metric for my model, so that probabilities would be well calibrated. I tuned my hyperparameters using GridSearchCV and brier_score_loss as a metric. Here's an example of a tuning step:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=123)

model = XGBClassifier(learning_rate=0.1, n_estimators=200, gamma=0, subsample=0.8, colsample_bytree=0.8, scale_pos_weight=1, verbosity=1, seed=0)
parameters = {'max_depth': [3, 5, 7],
              'min_child_weight': [1, 3, 5]}
gs = GridSearchCV(model, parameters, scoring='brier_score_loss', n_jobs=1, cv=cv)
gs_results = gs.fit(X_train, y_train)

Finally, I train my main model with chosen hyperparameters on two ways:

optimising for custom objective - brier, using custom brier_error function as a metric

model = XGBClassifier(obj=brier, learning_rate=0.02, n_estimators=2000, max_depth=5,
                      min_child_weight=1, gamma=0.3, reg_lambda=20, subsample=1, colsample_bytree=0.6,
                          scale_pos_weight=1, seed=0, disable_default_eval_metric=1)
model1.fit(X_train, y_train, eval_metric=brier_error, eval_set=[(X_train, y_train), (X_test, y_train)],
          early_stopping_rounds=100)
y_proba1 = model1.predict_proba(X_test)[:, 1]
brier_score_loss(y_test, y_proba1) # 0.005439
roc_auc_score(y_test, y_proba1) # 0.8567

optimising for default binary:logistic and auc as an evaluation metric

model2 = XGBClassifier(learning_rate=0.02, n_estimators=2000, max_depth=5,
                      min_child_weight=1, gamma=0.3, reg_lambda=20, subsample=1, colsample_bytree=0.6,
                          scale_pos_weight=1, seed=0, disable_default_eval_metric=1)
model2.fit(X_train, y_train, eval_metric='auc', eval_set=[(X_train, y_train), (X_test, y_train)],
          early_stopping_rounds=100)
y_proba2 = model2.predict_proba(X_test)[:, 1]
brier_score_loss(y_test, y_proba2) # 0.004914
roc_auc_score(y_test, y_proba2) # 0.8721

I would expect Brier score to be lower in the model1 since we optimise directly for it, but apparently it is not the case (see results above). What does it tell me? Does optimising brier is somehow harder? Should I use more boosting rounds? (Although this was found using grid search with brier_score_loss...) Is it explainable somehow but data distribution? (e.g. such an issue can occur in the event of unbalanced classes or something like that?) I have no idea where does that situation come from, but probably there is a reason behind that.


texts:
Given a set of sequence transitions, there are different orders of hidden markov models that can be fitted to a dataset.

Is there any test to determine which is the best model for a given sequence set?
How to compare two HMM fitted on to a sequence set?


texts: What techniques should I use to compare the similarity between a bunch of texts?

texts:
If I have a list of job postings stored as raw texts and I want to compare the similarity of all the job postings to a given resume, what techniques or algorithms should I use?

I'm thinking of starting with a model that converts the job postings into a vector space using TF-IDF and then just calculate the matrix cosine similarity. Can I improve on this?


texts: What cost function and penalty are suitable for imbalanced datasets?

texts:
For an imbalanced data set, is it better to choose an L1 or L2 regularization?

Is there a cost function more suitable for imbalanced datasets to improve the model score (log_loss in particular)?


texts: k-1 vs. k categories for categorical features in decision trees

texts:
I have a model with only categorical features. The outcome is binary. I am training binary classifiers.
Most of them are coded already as 0/1 .
Some have increased cardinality (ie 4 categories). For my logistic-like models, I use Patsy, which introduces an intercept term and I have them one-hot-encoded for k-1 categories.
For decision tree-like models (adaboost, gradient boosting classifier,xgboost, etc), shall I include a separate column (1/0) for the reference category for any feature with ≥3 categories? Does it add more granularity or just overfits or decreases learning efficiency?


texts: Can we use doc2vec to detect outlier documents?

texts:
I have a set of documents and I want to identify and remove the outlier documents. I am just wondering if doc2vec can be used for this task.

Or are there any recently evolved, promising algorithms that I can use for this task?

EDIT

I am currently using a bag of words model to identify outliers.


texts:
I am creating my own implementation of a Naïve Bayes classifier. While it’s behaviour and functionalities are clear to me, my concerns are on the nature of the training and testing data.

I acquired several sets of product reviews from Amazon. The first thing I do is parsing them, that is, taking the rating (1 to 5 stars) and the text, which I parse with a regex to only contain alphabetical lowercase characters and spaces. Next, I convert ratings to polar values, so 1 and 2 stars become “-“ and 4 and 5 stars become “+”. I’m intentionally skipping reviews with 3 stars; could this be an issue?

Here come my real concerns. When using a percentage split to generate training and testing sets, should both of them contain the same share of positive and negative reviews (such as 7 positive and 7 negative reviews for training and 3 positive and 3 negative reviews for testing)? Right now I’m acquiring as many positive as negative reviews from the chosen set, but I’m wondering if that should be the case. For instance, if a set contains 7 positive reviews and 4 negative ones, I discard 3 positive reviews to equate them.

Furthermore, I observed that negative reviews tend to contain longer texts on average. So, if I’m using an equal number of positive and negative reviews, but they differ on average text length, would this have an impact on the way my classifier tries to predict?


texts:
I'm very very new to data-science, so go easy on me.
I want to incorporate an SVD based game recommendation model (using the Surprise Python Lib) for a full-stack project I'm working on, but before I jump into it, I want to make sure that I have the right data. I've scraped a bunch of data from Steam using their API, and I have three main columns: user_id, game_id, and time_spent_playing (in minutes). I figure that I can create an 'score' for the game by calculating the standard deviation for each person's playtime, so if they've spent more time playing than the average person, they have a higher score. Is this sufficient for a SVD model? Or is there a better algorithm to use given the data I have?
Bonus question: One of the approaches I've thought about doing my application is to have users rank order their favorite games, and use that as a way to get a better idea as to what their preferences are. For the initial recommendation model, I was thinking that I could arrange the games the users from the training data have based on the score described above, and then use that as the score for the training model. Any thoughts on this would also be useful.
Edit: Calculate z-score, not standard deviation


texts:
I'm working on classification problem and decided to use KNN classifier for the problem.

so if k=131 gave me auc of 0.689 and k=71 gave me auc of 0.682 what should be my ideal k?

Does choosing higher k means more usage of computational resource? if that's the case can I go with k=71. (or) should I always use K with maximum score no matter what?


texts: MCMC vs Network Theory for Big Data and Data Science applications

texts:
[Apologies if this post sounds naive, I'm fairly new to the world of data science/big data and very unsure where I'm heading career-wise]

I'm currently an undergraduate MMath [integrated master's] Mathematics student in the UK who has finished the third year of the course [out of four years].

As I have been considering the possibility of doing further research in Mathematics/Statistics/Operational Research/Data Science, I have decided to stay on and complete the Master's component of the course [as it is the only Master's course I can get funding for at this stage]. After the Master's I may continue on and do a PhD.

There are currently two projects that appeal to me that seem to have relevant applications. The first one is on improved MCMC [Markov Chain Monte Carlo] methods, in particular MCMC using Hamiltonian Dynamics. There is scope for some big data applications here.

The other project that I could take part in is one on the centrality/communities detection of networks within network science. This could possibly be useful with applications in Operational Research.

Does anyone have an idea as to which project will be more relevant to data science/analytics?


texts:
So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase  variables which the goal is to sum (in a timely fashion)

Right now I'm using a couple for loops which I now is bad practice when using pandas:

for n in users:

     #Creating an empty array to use to shuffle the new values into
     #n being the user ID number
     uservalue = [n]
     #Add independent variables
     prods = df[df["User_ID"]==n]["Product_ID"].tolist()
     uservalue.append(prods)
     for v in range (0,len(xvars)):
         uservalue.append(df[df["User_ID"]==n][xvars[v]].mean())
     #Adding total spending variables
     for v in range (0, len(yvars)):
         uservalue.append(df[df["User_ID"]==n][yvars[v]].sum())
    user = pd.Series(uservalue, userdfcols)
    userdf = userdf.append(user, ignore_index = True)

So, rather than iterating over the entire 500k entry dataframe, putting everything in a container list and then appending that list into a new row, is there a more efficient way to transform by user ID? My feeling is that I should be appending values to the row individually using pandas methods and skip the list but would that even help?

Thanks


texts: How to do a t-test on rare events distributions?

texts:
I have a dataset, which contains several features (feature_1, feature_2, ..., feature_n) and a y binary feature.
I would like to check if means of the distributions for all n features between the y==1 & y==0 groups are different
The problem is that the y==1 is really small (around 25 observations) whereas the y==0 group has around 900 observations.
Is there an alternative way to do that, apart from the t-test ?


texts: does tensorflow use opencv to covert image to numpy array

texts:
Does tensorflow use opencv to covert image to numpy array ?
How is feature extraction done in tensorflow ? How is it different from opencv model ?


texts: I would like to create a loss function that encourages the output of the embedding of an autoencoder to be dense. I don't have an explicit condition for how density is defined, but one option would be that the average distance between a point and its k nearest neighbors should be minimized, where k is some parameter. This is balanced against the reconstruction error, so the embedded points won't all converge to the same value. Is there a way to construct a loss function of this, or something similar, in a way that allows the average density around each point to be backprogated through the autoencoder?


texts: Can Shapley/Lime values be used for unsupervised learning?

texts:
One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can they be used with unsupervised learning?
Let's say we are doing anomaly detection with tabular data and we run some algorithm like Isolation Forest (or any other).
Is it conceptually right to use Shapley or Lime to try to give a local explanation of the results when using unsupervised learning?


texts: Looking for research for separating conversational audio files

texts:
I have been looking for conversational audio speech separation. Looking for researches around this topic, I came to across asteroid framework that was implemented on LibriMix dataset for audio speech separation where two voices are overlapping on each other (You can listen to such a sample here). But when we talk about conversational speech data (e.g. two people are conversing with each other) the audio of those people do not overlap with each other. Instead, people take turns in speaking and speak (Here is a sample).
While looking online for research on audio separation for conversational audio, I wasn't able to find one. Datasets like WSJ, LibriMix, etc do exist, but they are not designed for conversational problems and are more oriented towards the problem overlapping voices.
If there is any project, dataset or research around this that one knows of, it would be really helpful.


texts: Does training/fine tuning a pre-trained model on a the same dataset but with sizes scaled down (e.g., by 70%) improve inference speed? More generally, does training a CNN on smaller images improve inference speed?


texts: Is there a way to get y_pred values from saved Keras model?

texts: I have a Keras model saved in a .h5 file. As you know there are a y_pred and a y_act that confusion matrix creates from, at run time, it's easy to get y_pred values but my model is saved and now I need the y_pred values from that saved model. Is there a way to do that?


texts: Word2Vec how to treat words that aren't in the vocabulary

texts:
I'm trying to assign a vector to every word in my sentence.
Some words are not recognised even though very similar words are - for example:
Going, gone, go are recognised while goes isn't.

How should I go about assigning any logical values to the word goes or any word of this like? Please notice that I don't know in advance what words won't be recognised.


texts: How to start learning Python for Machine Learning without spending money?

texts:
I want to start learning Python for Data Science without spending money.
Where to start learning?
Which resources such as books and websites do you recommend?

Thank in advance for answer!


texts: I recall a tweet by Andrej Karpathy sometime a while ago about a ML dataset made up of cifar-10, mnist, ImageNet, and maybe others. I want to use that dataset, but I can't find it. Does anyone know if this still exists?


texts: Why this TensorFlow Transformer model has Linear output instead of Softmax?

texts:
I am checking this official TensorFlow tutorial on a Transformer model for Portuguese-English translation.
I am quite surprised that when the Transformer is created, their final output is a Dense layer with linear activation, instead of Softmax. Why is that the case? In the original paper Attention is All You Need the image is pretty clear, there is a Softmax layer just at the end (Fig.1, p. 3).
How can you justify this difference, when your task involves building a language model and your Loss is based on sparse categorical crossentropy?


texts: Early-Stopping for logistic regression. Theano

texts:
I am trying to understand the code for the logistic regression on the official documentation, but I am struggling to understand the logic behind this code:

# early-stopping parameters
patience = 5000  # look as this many examples regardless
patience_increase = 2     # wait this much longer when a new best is
                              # found
improvement_threshold = 0.995  # a relative improvement of this much is
                               # considered significant
validation_frequency = min(n_train_batches, patience/2)
                              # go through this many
                              # minibatches before checking the network
                              # on the validation set; in this case we
                              # check every epoch

best_params = None
best_validation_loss = numpy.inf
test_score = 0.
start_time = time.clock()

done_looping = False
epoch = 0
while (epoch < n_epochs) and (not done_looping):
    # Report "1" for first epoch, "n_epochs" for last epoch
    epoch = epoch + 1
    for minibatch_index in xrange(n_train_batches):

        d_loss_wrt_params = ... # compute gradient
        params -= learning_rate * d_loss_wrt_params # gradient descent

        # iteration number. We want it to start at 0.
        iter = (epoch - 1) * n_train_batches + minibatch_index
        # note that if we do `iter % validation_frequency` it will be
        # true for iter = 0 which we do not want. We want it true for
        # iter = validation_frequency - 1.
        if (iter + 1) % validation_frequency == 0:

            this_validation_loss = ... # compute zero-one loss on validation set

            if this_validation_loss < best_validation_loss:

                # improve patience if loss improvement is good enough
                if this_validation_loss < best_validation_loss * improvement_threshold:

                    patience = max(patience, iter * patience_increase)
                best_params = copy.deepcopy(params)
                best_validation_loss = this_validation_loss

        if patience <= iter:
            done_looping = True
            break

Could any one, explain to me, what do the variables: patience, patience_increase, improvement_threshold, validation_frequency, iter,  represent ?

What this condition do ?

if (iter + 1) % validation_frequency == 0:


texts: Should I load my model pkl object every time when I predict?

texts:
I am predicting the data in real time , when I hit the url it should predict but I should not load my model again and again . Can someone help to optimize the code ?
Can I load globally or how can I persist my model object session while prediction ?

@app.route("/predict",methods = ['GET'])
def predictData():
    try:
        message = request.args.get('message', default = '')
        message = message.translate(str.maketrans(string.punctuation, ' '*len(string.punctuation)))

        text_clf_nb_fit = joblib.load('UpdatedAmtAlertClassifierModel.pkl')

        result = text_clf_nb_fit.predict([message])

Thanks in advance .


texts: Does Orange allow you to change the decision threshold in modelling widgets?

texts: I can't seem to find an option with the modeling widgets for Orange to allow changes in the decision thresholds.  Is this option available?


texts: How to resolve "invalid number of 'breaks'"

texts:
Code:

 f_custdata <- gsub(".", "", f_custdata)
 f_custdata <- as.numeric(f_custdata)
 hist(f_custdata)

Error: Error in hist.default(f_custdata) : invalid number of 'breaks'

I've also tried:

  df_custdata <- gsub(".", "", df_custdata)
  df_custdata <- as.numeric(df_custdata$income)
  hist(df_custdata)

It gives me 2 errors:

Error in df_custdata$income : $ operator is invalid for atomic vectors

AND

Error in hist.default(df_custdata) : 'x' must be numeric


texts:
Please advise on starting points, research (papers,frameworks) related to thematic clustering of text. In  particular on a system with two levels of clustering where second level has a temporal nature.
Thanks!

Update:

Sorry for ambiguity in my initial question. I need to clarify, that I have experience with clustering in general and document clustering in particular, for text using TFIDF, word embeddings (word2vec, Glove and BERT Sentence embeddings) in a vector space.

My question originates from some text mentioning "two levels of thematic clustering of text where second level has a temporal nature". Just wanted to know about this technique and in particular "temporal nature" used in clustering.


texts: ARIMA forecast for timeseries is one step ahead

texts:
I'm trying to forecast timeseries with ARIMA. As you can see from the plot, the forecast is one step ahead of the expected values. I read in some other threads that this behavior is expected but how? How can I synchronize?

The code I used:

history = [x for x in train]
predictions = list()

for t in range(len(test)):
    model = ARIMA(history,order=(2, 2, 1))
    model_fit = model.fit(disp=0)
    output = model_fit.forecast(alpha=0.05)
    yhat = output[0]
    predictions.append(yhat)
    obs = test[t]
    history.append(obs)
    print('predicted=%f, expected=%f' % (yhat, obs))

rmse = sqrt(mean_squared_error(test, predictions))
print('Test RMSE: %.3f' % rmse)
# plot
plt.plot(test, color='blue')
plt.plot(predictions, color='red')
plt.show()


texts: What does 'energy' in image processing mean?

texts:
I have been going through this paper: Seam Carving for Content-Aware Image Resizing which talks about resizing images by seam carving depending on the image energy or the energy function.

Some related quotes from the paper are:

  A seam is a connected path of low energy pixels in an image

and

  A seam is an optimal 8-connected path of pixels on a single image from
  top to bottom, or left to right, where optimality is defined by an
  image energy function

So, what does the 'energy' in an image mean, in this particular context?

PS: I have gone through this question, but the answers were very broad and vague because of this reason.


texts: Aggregate several attributes into one using PCA

texts:
I am working on a soccer dataset. As we all know, soccer has a lot of different metrics related to the game. We have Key passes, Accurate Passes, Shots on Target, Saves, Clearances and so on...
I have more than 200 columns which makes it infeasible to analyze every single player based on every stat.

There are some stats that are more important to one position that the others. For instance, I want to know the Shots on Target of a forward but not for goalkeepers. So, I'm thinking in a way to group important stats per position.

My first thought was using PCA and I would like to know what you think about it. Is it possible to get the principal component for the stats that I'm interested for every position?

Example:
Goalkeepers have the following interesting stats:
- Diving Save
- Saves Inside Box
- Saves Outside Box
- Sweeper Keeper

Strikers:
- Inbox Attempts
- Key Passes
- Shots on Target

I want to reduce the dimensionality of all these important stats and have a singular stat resumes how the player of that position is performing.

Is PCA the best choice or is there a better option?

Kind Regards


texts: As Least Mean Square is a very popular choice to be used in combination with neural networks topologies, what would be the most common machine learning algorithms (and easily) to combine with Kernel Least Mean Square?


texts: Hive / Impala best practice code structuring

texts:
Coming from a DWH-background I am used to putting subqueries almost everywhere in my queries. On a Hadoop project (with Hive version 1.1.0 on Cloudera), I noticed we can forego subqueries in some cases.
It made me wonder if there are similar SQL-dialect specific differences between what is used in Hadoop SQL and what you would use in a DWH-setting. So I would like to extend this question so that people can mention what they noticed as differences between Hadoop and DWH in when structuring their queries. I noticed there was very little reference to this topic for Hadoop.
There are some normal principles that apply to Hadoop like mentioned here: https://streever.atlassian.net/wiki/spaces/HADOOP/pages/3211279/Hive+SQL+Best+Practices
It would be nice to get a few of your best practices for working with Hadoop. E.g. You write your queries as neutral as possible so that it works in both Hive and Impala avoiding using language-specific functions such as left (Impala only)
The example I came across was that a group by-query worked in two different ways. In classic RDBMS I know that only one of both would work.
Here is an example (columns are displayed in different order then in group by):
Classic SQL (Oracle, SQL Server, etc)
select t2.b, t2.a

from (select t1.a,t1.b from table1 as t1

group by t1.a, t1.b) t2

Hive SQL
select t1.b, t1.a

from table1 as t1

group by t1.a, t1.b

Notice that everything is combined in 1 single query Hadoop. The classic SQL snippet first does the group by in a subquery before the data is displayed in the right order.
I would be curious to know if you came across other subtleties like this one.
I know, based on my experience, that it is good practice to respect the following guidelines while coding:

Keep your code clean (hence avoid sub-queries if you can do without)

Consider what impact changes to settings might have. E.g. yarn.nodemanager.resource.memory-mb=24576 is great to use, but what happens if you are not allowed to change the node memory size. In an automated job it is not necessarily good practice. Look at the general Hadoop settings, e.g. file size.

Avoid using functions that are specific to a program. E.g. select left("Hello world", 3) is useful in Impala but in Hive it has to be rewritten to select left("Hello world", 1,4). This can lead to problems when later on down the line it will run in a different program.


texts: Best approach for finding longest permutation lengths between two endpoints?

texts:
Curious on the best approach or mathematical topic category so I may research involving permutations I believe, or maybe it is number theory this question pertains to?

I am interested in if datasets were sending and receiving between two different endpoints, if the values may be constantly compared for permutations and then generate a key for these values, constantly changing keys, while syncing between a receiver and sender.

A good example would be if user A was sending a message from a pre-existing data set that contains the value of countries, and sends to user B "America, China, Afghanistan".

User A would be generating a permutation key of 1 and then adding that to its data set, where User B then receives 1 and with a special key attached to it knows to add that permutation combination of countries to its data set. User B then converts the message because it knows what this is. If it is not used in that combination for a specific amount of time, it is removed.

Is this considered data science and or is this more algorithmic design? I would love to research this topic and am completely stumped on what to even categorize this as. I am a humble individual trying to learn.


texts:
I have a dataset of book reviews:

| user_id | ISBN | vote | votes_for_user | average_user_vote | ISBN_categ |
   213      3242X   4.5        12                  3.4             1
   563      1245X   3.2        74                  2.3             2

where

  vote = rating given by user to a certain book
  votes_for_user = number of votes the user has in the dataset (nr of rows)
  average_user_vote = average of a user's votes
  ISBN_categ = integer categorical of the ISBN (since that is a string).

I want to apply a clustering algorithm such as DBSCAN to see how many clusters I can form with this dataset.

My question is:
Should I apply the clustering on the dataframe as it is (minus the ISBN column) or should I construct more features for every user and construct a dataframe where every user appears only once, together with their features, and cluster that?

Remember, the intent here is to cluster users (by user_id), not data points (votes).


texts: How would I approach training a model and encoding this categorical data

texts:
So I have the following data:
I have one series where each word has a value that describes the average review score that would get.
For example, if the word "excellent" showed up in reviews with a score of 2,3,5,4 it would gain a value of 3.5.
I also have a list of the words contained in a review, and the review scores of each of those written reviews.
For example,
Unique_words
["good","clean","hotel","enjoyed","stay","here"]

score
4

(These are ofc simplified examples, my actual data is a lot longer)
I also have the original reviews, from which the unique_words are taken from.
The question is, how would I use this data in order to train a machine-learning algorithm to predict what score a review would get, given the unique words contained inside it.


texts: Does this encoder-decoder LSTM make sense for time series sequence to sequence?

texts:
TASK
given $\vec x = [x_{t=-3}, x_{t=-2}, x_{t=-1}, x_{t=0}]$
predict $\vec y = [x_{t=1}, x_{t=2}]$
Whith an LSTM encoder-decoder (seq2seq)
MODEL

NOTE: the ? symbol in the shape of the tensors refers to batch_size, following tensorflow notation...
QUESTION
Is it worth trying this architecture? (I think it took me more time to draw the picture than coding it...)
The difference with typical seq2seq is that in the decoder, the input for the second time step is not the output of the previous step. The input for both time steps in the decoder is the same, and it is an "encoded" version of the all hidden states of the encoder.


texts: Predict class of items according to structure

texts:
I have a problem where I have some items that are related in space (Ex: I have a dataset of coordinates of where eyes, mouth and nose labelled on an image of that contains a face and I want to predict given only coordinates what are eyes,nose and mouth). I'd like to create a model to learn general relation between objects and classify them.
What model should I use, what kind of problem is it?


texts: Gradient calculation of the pre-trained model

texts:
I have pre-trained tensorflow model in a graph format.  I have not used tf.gradient on the graph structure.
In such cases, is there any way to calculate the gradient of some tensors operation?


texts: python sklearn decision tree classifier feature_importances_ with feature names when using continuous values

texts:
I'm using sklearn Decision Tree Classifier with some continuous features.
When I run export_graphviz I see the same features in more than one nodes and with different values. Example:

I would like to take the top important ones and want to use feature_importances_ for that.
The problem is that feature_importances_ is array without reference to the tree nodes. I have the original features but as each one can be more than one time in the tree I'm not sure how to relate importance to node.


texts: Clustering documents - how to evaluate results?

texts:
I'm using DBSCAN clustering on a set of documents. The documents' content was converted to TF-IDF matrix, and I'd like to find consistent ways to evaluate the clusters when no added information is given (labels etc.).

Metrics comparing clusters - a score for each cluster - the goal here is to figure out which clusters are 'better' than others (more dense and more unique). Thought of two metrics:

*Intra-cluster similarity - measure the similarity of documents inside each cluster - I used the cosine distances between documents in the TF-IDF representation

*Inter-cluster similarity - measure the similarity between clusters (to highlight the more unique clusters) - similarly, planning to use the cosine distances between clusters' kernels
Metrics comparing clustering algorithms - a score for the clustering technique (such as Silhouette score). Will be used for hyper-parameter tuning and choosing an algorithm.

Any ideas?


texts: Scikit Learn Logistic Regression Memory Leak

texts:
I'm curious if anyone else has run into this. I have a data set with about 350k samples, each with 4k sparse features. The sparse fill rate is about 0.5%. The data is stored in a scipy.sparse.csr.csr_matrix object, with dtype='numpy.float64'.

I'm using this as an input to sklearn's Logistic Regression classifier. The documentation indicates that sparse CSR matrices are acceptable inputs to this classifier. However, when I train the classifier, I get extremely bad memory performance; the memory usage of my process explodes from ~150 MB to fill all the available memory and then everything grinds to a halt as memory swapping to disk takes over.

Does anyone know why this classifier might expand the sparse matrix to a dense matrix? I'm using the default parameters for the classifier at the moment, within an updated anacoda distribution. Thanks!

scipy.__version__ = '0.14.0'
sklearn.__version__ = '0.15.2'


texts:
I have a dataset I want to perform multivariate linear regression to it. The dimensions of the dataset are 832085 rows and 11 columns. The data are quite messy and given the size and my lack of experience I am confused on how to clean them.

First of all, 6 out of 11 columns have more than 277000 NA values. In that case I know that I cannot remove them because they are a big portion of the dataset so I have to impute them. Usually I would do mean substitution but I've read that this approach might create bias in the data and I wouldn't like to have that. I have tried the packages Amelia and mice in R, mice couldn't run properly and after a while it was giving me an error, Amelia was very fast and completed 5 imputations but it introduced many negative values in the dataset.

Anyone with similar experience on that?


texts: How can I model open environment in reinforcement learning?

texts:
I'm studying reinforcement learning in order to implement a kind of time series pattern analyzer such as market.

The most examples I have seen are based on the maze environment.

But in real market environment, the signal changes endlessly as time passes and I can not guess how can I model environment and states.

Another question is about buy-sell modeling.

Let's assume that the agent randomly buy at time $t$ and sell at time $t + \alpha$.

It's simple to calculate reward.
The problem is how can I model $Q$ matrix and how can I model signals between buy and sell actions.

Can you share some source code or guidance for similar situation?


texts:
I recently was introduced to a AUTO ML library based on genetic programming called tpot. Thanks to @Noah Weber. I have few questions

1) When we have AUTO ML, why do people usually spend time on Feature selection or preprocessing etc? I mean they do at-least reduce the search space/feature space

2) I mean atleast, they reduce our work to some extent and we can work from the output of AUTO ML solution and tune further if required. We don't really have to do gridsearchCV by manually keying in range of values that we might require. Right?

3) Is there any disadvantage to it? I understand it might be black box but for data analysis, don't they make it easier? Computer scientists,may not prefer it.  Ofcourse, we need to have some sort of knowledge to be able to fine tune the model, interpret the results etc

4) What's the advantage of doing manual ML when compared to AUTO ML

5) Will it be possible for us to improve the results further? I mean once we get the output from Auto ML

Can you help me understand this?


texts: Extracting "hidden" costs from financial statements using NLP

texts:
I'm designing a NLP model to extract various kinds of "hidden" expenses from 10-K and 10-Q financial statements. I've come up with about 7 different expense categories (restructuring costs, merger and acquisitions, etc.) and for each one I have a list of terms/synonyms that different companies call them. I'm new to NLP would like some advice on the best approach for extracting them.
Values are usually hidden in two different areas of the document:
Type 1: Free-form text (footnotes)
Values are nested in sentences. Here are some examples, with the Expense Type and Monetary value indicated.

Exploratory dry-hole costs were \$12.7 million, \$1.3 million, and \$1.0 million for the years ended December 31, 2012, 2011, and 2010, respectively.

2012 includes the recognition of a $3,340 million impairment charge related to the carrying value of Citi's remaining 35% interest in the Morgan Stanley Smith Barney joint venture

During the year ended December 31, 2017, we decided to discontinue the internal development of AMG 899, resulting in an impairment charge of $400 million for the IPR&D asset

Type 2: Table data
SEC statements also contain "structured" data in HTML tables. Some line items, like the first row below, correspond to the expense type I'm looking for:

Item
2020
2019
2018

impairment related to real estate assets(2):
398.2
200
0

research and development
100
200
300

other expenses
20
30
40

Correct value = 398.2

I'm thinking about a two-model approach:

Define a new NER model based off the terms I already know (e.g. "dry-hole costs", "impairment charges"). I would need to manually annotate extracts from historic statements that contain these terms for the training set.

For free-form text, it would match the sentence and pass it on for further processing (see 2).
For table data, I would loop over each row using beautifulsoup and pandas, check the first column for a match (e.g. using spaCy's comparison function), and then grab that year's value from the dataframe and finish.

For free-form matches, I still need to grab the monetary value for the correct year (sometimes multiple values are given for various years, see the first example above).

One potential problem here is that sentences like this would cause problems:

We gained $100 million this year, despite facing restructuring charges.

If the NLP algo is split into the above two-model process, model 1 would pass (because it contains a known term like "restructuring charges"), and model 2 would extract $100 million, which is incorrect because it doesn't actually correspond to the expense itself.
Is there a better solution here? As I said, I'm new to NLP and data extraction so would really appreciate any advice or resources to learn more about solving these types of key/value problems.


texts: ROC-AUC loss for GRU Model: Cannot use tflearn's loss in keras

texts:
I am trying to use tflearn.objectives.roc_auc_score as a loss function for a GRU network in Keras but I get the following error:

> ValueError: An operation has `None` for gradient. Please make sure
> that all of your ops have a gradient defined (i.e. are
> differentiable). Common ops without gradient: K.argmax, K.round,
> K.eval.

This is surprising as the implementation is apparently based on an approximation that is supposed to be differentiable.

For your reference, here is the code from the tflearn Github:

def roc_auc_score(y_pred, y_true):
    """ ROC AUC Score.
    Approximates the Area Under Curve score, using approximation based on
    the Wilcoxon-Mann-Whitney U statistic.
    Yan, L., Dodier, R., Mozer, M. C., & Wolniewicz, R. (2003).
    Optimizing Classifier Performance via an Approximation to the Wilcoxon-Mann-Whitney Statistic.
    Measures overall performance for a full range of threshold levels.
    Arguments:
        y_pred: `Tensor`. Predicted values.
        y_true: `Tensor` . Targets (labels), a probability distribution.
    """
    with tf.name_scope("RocAucScore"):

        pos = tf.boolean_mask(y_pred, tf.cast(y_true, tf.bool))
        neg = tf.boolean_mask(y_pred, ~tf.cast(y_true, tf.bool))

        pos = tf.expand_dims(pos, 0)
        neg = tf.expand_dims(neg, 1)

        # original paper suggests performance is robust to exact parameter choice
        gamma = 0.2
        p     = 3

        difference = tf.zeros_like(pos * neg) + pos - neg - gamma

        masked = tf.boolean_mask(difference, difference < 0.0)

return tf.reduce_sum(tf.pow(-masked, p))


texts: Understanding Classifier performance on text data

texts:
I am working on a multi-label text classification problem(Total target labels 90). The data distribution has a long tail and class imbalance and around 1900k records. Currently, I am working on a small sample of around 100k records with similar target distribution.I am using the OAA strategy (One against all). I have tried many algorithms on data.

Currently, each label has atleast 5000 data rows. The class imbalance is high with around 80k records for the most common label and the most rare with just one data row which I have not considered in the modelling. This dataset contains text from academic journals. It has Title and Abstract  columns.

I am using HashingVectorizer(number of features 2**20, char analyzer) to generate features and TSVD to reduce the dimensionality(n_components=200).

LinearSVC(class_weight='balanced') # Got many warnings that it could not converge. I came to know that it may due to data not scaled properly. How can I scale text data??
LogisticRegression(solver='lbfgs') # Converged very quickly
RandomForestClassifier(n_estimators=40,class_weight="balanced") # Train time ~2hr

I noticed that LinearSVC has good recall(less false negatives) while Logistic and RF has good precision(less false positives) scores. Can anyone help me in identifying the reasons behind these scores and how can I improve them.

Currently, I am not using deep learning/transformer models due to limited computation resources.


texts: How do I force specified coefficients in a Linear Regression model to be positive?

texts:
Looking for a way to do this in Python. scipy.optimize.nnls forces all coefficients to be positive.

Some additional context: I have a data frame with a some explanatory variables and a response variable. When I run a regular linear regression, the coefficients of some explanatory variables become negative. This is okay for some variables, but not for all. I want to prevent the coefficients of specific variables from becoming negative.

I want to force these coefficients to be positive because they should have a positive contribution to the response. The variables I want to have positive coefficients are investment dollars in a few different channels. The response is revenue. I don't want my model to say that investing more dollars in a certain channel lowers revenue (even if that creates a more accurate model).


texts: With a MLP (regression), is it appropriate to initialize bias in the final layer to be a value near the expected mean?

texts:
For instance, when predicting IQ in a population you would expect the mean to be 100. If you initialize the bias in the final layer you are basically giving the network a head start, telling it in what range of values it should be guessing. Another way of getting at this would be to predict a scaled outcome value with a mean of 0, which would make the standard bias initializer value of 0 be spot-on.

Are there any guiding rules/norms when it comes to this situation? Are both methods appropriate?


texts: AUC with sklearn vary each time script is started

texts:
I'm using the following code to perform a tree classification. I set up an int value for random_state in train_test_split function but each time I got different values for auc or accuracy_score.
I don't see what I am missing...

X_train, X_test, y_train, y_test = train_test_split(X, Y, random_state=1,stratify=Y, test_size=0.33)

clf = clf.fit(X_train, y_train)

predicted_probas = clf.predict_proba(X_test)
y_predict = clf.predict(X_test)

print(accuracy_score(y_test, y_predict))

print(classification_report(y_test, y_predict))

classes = np.unique(y_test)
probas = predicted_probas

fpr = {}
tpr = {}
roc_auc = {}

for i in range(len(classes)):
        fpr[i], tpr[i], _ = roc_curve(y_test, probas[:,i],pos_label=classes[i])
        roc_auc[i] = auc(fpr[i], tpr[i])
        print(classes[i])
        print(fpr[i], tpr[i])
        print("roc_auc")
        print(roc_auc[i])


texts:
I want to improve the graphic

that I created with R's ggplot2. The graphic is for a scientific publication. These are the runtime of 12 algorithms. There are four at the top, four in the middle and four at the bottom. They are not necessarily related to each other. It is very hard to discern the different colours from each other.

The main point that I want to drive home is that there are four that are slow, four that are middle fast, and four that are fast. Indeed, the groups should be more or less equally fast, and the fact that they differ is mostly due to the fact, that we talk about runtime (which always differs).

How can I improve the plot such that I drive home the main point without being distracted by the many colour, but possibly still have differentiation within the three groups?

Maybe it is not smart to keep the differentiation in the first place?

I would like to keep the line plot, but I am open to alternative suggestions if there are really good ideas out there.


texts: suitable algorithms for very high dimensional ”binary pattern" recognition?

texts:
I have a dataset consisting of 3000 binary features and one binary ouput. subset of these binary features form binary patterns. these subsets could be neighbouring features or from different regions (which one is the case is unknown). also these patterns can show up anywhere within the feature vector.
any ideas what algorithms are best suited for a problem like this? I have done some searching and seem like logical neural networks, CNNs and HMMs be good candidates. also feature reduction is something I am currently looking at. data points are not images but rather 1D arrays.


texts: Determining Confidence Probabilities for Distribution

texts:
I have a LSTM neural network that predicts the number of items required in each room for the next time step. So for example for room A, B, C and D the model predicts

A -> 3 items
B -> 4 items
C -> 2 items
D -> 2 items

where all the items are assumed to be of the same type. The problem I have is how to distribute the items across the rooms when the prediction asks for more items then I have available.
Is there an approach you would suggest where I would be able to determine the distribution based on the confidence level of the number of items in each room? As an example, using the predicted distribution for the 'rooms' above where the total number of items predicted to be required are 11:
A -> probability 1 item: 0.9,
2 items: 0.8,
3 items: 0.3
B -> probability 1 item: 0.9,
2 items: 0.8,
3 items: 0.8,
4 items: 0.3
C -> probability 1 item: 0.7,
2 items: 0.2
D -> probability 1 item: 0.7,
2 items: 0.7
if I only have 10 items it would be possible to identify that C has the least probability of needing it thus I would be able to distribute as follows:

A -> 3 items
B -> 4 items
C -> 1 item
D -> 2 items

Any suggestions or references would be of great help.
Thanks in advance.


texts: I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.


texts:
A client would like to sort out his filesystem (~ 1,000,000,000 files), which has been fed by numerous workers over the years, each with their own unknown naming convention, e.g.:

[DATE]-[CLIENT]-[FILENAME]
[TYPE]-[CLIENT].[DATE][FILENAME]
...

Here are four examples (out of ~1,000,000,000 files) to make things clearer:

JPM_TPD0001662_2009124012302000451.pdf

JPMF_STA_1712010832_18001_LUX_approval.pdf

CHACN05CTRP_00111.001.pdf

CHACN63CJO1_00018.001.pdf

The purpose is to find out patterns in the naming conventions, but I can't use regular expressions, since the conventions are unknown a priori.

I was wondering whether there was a kind of clustering algorithm to be able to group files according to their naming conventions.

Any K-Mean philosophy applied to strings?


texts: What does linearly decreasing training and accuracy loss means?

texts:
I am trying to train an LSTM autoencoder for sentence embeddings:-

Embedding size = 600
LSTM Hidden size = 600
Vocab Len = 3972
Number of LSTM layers = 1
Bidirectional = True

Here the architecture of the encoder is:-

Embedding layer mapping 3972 nodes to 600
LSTM taking the whole sequence and giving output and state of shape (2*num_layers, 600)

This output and state is then used to set state and hidden state of a decoder, whose architecture is:-

Embedding layer mapping 3972 nodes to 600
LSTM taking the whole sequence and giving output and state of shape (2*num_layers, 600)
Softmax(linear_layer mapping 600 to 3972)

I have tried many different architectures and most have been just an overfitting failure. Although, training the model with this config, here is the loss vs epoch and accuracy vs epoch :

This article, although says that if the training loss and validation loss are decreasing linearly then the model is underfitting. Still, there is some confusion as the same article says that it is underfitting coz of :-

1) The model complexity is not able to learn the data and the model should be increased in complexity(increase the layers/nodes per layer)

2) The training is stopped prematurely.

My concern is:-
1) the loss graph is not linear exactly, so is it underfitting? if it is, then,

2) Should I continue training or should I change the architecture?


texts: Conditional Statement to update columns based on range

texts:
I have a dataframe called clients that has 5000+ rows with the following:

ID   Ordered   Days
1     101565    131
2     202546    122
3     459863    78
4     328453    327
5     458975    -27

I'm trying to create a loop that looks at the numbers of days and replace them with a new columns if it does meet the days critetaria based on the
following:

Days           NEW_COLUMNS
0-119          0-3 Months
120-209        4-6 Months
210-299        7-9 Months
300+           10+ Months
-280 to -196   Reach out clients
-195 to -104   Send promotion
-103 to -1     Close case
< -280         Plan

I have the following code but hasn't worked so far:

if(days <-280)(NEW_COLUMNS ="plan")
else if (days>-280 && days <-196)(NEW_COLUMNS=" Reach out clients";)
else if (days>-195 && days<-104)(NEW_COLUMNS =" Send promotion";)
else if (days>-103 && days <-1)(NEW_COLUMNS="Close case";)
else if(days> 0 && days <119)(NEW_COLUMNS="0-3 Months";)
else if(days > 120 && days <209)(NEW_COLUMNS="4-6 Mos";)
else if(days > 210 && day s<299)(NEW_COLUMNS="7-9 Mos";)
else if(days > 300)(NEW_COLUMNS="10+ Mos";)

Eventually I want a table that looks like this:

ID   Ordered   Days    New_Columns
1     101565    131     4-6 Months
2     202546    122     4-6 Months
3     459863    78      0-3 Months
4     328453    327     10+ Months
5     458975    -27     Close case


texts: In which epoch should i stop the training to avoid overfitting

texts:
I'm working on an age estimation project trying to classify a given face in a predefined age range. For that purpose I'm training a deep NN using the keras library. The accuracy for the training and the validation sets is shown in the graph below:

As you can see the validation accuracy keeps rising with smaller steps than the training accuracy. Should i stop training at the epoch 280 in which the training and the validation accuracy have the same value or should i proceed the training process as long as the validation accuracy is rising, even thought the training accuracy value is also getting at overfitted values (eg. 93%).


texts: Extracting Part of Speech (Source and Destinations) using text mining/NLP?

texts:
I need to extract the source and destination terms from the text
 documents using text mining / NLP / Information Retrieval ?

Example input:

  I am travelling from New York to London.
  I am heading towards playground from home.
  I will be going to Sweden from Boston.
  I was flying from School to Home.

The output can be as follows :

S. No. |  source    | Destination
------ |  ----------|------------
      1| New York   | London
      2| playground | home
      3| Sweden     | Boston
      4| School     | Home


texts: Predicting quality results from operating data

texts:
Background: I have process data (table 1) that is "batch" in the chemical engineering sense of the word. Each batch ID represents the start and end of a run. Throughout the batch, different process parameters are either adjusted or they vary as a result of the chemistry going on in the tank.
At the end of a batch, a sample is taken of the product and analyzed in a lab. There is only one row per batch ID of results (table 2), while the operating data is sampled every minute.
Goal: There are years worth of data for these batches and I'm trying to determine what the optimal conditions are to reached the desired quality results. I'd also like to have a machine learning model where I can input parameters that might not be exactly identical to one of the previous runs, and have a decent prediction of the product quality.
In the example below, these quality results would be purity and pH (table 2). Ideally, I would have a purity of 99% and a pH of 4.
What I've researched thus far: I've read a few papers that address this with statistical methods (principal component analysis or partial least squares regression). Since there are many rows of operating data for each batch ID but only one quality row, they shape the data such that each variable at each time t is a column. This creates a table where each batch can be represented by a row (table 3). I'm going to continue working through this method, but I wanted to know if this could be a machine learning application.
I've been reading about neural networks (RNN, ANN, LSTMs), but I haven't been able to find an example that is similar to this. For applications of machine learning involving time series, it seems that they are meant to forecast a future value or predict a value within the time of a dataset with regression. I have multiple inputs and multiple outputs, but the examples I've seen seem to be situations where x1 and x2 affect y1 and y2 in the same plot. In my example below, var1 and var2 are both inputs and they might not necessarily be indictive of quality on their own if you looked at them separately. Also, in reality there are probably 10 or so variables that are inputs.
What I'm after is more of an optimization model that looks at all of the process parameters and predicts the kind of quality I would get from running the batch with a particular set of conditions. I tried to look at this as a classification problem to join the quality data with the time series process data, but I don't think that works. Could I possibly take the quality data and turn it into ranges so that I can apply a classification solution? For example, instead of targeting a pH of 4, I can create a category where pH is between 3.5 and 4.5.
Any recommendations as to what I should look into?
Process data (table 1):

ID
time
var1
var2

1
0
65
21

1
1
66
20

1
2
68
23

2
0
65
24

2
1
70
24

3
0
65
26

3
1
66
27

3
2
66
27

3
3
66
28

End of trial quality results (table 2):

ID
purity
pH

1
98%
4

2
97%
4.5

3
99%
3.9

Batch-wise unfolding (table 3)

ID
t = 0, var1
t = 1, var1
t = 2, var1
t = 0, var2
t = 1, var2
t = 2, var2
purity
pH

1

2

3


texts:
I got it that when using time series data, I have to use a RNN

The highlight here is that the neurons receive their output again as an input, so they can take into account the previous step.

But what I do not understand is: Why cannot regular NNs do the same? For example, when I provide an order like

0, 1, 2, 3, 4, 5, 6, 7, ..

the NN should register that there is a pattern in it. If these numbers represent days or seconds, it is not important.
So why are self-looped neurons now necessary to do so?


texts: Creating dataset - imbalanced or balanced?

texts:
I'm trying to make an image classification model and I have 5 classes - A, B, C, D, E. The goal is to get the highest possible classification accuracy.
I have a database of images and I'm selecting the number of images I will use for each class for my model. I'm trying to figure out how many images I should pick for each class if the distribution of available data is as shown below.
Should I randomly choose something like 8,000 images for each class, in order to avoid class imbalance? Or should I just use as many images per class?
Images available per class:

A - 100,000
B - 70,000
C - 40,000
D - 10,000
E - 8,000


texts: Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?

texts:
I'm trying to understand the policy gradient approach for solving the cartpole problem.
In this approach, we're expressing the gradient of the loss w.r.t each parameter of our policy as an expectation of the sum of gradients of our policy gradient for all actions in a sequence, weighted by the sum of discounted rewards in that sequence:

$$\nabla_\theta L(\theta) = E[ G(S_{0:T}, A_{0:T})\sum_{t=0}^{T}\nabla_\theta log\pi_\theta (A_t|S_t) ]$$

and we estimate it using an empirical average across all samples in an episode - which makes sense intuitively.

BUT the less intuitive part is that I saw a common practice to normalize advantages in between episodes in several implementations (and indeed it works better). So after they calculate the they wouldn't directly use the advantage, but rather would normalize it, e.g. here they do after every episode:

discounted_epr = discount_rewards(epr)
discounted_epr -= np.mean(discounted_epr)
discounted_epr /= np.std(discounted_epr)

what's the justification for that - both in theory and in intuition? It seems to me that if an episode is long and as such has large advantages, it's worth learning more from that episode than from a 3 moves episode. What am I missing?


texts:
I have a table of values like this one:

Age
Population size
# of individuals who like apples
# of individuals who like oranges

Children
100
88
73

Adult
50
32
25

I am looking for a way to decompose each cell value into the sum (or product, or other simple operation) of 4 factors:

the population size
the intrinsic attractiveness of each fruit
the propensity of each age group to like fruits in general
the remaining variation that cannot be explained by any of the previous 3 factors

Just looking at the numbers, we can see that apples are more popular than oranges (regardless of age), that children tend to like fruits (regardless of the fruit) more than adults. I would like to quantify these characteristics of the data set.


texts: Merging two datasets with unclear identifier in Excel

texts:
I'm trying to merge two datasets on Mergers & Acquisitions. They both consist of c.10'000 observations with c.50-100 variables each. One contains information about the actual M&A deal whereas the other one contains info on how a deal was financed.

The problem is that there is no clear and unique identifier. For example, I could use the date that the deal was announced but that wouldn't be unique because on some days 10 deals were announced. Using company names is difficult since they mostly aren't identical in both datasets. For example if in one dataset I find "Ebay", in the other the same company could be called "eBay", "Ebay Inc", or "Ebay, Inc."

I've been working with the Fuzzy Lookup add-on for Excel, as well as concacenating various identifiers that are not unique but in their combination become useful (e.g. Date & Country & SIC Industry Classification Code, etc.). However I haven't been able to generate as many matches as I would have hoped.

I'd be grateful for any ideas or pointers towards resources that would help me merge the datasets more efficiently.


texts: How can I get prediction for only one instance in Keras?

texts:
When I request Keras to apply prediction with a fitted model to a new dataset without label like this:

model1.predict_classes(X_test)

it works fine. But when I try to make prediction for only one row, it fails:

model1.predict_classes(X_test[10])

Exception: Error when checking : expected dense_input_6 to have shape (None, 784) but got array with shape (784, 1)

I wonder, why?


texts: Muti-Output Decision tree with classification and regression in output

texts:
I'm working on a project and want to use decisions tree (because I have both catgorical and numerical values in my input and don't want to transform the categoricals variables) to predict an output varaible , but the problem is that I don't have only one output variable but 4 !

that picture is a look for my output dataset:

The first one will can be predict with a regression decision tree .

the the rest if can be predict with a classification decision tree but with more than 2 classes so a multi-class.
So I want to know what is the best solution to approach this problem:

Using 4 differents decisions trees one tree for each variables ?

Or using 1 multi-output decision tree?

What are the avantages and inconvenients of each approach?


texts: What Kind of Network should I be considering?

texts:
was hoping someone can guide me in the right direction with this problem.

I have a dataset that contains the history of an experiments that lasts 40 minutes; taking a measurement every minute. I have historical data for about 600 runs. So I have a total of 24000 rows in my dataset.

I would like to set up a model in where I can train it on the different multiple runs where all 40 minutes are available; and each minute has same feature count of 11.

My plan then was to feed the model anywhere from 2-35 rows of live data, and I would like it to predict one of the values at the 40 minute mark.

Any help would be greatly appreciated.


texts:
I have a dataframe of X values and one with the y values in what I think is the proper format. I took the single feature y and converted it to dummies so it's binary and is m rows by n features.

from sklearn.model_selection import train_test_split

X = df.drop('target',axis=1)
y = pd.get_dummies(df['target'], sparse=True, drop_first=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=99, train_size=0.60

I've been trying to figure out how exactly the data should look from the documentation but am not having much luck.

I first tried the following:

import skmultilearn.problem_transform import BinaryRelevance
from sklearn.naive_bayes import GaussianNB

classifier = BinaryRelevance(GaussianNB())
classifier.fit(X_train, y_train)

where X, y train sets are pandas dataframes. This results in the following error:

TypeError: no suppoerted conversion for types: (dytype('O'),)

Then I converted to matrices:

X_train = X_train.asmatrix()
y_train = y_train.asmatrix()

I received the same error. Then I tried

X_train = np.matrix(X_train)
y_train = np.matrix(y_train)

Same error again. Is there an easy way to go from dataframe to proper format for skmultilearn models?


texts: Variation in output of Logistic Regression when using SMOTE

texts:
I am working on a logistic regression case with an imbalance in the target variable. To fix this I am using SMOTE (Synthetic Minority Oversampling Technique), but each time I run my regression model, I get different numbers in my confusion matrix. I have set random_state parameters while invoking SMOTE as well as Logistic Regression still to no avail. Even my features are the same in each iteration. I was able to get the best value for recall as 0.81 and AUC value as 0.916 once but they are not coming anymore. On some occasions, the value of False Positives and False Negatives shoots up very much indicating that the classifier is very bad.

Please guide what I am doing wrong here, below is the code snippet.

# Feature Selection
features = [ 'FEMALE','MALE','SINGLE','UNDER_WEIGHT','OBESE','PROFESSION_ANYS',
            'PROFESSION_PROF_UNKNOWN']

# Set X and Y Variables
X5 = dataframe[features]

# Target variable
Y5 = dataframe['PURCHASE']

# Splitting using SMOTE
from imblearn.over_sampling import SMOTE
os = SMOTE(random_state = 4)

X5_train, X5_test, Y5_train, Y5_test = train_test_split(X5,Y5, test_size=0.20)
os_data_X5,os_data_Y5 = os.fit_sample(X5_train, Y5_train)
columns = X5_train.columns

os_data_X5 = pd.DataFrame(data = os_data_X5, columns = columns )
os_data_Y5 = pd.DataFrame(data = os_data_Y5, columns = ['PURCHASE'])

# Instantiate Logistic Regression model (using the default parameters)
logreg_5 = LogisticRegression(random_state = 4, penalty='l1', class_weight = 'balanced')

# Fit the model with train data
logreg_5.fit(os_data_X5,os_data_Y5)

# Make predictions on test data set
Y5_pred = logreg_5.predict(X5_test)

# Make Confusion Matrix to compare results against actual values
cnf_matrix = metrics.confusion_matrix(Y5_test, Y5_pred)
cnf_matrix


texts: Is there a built-in pyspark function for the paradigm union(filter, take)?

texts:
Essentially, filtering by key, taking a specific number (based on key) from the DataFrame, and unioning these results for a number of values.

For example,

| First Name    | Last Name     |
|------------   |-----------    |
| Jim           | Pratchett     |
| Jim           | Adams         |
| Karen         | Gaiman        |
| Karen         | Bryson        |
| Karen         | Amis          |
| Steve         | Wodehouse     |
| Steve         | Hiassen       |
| Steve         | Heller        |
| Steve         | Toole         |
| Steve         | Sedaris       |

Let's say my filtering dictionary is as follows -

{
    'Jim': 1,
    'Karen': 2,
    'Steve': 3,
}

So I want 1 Jim, 2 Karens and 3 Steves from my DataFrame. I'm not frightfully picky about which ones I end up with. I figured that the standard way to do it would be to filter and take for each key, value pair, and then union all the DataFrames.

I was wondering if there is a) a better way to approach this problem or b) a built in function to address the union(filter, take) paradigm I mentioned earlier.


texts: Does nearest neighbour make any sense with t-SNE?

texts:
Answers on here have stated that the dimensions in t-SNE are meaningless, and that the distances between points are not a measure of similarity.

However, can we say anything about a point based on it's nearest neighbours in t-SNE space? This answer to why points that are exactly the same are not clustered suggests the ratio of distances between points is similar between lower and higher dimensional representations.

For example, the image below shows t-SNE on one of my datasets (15 classes).

Can I say that cro 479 (top right) is an outlier? Is fra 1353 (bottom left) is more similar to cir 375 than the other images in the fra class, etc? Or could these just be artefacts, e.g. fra 1353 got stuck on the other side of a few clusters and couldn't force its way through to the other fra class?


texts: split flattened tensor to perform class wise loss

texts:
I would like to adapt my dice loss function for multi-class averaging. y_true and y_pred are one-hot 3D images (mask of label). For example, if the flattened tensors contain the X classes sequentially, I would like to split them into X equal lengths, calculate their respective losses and average them. Note that it need to work in batch.
Here is my code;
def dice_coef(y_true, y_pred, smooth=1e-10):
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)

    # here I would like to split y_true_f and y_pred_f in X sub tensor
    # of equal length. The following code would then be performed on each
    # sub tensor and averaged to output the mean-class wise loss.

    intersection = K.sum(y_true_f * y_pred_f)
    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)

def dice_coef_loss(y_true, y_pred):
    return 1 - dice_coef(y_true, y_pred)

How could I implement that ?


texts: Is it useful to consider a categorical features into a `polynomial` or `linear model`?

texts:
In Titanic dataset, i have a dataset containing Categorical features (such as Cabin, Embark and Sex).

I need to build a linear or polynomial model with multiple variables to predict Survive. I wish to consider Categorical features as well. Let's say, i can replace all categorical values with a respected numerical value. For example about Sex column, I can replace: 1 with male and 2 with female

My question is:

Is it useful to consider a categorical features into a polynomial or linear model?


texts: What do you think of Data Science certifications?

texts:
I've now seen two data science certification programs - the John Hopkins one available at Coursera and the Cloudera one.

I'm sure there are others out there.

The John Hopkins set of classes is focused on R as a toolset, but covers a range of topics:

R Programming
cleaning and obtaining data
Data Analysis
Reproducible Research
Statistical Inference
Regression Models
Machine Learning
Developing Data Products
And what looks to be a Project based completion task similar to Cloudera's Data Science Challenge

The Cloudera program looks thin on the surface, but looks to answer the two important questions - "Do you know the tools", "Can you apply the tools in the real world".  Their program consists of:

Introduction to Data Science
Data Science Essentials Exam
Data Science Challenge (a real world data science project scenario)

I am not looking for a recommendation on a program or a quality comparison.

I am curious about other certifications out there, the topics they cover, and how seriously DS certifications are viewed at this point by the community.

EDIT: These are all great answers.  I'm choosing the correct answer by votes.


texts: ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

texts:
X = train_encoded_df.iloc[:, 1: ]
y = train_encoded_df["Loan_Status"]

print("Precision:",metrics.precision_score(y_test, y_pred))

My training data contains the categorical features encoded using get_dummies().

This is causing the error:

> ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='U1')

How to fix this?


texts: How to estimate the marginal distribution of a class with respect to one predictor in a classification task?

texts:
I have a dataset with a binary dependent variable $y \in \{0,1\}$ and a set of predictors $x1,x2,..,t$. Here, $t$ is the time in minutes (in 24 hrs, that is $t \in (0,1440)$). I want to estimate the marginal probability distribution of $y$ with respect to $t$.
I am approaching this problem as a binary classification task and creating a classifier with all $x1,x2,..,t$ predictors, then, am planning to vary $t$ between 0 to 1440 and try to estimate the probabilities.
Will this work? Is there any efficient method to do this? Also, please suggest some machine learning/deep learning algorithms for this task? I am betting on RF, Xgboost, Deep Neural networks. I have 84k records. (Note that I want to estimate $P(y=1|\{x1..t_i\}$, probability of y=1 as time increases over the day)


texts: How can there be more true positive than positive?

texts:
Currently reading  Learning from Little: Comparison of Classifiers Given Little Training

In 3 Experiment Results, the following graph is shared:

The experiment is described as follow

  We begin by examining an example set of results for the average TP10
  performance over all benchmark tasks, where the training set has P=5
  positives and N=200 negatives.We vary the number of features selected
  along the logarithmic x-axis.

I understand this as "we use a training set of 205 elements, 5 being positives and the 200 remaining being negatives".
But looking at the results, Naive Bayes using Information Gain with a few features, and Multinomial Naive Bayes using Bi-normal separation with several hundred features  both end up with 6.5 true positives in the top 10 (The TP10 metric is the number of true positives found in the 10 test cases that are predicted most strongly by the classifier to be positive).

I would have assumed that the models being closest to 5/10 would be the most accurate, but reading their results, it looks like the higher the better.
Thus, it feels like I overlooked and misunderstood something. Can somebody enlighten me on this issue ?

Thank you


texts: Meaningful Information retrieval and question answering for unstructured data - Is it even possible?

texts:
Hello good NLP people,

I am working on a task that gradually seems not solvable for me. My data-set consists of long, messy, unstructured documents (pdfs, doc, docx, scans with tables, graphs, text, etc)  and the client wants to obtain a system that allows to query basically as much information as possible from the documents.

An example query could be: “Who are the beneficiaries of project xxx that is implemented by ORG xxx?”.
Or: “How much co-financing was directed to projects that concern focal areas x,y,z?”

My initial idea was following:

1) Process documents (Python, OCR, etc) into machine readable form.

2) Pre-process/clean/normalise text.

3) Manually annotate entities and build customised NER model.

4) Manually annotate entity relations and build Relation extraction model (I am not sure how to do this the best way?)

5) Extract triples.

6) Store triples in knowledge graph/base.

7) Query (however the query will then be limited by the extracted entities and relations).

The end result could look like this:
http://semanticparsing.ukp.informatik.tu-darmstadt.de:5000/question-answering/static/index.html

However, I have the feeling that QA systems and Relation Extraction frameworks can be built on top of large quality datasets (like Wikipedia - or even better KB like Wikidata). In reality, dealing for example with messy documents, it seems very difficult to replicate.

Please let me know about any important work on this lately, or how you would proceed.
Thanks!


texts:
I'm trying to do a majority voting of the predictions of two deep learning models.The shape of both y_pred and vgg16_y_pred are (200,1) and type 'int64'.

max_voting_pred = np.array([])
for i in range(0,len(X_test)):
    max_voting_pred = np.append(max_voting_pred, statistics.mode([y_pred[i], vgg16_y_pred[i]]))

I run into the following error:

TypeError: unhashable type: 'numpy.ndarray'

How should I pass the data?


texts: Dataset with features as SNOWMED codes and outputs as names of medications prescribed.What are some ML algorithms that suit this case?

texts:
So, i am constructing a dataset (from doctor prescription notes and diagnosis) where

X (input features) represent the various snowmed codes for illnesses/surgeries performed/chronic conditions etc.

Y is a set of pills prescribed for X.

Presently I do not have this data in a matrix format. For ex. i have the snowmed codes for an illness that reads as follows: "Diabetes, Hypertension, Cardiac arrest, Left MCA stroke". Y is the set of pills that the doctor noted down for the above.

As one can note, the above data is not in a matrix format.
There are many snowmed codes and various different pills overall. I'd like to apply ML algorithms to predict the pills for a given set of snowmed codes about the illness.

How to organize this data in a format convenient for ML? I am open to other analytics approaches as well. Any discussion in this matter will be helpful.


texts: centrifugal pump vibration data predictive model

texts: I am a beginner in data analysis. I have vibration data of 5 centrifugal pumps in .csv format and I want to make a model which predict the error using vibration data. Can anyone guide me, how to start and which algorithm will be helpful to detect the error? Thanks in advance for your help and suggestions.


texts: Error calculation in Logistic Regression

texts:
Suppose we have a linear regression model that predicts an item’s price. If the item’s prediction is 8 USD and the actual value is 10 USD, then it is clear that the error is pow(10-8, 2)=4.
But how is the error calculated when there are more than two classes?

For example, the model that predicts the numbers trained on the MNIST dataset. In this case we have 10 labels that mark the actual values - range between 0-9. But if we use the sigmoid function for the activation then the range for possible predicted values is between 0-1. Right? How can we compare these values if they are on a different scale? For example, the sigmoid function outputs the values 0.5 and we have to compare it to 3?


texts: How to normalize test data according to the training data if the normalization on the training data is performed row wise?

texts:
I read on several places about the normalization of features in the machine learning method. But I normalize my training data row-wise as shown in the following code. I showed only two samples of training data. My question is that while performing the normalization on test data, should I choose the minimum and maximum value of each test sample to normalize each test data or should I uses minimum and maximum values from the training data?
As an explanation in first row -3 is one feature, -2 is second 0 is third and 3 is fifth feature. And second row is second sample comprising of 5 features from -4 to 2. Similar to all other machine learning algorithms each row correspons to one sample consisting of 5 features.

data = np.array([[-3,-2,0, 2,3],[-4,-1,0,3,2]])
print(data)

print(data.shape)
for i in range(len(data)):
    print("i: ",i)
    old_range = np.amax(data[i]) - np.amin(data[i])
    new_range = 2
    new_min = -1
    data_norm = ((data[i] - np.amin(data[i])) / old_range)*new_range +
new_min
print(data_norm)

Result

[-1.         -0.66666667  0.          0.66666667  1.        ]
[-1.         -0.14285714  0.14285714  1.          0.71428571]


texts: Is recommendable look for high variance when your data is imbalance?

texts:
Hello I have a dataset with the following classes A, B, C, and this classes have the following representation of the dataset 60% 39% 1%.
Is it a good idea try to get a model with high variance in this cases?
Thanks


texts: What could be the impacts on the intercept and slops of each predictor in Logistic Regression by rare event cases?

texts: I am wondering how data set with rare event cases would impact on my Logistic Regression model in terms of the intercept and slopes of each predictor?


texts:
I want to do a voice command classification without using speech to text or cloud technologies. This is a followup of question spoken-utterance-classification-on-ravdess-using-mfcc which did not get much attention.
Using cloud speech to text is easy but dont want to stay offline. Current on prem speech to text solutions do not have decent performance.
Would anyone like to suggest some implementations which actually are working.


texts: How to predict unknown unknowns in machine learning

texts: I am dealing with a problem about classifying bird species through analysing MFCCs. I already built a dataset with 13 MFCCs for two kinds of birds. And I trained the data with Naive Bayes & KNN model. However, when I tried the model with prediction of third bird species, it is classified as the one of the two species. I am wondering how can I achieve to predict unknown species as unknowns? And I know my existing classification model may not work. So, what kind of model might be helpful? Does SSL useful in my case? Or treat these unknowns as outliers? But how can that be applied in MFCC?


texts: What are the output shape of LSTM with "return_sequences" equal to "True" or "False"?

texts: What are the output shape of LSTM with Keras implementation when "return_sequences" equals to "True" or "False" ?


texts: Platt Scaling vs Isotonic Regression for reliability curve

texts:
I am learning classifier probability calibrations and have calibrated an eleastic net model using both Platt scaling and isotonic regression. As you can see in the attached image Platt scaling (on the bottom) better approximates the diagonal line compared to isotonic regression (top), however I noticed I am losing information with any predictions where the predicted probability <0.4, I have seen this happen in uncalibrated plots as well. Therefore I am wondering which calibration method I should be using. Furthermore, generally what about the model/data causes the curve to be cut like this and are there things I can do to address this during modeling?

Many thanks!

Update: per suggestion included a plot of the uncalibrated probabilities for comparison purposes.


texts:
I have a set of datapoints from the unit interval (i.e. 1-dimensional dataset with numerical values). I receive some additional datapoints online, and moreover the value of some datapoints might change dynamically. I'm looking for an ideal clustering algorithm which can handle these issues efficiently.
I know sequential k-means clustering copes with the addition of new instances, and I suppose with minor modification it can work with dynamic instance values (i.e. first taking the modified instance from the respective cluster, then updating the mean of the cluster and finally giving the modified instance as an input to the algorithm just as the addition of an unseen instance).
My concern with using the k-means algorithm is the requirement of supplying the number of clusters as an input. I know that they beat other clustering algorithms (GAs, MSTs, Hierarchical Methods etc.) in time&space complexity. Honestly I'm not sure, but maybe I can get away with using one of the aforementioned algorithms. Even that my datasets are relatively large, the existence of a single dimension makes me wonder.
More specifically a typical test case of mine would contain about 10K-200K 1-dimensional datapoints. I would like to complete the clustering preferably under a second. The dynamic changes in the value points are assumed to be smooth, i.e. relatively small. Thus being able to use existing solutions (i.e. being able to continue clustering on the existing one when a value is changed or new one is added) is highly preferred.
So all in all:

Can you think of an algorithm which will provide a sweet spot between computational efficiency and the accuracy of clusters wrt. the problem defined above?
Are there some nice heuristics for the k-means algorithm to automatically compute the value of K beforehand?


texts: What can help decrease outliers' influence on non-tree models?

texts:
I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models:

StandardScaler
Apply rank transform to the features
Apply np.log1p(x) transform to the data
MinMaxScaler
Winsorization

I wasn't able to imagine any other ... I guess that's all?


texts:
Given a correlation matrix M and after an event, M changes.
What is the best way to analyse the change? Verify significance (presumably T-test, ANOVA..)
Currently looking at the %, count & average of strong + & - correlation coefficients. Running T-tests and ANOVA.
Thanks!


texts: How to determine the correct target for classification probability when the observed samples are probabilities of each class?

texts:
I have data in which each event's outcome can be described by a probability of a categorical occurrence. For example, if all of the possible class outcomes are A, B, C, or D suppose in one event 7/10 people selected category A, 2/10 selected category B, and 1/10 selected C.
The goal of the model is to predict the probability of each class as close to the observed truth as possible. However, it is unclear how to determine the proper target. Here are the options I've thought about doing:

The target becomes the predominant class selected. That is, in the event described above A=1, B=0,
C=0, D=0.

Additional classes/categories are created and then become bins of all possible observed
probabilities. For example, the event above would become A=1 when Pr(A) >= 0.7.

I THINK option 1 is the the best as option 2 would result in a lot of categories and spread. However, when considering verification of the model output, i.e. Y = {A=0.9, B=0.1, C=0, D=0} for each sample would it not be advantageous to use the actual observed probabilities? Are there other options for the target?


texts: Why use sum and not average for sentiment analysis?

texts:
In Sentiment Analysis, I've seen the sentiment of an entire string of words (tweets, paragraphs, sentences, etc.) computed by summing the sentiment of all of the terms in the string.

I know there is a monotonic relationship between summing and averaging.  So, if sentences are only ever compared to other sentences, then of course it wouldn't make a difference.

So, why is it that the average is not used instead of the sum?  Especially if there are cases where strings could contain only one word, or many words, it seems to me that an average sentiment would be the way to go.  Is that not true?


texts: Why are only 3-4 features important in my random forest?

texts:
I am running a random forest regression with Python's Scikit-Learn, code's below (X - features, y - to be predicted).
# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 1)

# Scaling
from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test  = sc_X.transform(X_test)

# Random forest
from sklearn.ensemble import RandomForestRegressor
rf =RandomForestRegressor(max_depth=2, n_estimators = 100, random_state=0)
rf = rf.fit(X_train,y_train)
pred_train = rf.predict(X_train)
pred_test = rf.predict(X_test)

I am running this code for randomly sampled 100k dataset, that has 60+ features. Each time when I check feature importance I get 3 to 4 variables as important (with one of them holding over 80% of importance), and others' importance is set to 0. It is not reasonable to me that only these are important for prediction and the rest is rubbish.
var_num = X_train.shape[1]
plt.barh(range(var_num), rf.feature_importances_, align='center')
plt.yticks(np.arange(var_num), variable_names)
plt.xlabel('Variable Importance')
plt.ylabel('Variable')
plt.show()

Is it possible that I am missing something? That some other parameters needed to be defined? Could this be caused by a high correlation between variables themselves? Or is it really that the rest of my features are useless..?


texts: Where can I get pre-trained weights for character recognition?

texts: I'm doing a simple pipeline and I need a digit recognition algorithm. There are definitely pre-trained weights out there for character recognition, but I can't find them. They can be in Keras, Tensorflow, or Pytorch.


texts: How does ML Clustering put to a practical real-world use?

texts:
Newbie alert to data science and ML.
I'm learning Supervised and Unsupervised learning at the moment and Supervised learning is easy to digest and I can relate to a lot of practical use cases. Unsupervised learning is where I just couldn't correlate to a the real-world use cases (although I found numerous of quotes where people say they use it for Customer Segmentation, Fraud Detection etc).

For argument sake, I'll just quote a sample taken from one of the MS Azure Studio examples.

The dataset contains Countries and their average protein intakes in various forms of food).

When this dataset is run through a KMeans algorithm, it creates 3 clusters and fits the country names in these clusters.

So in this specific example what is the problem I'm trying to solve?

Am I looking to find similar countries based on protein intake habits?

Am I creating groups based on the given dataset and then there is a human intelligence to qualify these groups (or clusters) to say "Vegetarian Rich Countries", "Red Meat Rich Countries" etc. Then when a new country comes we predict whether this country falls in which cluster?

In this case, there is an intermediate human intelligence is needed in the workflow which requires labelling the cluster (as opposed to labelling each datapoint in classification). Is this a correct understanding?


texts:
Let’s say I have tens of thousands of datasets (rows) consisting each of 300 columns of integer, double and strings (no image, no audio). Five of these target columns represent interesting resulting properties (integer) of the product that is represented by of each dataset (row) that I want to predict.

My current idea is to create classes for each target column and combine these classes: column1 has classes 1-5, column2 has also class 1-5… so I get combined target classes in the form “11111”, “15312”, “43151”, “55555”. Now I have only one target-“label” for each dataset (row) that could be the output of an ANN.
This approach has at least two disadvantages:

some combined classes have only a small number of datasets and
the binning of each column in only five classes decreases the precision.

If I increase the number classes of each column, I get even a smaller number of datasets in some combined classes.

So my question is if I should go on with the approach or are there better ideas? It is possible to use ANN to predict several properties (i.e. classes) at the same time?


texts: I have data that have been grouped in 27 groups by different criteria. The reason for these groupings is to show that each group has a different behavior. However, I would like to normalize everything to the same scale. For example, I want to normalize to a 0-1 scale or 0-100, that way I could say something like 43rd percentile and it would have the same meaning across groups. If I were to just, say, standardize each individually by subtracting the mean of each and dividing by standard deviation work? Would I have to calculate the mean/st. dev of all of the combined data or do each of the 27 groups individually?


texts: Cosine Similarity for Ratings Recommendations? Why use it?

texts:
Lets say I have a database of users who rate different products on a scale of 1-5. Our recommendation engine recommends products to users based on the preferences of other users who are highly similar. My first approach to finding similar users was to use Cosine Similarity, and just treat user ratings as vector components. The main problem with this approach is that it just measures vector angles and doesn't take rating scale or magnitude into consideration.

My question is this:

Are there any drawbacks to just using the percentage difference between the vector components of two vectors as a measure of similarity? What disadvantages, if any, would I encounter if I used that method, instead of Cosine Similarity or Euclidean Distance?

For Example, why not just do this:

n = 5 stars
a = (1,4,4)
b = (2,3,4)

similarity(a,b) = 1 - ( (|1-2|/5) + (|4-3|/5) + (|4-4|/5) ) / 3 = .86667

Instead of Cosine Similarity :

a = (1,4,4)
b = (2,3,4)

CosSimilarity(a,b) =
(1*2)+(4*3)+(4*4) / sqrt( (1^2)+(4^2)+(4^2) ) * sqrt( (2^2)+(3^2)+(4^2) ) = .9697


texts: scipy.spatial.distance.mahalanobis return null values for some vectors in python

texts: I am using scipy.spatial.distance.mahalanobis to calculate distance between two vectors but i'm getting null values for some vector I don't know why? null value is possible?