texts: which NN should I use for Time-series dataset, whose pattern change as time goes

texts:
I am analyzing a time-series dataset using (supervised) tensorflow deep learning. The tensorflow code is given a series of inputs, and based on each input, the NN has to predict output value in near future.

For training there are lots of pre-labeled input/output of past. Each input is an array of state per time step, and each input is labeled with a result value. So far this kind of task is common at deep learning, and I used multiple GRU layers to solve this.

But problem is, I found that the pattern for finding appropriate output actually changes as time goes. For every 10 hours (approximately), the pattern changes so at every start of each pattern period (like first 1-2hours of total 10 hours) the NN is bad at predicting, but after that, prediction rate enhances. Plus, I assume the data also has some noise too.

So far my implementation with GRUs do its job, but I want to find better way to build NN if possible. I currently know some basic supervised learning techniques and some advanced like LSTM, and for unsupervised learning I know DQN and policy-gradient.


texts:
I currently have collected a dataset of different social media posts for each user with labels assigned to each user. I tried to use LSTM, and BERT for the text classification problem, So for each post, I try to predict the label(for example age). This is not sufficient because you need all of the information contained in the sum of posts to determine the user's age for example.
My first thought was to concatenate all of the posts for a single user but since I am currently using BERT which has a maximum sequence length of 512 it wouldn't work. My second idea was to use a text summary and combine them in one vector and hope it doesn't pass the maximum length limit.

Do you have any suggestions for a possible solution? I would assume this problem has been dealt with in the scientific literature and I would be thankful if anyone could point me in the right direction.


texts: Activity recognition in smart homes with different sources

texts:
I have a dataset collected in a smart home and I would like to do Activity Recognition.
I am used to classification but in this case data comes from 4 different sources:

Accelerometer from wrist
Binary sensors around the house
proximity beacons (to understand which room the user is at any moment)
ground pressure from floor

Data is labelled and has 25 activities.

At the moment I am working with a CNN on the activity images coming from the sensors and I can reach 45% accuracy, but I need to add the other sources to achieve better results.

My questions:

how to use all (or a group of) different sources for example in a NN?
Can I train 4 different classifier that works together to make a prediction?
Maybe some classifiers try to understand simpler activities and other try to find more complex activities?


texts: To map topic to a document after topic modeling is done with LDA

texts: Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and segregating it into appropriate cluster. Any link, code example, paper will greatly be appreciated.


texts: Machine Learning methods for finding outliers

texts:
I have a csv files of thousands of lines, the data is put down into a Dataframe of columns.
Some of the column have text information while others might have numbers.
I want to detect anomalies or outliers inside this csv file.

Being relatively new to this and not knowing much I would appreciate if I get some help or a small guidance and how I should tackle this problem. What methods or what is the best approach to find outliers in a Dataframe.

My language of choice is Python


texts: SVM - Shuffle image data before GridSearchCV or not?

texts:
I have different image datasets, most of them are sorted by class, others are already mixed.
For each of these data sets, I would like to train one SVM (in Python with Scikit-Learn), whereby in each case the hyperparameters are previously optimized using GridSearchCV.
Some of the ordered data sets still need to be split into training and test data, which would then be in mixed form after using train_test_split and, to my understanding, would not have to be mixed again before using GridSearchCV.

Other ordered data sets are already split into training and test data, so I would directly apply GridSearchCV. Now I have found that StratifiedKFold does not mix the data before splitting into batches, if I understood correctly. In those cases, I should mix the data before using GridSearchCV, right?

If I am mistaken in my assumptions, I would be grateful for any hint.


texts: Given is the result of the model performance. Help me with this MCQ

texts:
You also evaluate your model on the test set, and find the following:

Human-level performance 0.1%
Training set error  2.0%
Dev set error   2.1%
Test set error  7.0%
What does this mean? (Check the two best options.)

You have underfit to the dev set.
You should get a bigger test set.
You should try to get a bigger dev set.
You have overfit to the dev set.


texts: Given a t-SNE plot, how can I infer the "most correct" labels? How does one understand its structure?

texts:
Let's say I begin with an exceptionally large dataframe (e.g. imported/munged from tsv files). Several of these columns are categorical labels.

(As a more concrete example, let's imagine a group of students in a school district, pre-school to high-school).

Now, I begin using sklearn and instantiate a t-SNE model, similar to the example here:

http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html

 import numpy as np
 from sklearn.manifold import TSNE
 X  # my data
 model = TSNE(n_components=2, random_state=0)
 np.set_printoptions(suppress=True)
 model.fit_transform(X)

and then we plot this. The plot might look something like this: http://imgur.com/a/3amkJ

Here's my problem: with real datasets, after using t-sne to learn/cluster, you will have a number of "clusters". Then, using the categorical labels, I try to go through each of these, and try to figure out what structure the t-SNE plot is giving me.

For our school example, I'd get the t-SNE output, then I would label the datapoints. (Let's assume that the clusters are actually representative of age/classroom, e.g. the first-graders group together, the second-graders are a group, etc.)

If I try to color this plot with "grades", I'll see that the grades does not really explain the structure of this plot. (Why? Because every class-level has students with As, Bs, Cs, etc.) Then I might try height...that does pretty good (because there's a correlation between short students--> pre-school, tall students --> high school seniors).

How does one use a t-SNE plot to infer the "most correct" labels of the data? How does one use t-SNE plots to explain (and further explore) the plot structure?


texts:
I have tried GridSearchCV and BayesSearchCV for tuning my LightGBM algorithm (for binary classification).

I have used 10 iterations and I have indicated scoring ="roc_auc"

In the first iteration, I have got:

best score (e.g :0.71...)
and best param (e.g: max-depth: 10 , learning-rate: 0.17..., num-leave:175, n-estimators: 176, ....)

In the 10th iteration, I have got :

best score (e.g :0.72...)
and best param (e.g: max-depth: 9 , learning-rate: 0.19..., num-leave:168, n-estimators: 172, ....)

Then I tried to train my LightGBM classifier with the 10th param (which supposed that it get the best score!!). I have got :

AUC : (0.7541.., 0.6467..)
Accuracy: 0.7338..
RMSE: 0.5216..

Then because I had some curiosity I have tried to train my classifier with (best param) of the First iteration (which is considered as worst score)!. I had surprised by the result that I have got:

AUC : (0.7545., 0.6592..)
Accuracy: 0.7332..
RMSE: 0.5152..

Because I have fixed previously scoring by roc-auc, Generally, I should get AUC in the 10th iteration better than the first iteration but I have got the contrary.

I have supposed that it is considered AUC train of 10th iteration 0.7541 as better than 0.7545 of the 1st iteration because of the overfitting but when I tried to check the 3rd and the 5th iteration I get on the 3rd : 0.7532   and in the 5th : 0.7548.

So I don't know what the best score in this algorithm means. And what is its role exactly if it gets values as the described situation. I had tried then many times with other tuning parameters but I have got the same case. I don't know where is the problem exactly.

My source code :

# X and y data are splited to 10 fold Cross- validation

import lightgbm as lgb
from skopt import BayesSearchCV
from sklearn.model_selection import StratifiedKFold

ITERATIONS =10
lghtboost = lgb.LGBMClassifier(silent=False, n_jobs=7, verbose=0,  metric=['roc_auc'])

param_grid ={'max_depth': [6, 15], 'learning_rate': [0.1,0.2], 'num_leaves': [150, 210],
                       'n_estimators': [100,180], 'min_child_samples': [5], 'colsample_bytree': [1] }

bayes_cv_tuner_lgh = BayesSearchCV(
        lghtboost,
        param_grid,
        n_iter=ITERATIONS,
        scoring = 'roc_auc',
        cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0),
        refit = True,
        random_state = 0,
        verbose=0,
        n_jobs=7
        )

def status_print(optim_result):
     print('Best ROC-AUC: {}'.format(bayes_cv_tuner_lgh.best_score_))
     print('params = {}'.format(bayes_cv_tuner_lgh.best_params_))

result_Lightboost = bayes_cv_tuner_lgh.fit(X_train.values, y_train.values, callback=status_print)

d_train = lgb.Dataset(X_train, label=y_train)

## I press the last best parameter:
params = {max-depth: 9 , learning-rate: 0.19..., num-leave:168, n-estimators: 172, ....}

model_lgh = lgb.train(params, d_train)
y_pred = model_lgh.predict(X_test)

def auc_LGBM(m, train, test):
    return (metrics.roc_auc_score(y_train,m.predict(train)), metrics.roc_auc_score(y_test,m.predict(test)))

print('AUC : ', auc_LGBM(model_lgh, X_train, X_test))
print("Accuracy : ", metrics.accuracy_score(y_test, y_pred.round()))
print("RMSE : ", np.sqrt(metrics.mean_squared_error(y_test, y_pred.round())))


texts: What is the difference between these two methods of batching with stateful LSTM

texts:
With stateful LSTM, are these two methods effectively the same:

for _ in range(3):
    batch_input_shape = [1,10,2]
    x_batch, y_batch = batcher()
    model.train_on_batch(x_batch, y_batch)
model.reset_states()

and

batch_input_shape = [3,10,2]
x_batch, y_batch = batcher()
model.train_on_batch(x_batch, y_batch)
model.reset_states()

i.e. the first method 30 rows of data spread across 3 separate batches of 10 sequences and reset_states() called after all 3 train_on_batch() calls. The second method 30 rows of data submitted in a batch of 3 with 10 sequences, and reset_states() called immediately after.

Assuming it all the same data, and putting aside training times (first method will take longer) is the end result the same?


texts: Creating the Yolo Training datasets from Scrapped video as training model

texts:
I have gathered the 200 car type video for one car type and model Auto detection with YOLO (Accessible by this link), and converted the video of folders to photo by this colab page shown here:

So i needs to detect every car type folder by default YOLO weight and label the *.txt result file to the folder car type name, I have this Colab Page which could detect objects for video by TensorFlow object_detection model like show below which could be used for this purpose:

But i am not so familiar with that part of work and searching for some sample code in GitHub or google colab to do it faster.
Thanks for you r attention.


texts: Intuition behind using non-hypercubic kernels in density estimation

texts:
Suppose that we perform density estimation in m-dimensional space: we estimate the value $p(a)$ for some point $a$ given observations $\{x_1, \dots, x_n \}$.

It is known that if region $A \subset \mathbb{R}^m$ is "small" enough to consider density being constant on points from $A$ then we can make the following estimate:
$$ p(a) \approx \frac{k / n}{|A|} $$
where $k$ is the number of observations that lie in $A$ and $|A|$ is Lebesgue measure of $A$.

Let parameter $h$ be small enough to consider density as constant inside hypercube centered at $a$ with side length equal to $h$. The volume of this hypercube is equal to $h^m$ and point $x$ lies inside this hypercube iff $K(\frac{x-a}{h}) = 1$ where
$$K(u) =\cases {
      1\text{, if $|\frac{u^k - a^k}{h}| \leq \frac{1}{2}, k = 1,\dots,m$}\cr
      0\text{, otherwise}
}$$
It's easy to see that the number of observations inside this hypercube equals to
$$k = \sum_{i = 1}^{n} K(\frac{x-a}{h})$$
and so the estimation described above gets the following form:
$$p(a) \approx \frac{1}{n h^m} \sum_{i = 1}^{n} K(\frac{x-a}{h}) $$

We can interpret $K$ as "weight" given to particular observations and one of the drawbacks of hypercubic approach is that all observations lying inside hypercube have equal weights despite having different distances from $a$. Yet another drawback is that the resulting estimate is not continuous. That's what i understand to be the main reason of using non-hypercubic kernels such as gaussian kernel which give more weight to points close to $a$ and yields continuous estimate.

But i have troubles with interpreting the usage of such kernels. The sum $\sum_{i = 1}^{n} K(\frac{x-a}{h})$ is no longer equal to $k$ so we can't justify the usage of these kernels by formula $p(a) \approx \frac{k / n}{|A|} $. Finally here are my questions: how do we justify the usage of smooth kernels? how can one interpret this usage?

Thank you for any ideas.


texts: Custom convolution kernels for dot-product between features in a feature map

texts:
I am trying to implement Interaction and Aggregation block from this paper: paper, and I faced difficulties with SIA Module.

As I understand the paper, I need to:

get KxK patch for each (h, w) position in original feature map F. In other words, get KxK patches with stride=1 from feature map. So I should get tensor with shape (BATCH, C, K, K, L), where L is the number of patches and it may be in other dimension, it doesn't matter.
reshape (BATCH, C, K, K, L -> BATCH, C, K*K, L), so each patch is a 2d matrix with shape (C, K*K)
then get resulting matrix by

  "accumulating the dot-products between features of corresponding
  positions, and then normalizing across all spatial positions in F with
  softmax":

So p_ik and p_jk (position k in patches Pi and Pj) are tensors with shape (BATCH, C, 1). Result of their dot-product has shape (BATCH, 1). OR this operation may be performed as a convolution operation, where the whole patch Pi is a kernel applied on feature map F.

So, if KxK kernel usually has shape (K, K, C, S), where C is the number of input channels (number of channels in input feature map) and S is he number of output channels (number of channels in resulting feature map), then we can represent our patches as a kernel with shape (K, K, C, L), where L is a number of patches.

My questions:

Am I right thinking that matrix Sk - result of accumulated and normalized dot-product between features in F for patch size KxK - has shape (BATCH, H*W, H*W, 1)? It has only one channel?
Am I right thinking that dot-product accumulation can be implemented as a convolution operation with custom non-trainable kernels, which are sets of L KxK patches? If yes, what to do with batch_size dimension? Is there a way to work with the whole batch? I will have different kernels for each feature map in batch..


texts: Require Guidance - Identifying risk factors

texts:
I have a dataset of patients who have been inpatients (admitted to hospital) and not admitted (but visited as outpatients). Class proportion is 66:34.

I have collected a list of features for all these patients.

Now my objective is to find/identify the risk factors that leads to hospital admission? Meaning what are the risk factors that can influence a patient to be admitted? How the risk factors are different between two classes? For example patient with High heart rate or some sensitive clinical parameter(just example) could get admitted whereas person with normal clinical paramters may not get admitted but visit only for consulation.

Can you confirm whether my steps below are right?

1) I have two classes (Admitted & Not-admitted)
2) Around 25 input variables
3) Run a logistic regression (Statsmodel logit or Scikit-learn?)
 Do we always have to predict the outcome class to know the risk factors that lead to admission/hospitalization?
5) Then identify the significant risk factors based on p-value.

Though my objective is to identify the risk factors that leads to hospital admission, do I still have to predict the outcome class to know the risk factors?

Can you guide me on this?


texts: Neural network back propagation gradient descent calculus

texts:
So I've drawn a neural network diagram below:

where $x_1, x_2,\ldots,x_m$ are the input layer, $h_1, h_2$ are the hidden layer and $\hat y_1, \hat y_2,\ldots \hat y_k$ are the output layer. In the $W^h_{im}$ notation, it represents the weight, where $i$ is representing to which node it's pointing to in the hidden layer, which is either $h_1$ or $h_2$ and $m$ is representing from which input node.

For example, if $W^h_{11}$, this means it is the weight represented by the line connecting $x_1$ node to $h_1$ node. $W^o_{ki}$, where $k$ is the output node and $i$ is the hidden layer node. Therefore, $W^o_{11}$ is the arrow connecting $h_1$ node with $\hat y_1$ node.

I'm currently working on the back propagation process of updating the gradient descent equation for $W^h_{11}$. Before this, I've only learnt to deal with a diagram with only one output node, but now it's multiple output nodes instead.

So I've attempted the beginning part of the calculus, where I'm not entirely sure if $\hat y$ equation i wrote below is correct and also for the loss function?

For $W^h_{11}$:

$Input: (x,y)$

$\hat{y} = forward(x)$

$\hat{y} = \sum ^K_{k=1} h_1W^o_{k1} + \sum ^K_{k=1} h_2W^o_{k2}$ *am i doing this correctly?

$h_1 = \sigma(\bar{h}_1)$ where $\bar{h}_1 = \sum_{j=1}^{M} x_jW^h_{1j}$

$Loss function: J_t(w) = \frac{1}{2}\sum(\hat{y}-y)^2$ where $\hat y$ is a vector of $\hat y_1,\hat y_2,..\hat y_k$ *would the loss function be included with a summation?

Would like to know if I've done the beginning part right.


texts:
Suppose I have a response variable y and and a set of feature variables (x1, x2 ... xn). I wish to find which of x1...xn are the best features for y in a regression problem (the relationship might not be linear).
Is there any way I can do this kind of feature selection without using any correlation measure or regression function in the process (i.e. I cannot use any filter or wrapper methods)?


texts: Modeling and Predicting Co-Occuring Values

texts:
I have data for a 100,000 people with different personality traits. Here would be sample data:

Person  Trait-1   Trait-2   Trait-3 .....   Trait-N
John     1          0         1                1

I need a model where for a new user when I see Trait-X, I need a prediction of the likelihood of all other Traits i.e. how likely is he to have any of the other traits. Can someone point me to a possible model I could use? I am a novice so don't know much about this space.


texts: How to get the formulas used by seasonal_decompose for Trend and Seasonality

texts:
I'm trying to use decomposition to forecast into the future.  From my reading I understand that I can do this by adding a trend formula to a seasonality formula.  I know that I can decompose a time series with this:

import statsmodels.api as sm
result = sm.tsa.seasonal_decompose(series.values, model='additive',freq=12)

trend = result.trend
seasonal = result.seasonal
residual = result.resid

The trend, seasonal and residual variables are arrays of numbers.  I've searched google and the documentation for seasonal_decompose and haven't found a way to see/get access to the formulas used to calculate the numbers for trend and seasonal.

From my understanding, I need those formulas in order to be able to make projections.  Do you know of a way to get those formulas from seasonal_decompose?  Is there another function or method that works better for this?


texts: Logic Check: Building a SKLearn Pipeline

texts:
I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set.

Background:

I have a sparse, high-dimensional data-set (370x1000) with a continuous variable as the target. At present, I have been running a random forest regression on all the features with a 90/10 split, followed by parameter tuning via grid search on the training set, followed by 5-fold cross validation on the optimized model (with the entire dataset).

Problems with this approach:

As I understand the situation, there are a number of things I am doing that might be harming the model and introducing undesired bias. Specifically, my concerns are:

As I am tuning parameters only on the initial train/test split, I am not accounting for other split combinations that arise during the K-Fold CV. Might optimal settings for fold 1 be different for fold 2? Intuitively I would assume so.
I am not doing any feature selection that might remove otherwise redundant features and shorten my feature-space (I know RF is generally quite good with high-dimensional spaces but I would still like to try). Some suggestions I have read have included removing features with very low variance. But I find myself in the same conundrum as above: if I remove low variance features from the original training set I am not accounting for other combinations during K-fold. Comparatively, if I remove all low-var features prior to splitting the data I am surely leaking info between the train/test states.
An alternative approach I have seen is recursive feature selection (with CV as per SKLearn) - this looks promising, as I think it means that it will partition the data-set in folds and conduct RFE on each fold, presumably giving me an averaged score of the best number of features to keep.

My possible solution:

I have been doing some reading around Pipelines in SKLearn and think that might be the way to go. My understanding is that an advantage of a pipeline is that i can stack transforms together, preserving the individual folds and allowing me to address the problems I have detailed above. What I am considering, and what I would appreciate sense-checking from anyone with more experience, is the following:

As the data-set is small, I would not split the data-set in the conventional train/test split manner, but would use K-Fold across the whole data-set.
Run a RF (using default params) with K-Fold to get a baseline level of performance.
Create a pipeline whereby I (3.1) create folds, (3.2) and then within each fold find the optimal number of features to keep, (3.3) tune hyper-parameters for that fold, and finally (3.4) predict the values in the test fold.

As you might be able to tell I am struggling to get to grips with what order things should go in, and whether step 3 is actually what a Pipeline does. If someone can provide pointers/recommendations/corrections it would be appreciated.


texts:
So i don't find the problem in my code. I just train my model, and after that i just want to use for classification.
Model:
model = tf.keras.models.Sequential(
        [
            tf.keras.Input(shape=(200, 256, 3)),
            self.__data_augmentation(),
            tf.keras.layers.Conv2D(filters=16, kernel_size=(3, 3), activation='relu'),
            tf.keras.layers.BatchNormalization(),
            tf.keras.layers.MaxPool2D(pool_size=2, strides=2),

            tf.keras.layers.Conv2D(filters=32, kernel_size=(3, 3), activation='relu'),
            tf.keras.layers.BatchNormalization(),
            tf.keras.layers.MaxPool2D(pool_size=2, strides=2),

            tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'),
            tf.keras.layers.BatchNormalization(),
            tf.keras.layers.MaxPool2D(pool_size=2, strides=2),

            tf.keras.layers.Conv2D(filters=128, kernel_size=(3, 3), activation='relu'),
            tf.keras.layers.BatchNormalization(),
            tf.keras.layers.MaxPool2D(pool_size=2, strides=2),

            tf.keras.layers.Dropout(rate=0.2),
            tf.keras.layers.BatchNormalization(),
            tf.keras.layers.Flatten(),

            tf.keras.layers.Dense(units=1024, activation="relu"),
            tf.keras.layers.Dense(units=64, activation="relu"),
            tf.keras.layers.Dense(
                units=len(self.__get_datasets()['train_dataset'].class_indices), activation="softmax")
        ])
    model.summary()
    model.compile(loss=self.MODEL_LOSS, optimizer=self.MODEL_OPTIMIZER, metrics=['accuracy'])

    callback = ValidationAccuracyCallback(threshold=self.VALIDATION_ACCURACY)

    model.fit = model.fit(
        self.__get_datasets()['train_dataset'],
        epochs=self.EPOCHS,
        validation_data=self.__get_datasets()['validation_dataset'],
        batch_size=self.BATCH_SIZE, class_weight=self.__get_weight_of_classes(),
        callbacks=callback)

And here is the code where i want to use it:
# Grab a screenshot and resize it
frame = cv2.imread(filename=file_path))
frame = resize_image(frame, 200, 256)

# Scale the pixel values to [0, 1]
frame = frame.astype("float") / 255.0

# Flatten the screenshot
frame = frame.flatten()
frame = frame.reshape((1, 200, 256, 3))

# Make a prediction on the image
predictions = model.predict(frame)

And i got this error message:
ValueError: Input 0 of layer sequential_1 is incompatible with the layer: expected ndim=4, found ndim=3. Full shape received: (None, 256, 3)


texts: Hi for a project I am representing movies in a vectoral format. One thing I want to ask is can I use both one hot encoding and multi hot encoding together on my representation. There are limited number of genres but there are thousands of directors which I would like to represent with multi hot encoding instead of one hot encoding to reduce the dimension of my vector. Addition issue related to multi hot encoding will not be an issue as I am only using movies with 1 director. Some movies are of more than 1 genre which will be added together in one hot encoding method without any problem. What can you say about this?


texts: Using LSTM for binary text Classification, getting almost same accuracy at each epoch

texts:
I am doing Twitter sentiment classification. For that I am using LSTM with pretrained 50d GloVe word embeddings(not training them as of now, might do in future).

The tweets are of variable lengths ranging from 1 to 250. Hence I sorted the tweets and divided them into batches of almost similar length. Further I zero padded them with maxLen equal to the highest length of tweet in that particular batch, also in embedding layer I have set 'mask_zero = True'.

I am using adam optimizer with default values, and fit generator to train the model.

The first issue I observed was that for each epoch initially the accuracy starts increasing and goes till 80, but  when its around halfway through the training set the accuracy starts decreasing and fall to 74 point something. This happens for each epoch.

To solve this I shuffled the batches and then fed them to model, as I thought the model might be adjusting itself to smaller length tweets as those are being fed to it first and hence not generalizing to longer tweets.

After doing this, the accuracy randomly fluctuates and again lands on 74 something at the end of each epoch.

I don't know why I am getting stuck at local minima, ANY HELP IS MUCH APPRECIATED...:)

I will attach some code if required...


texts: Interpretation of coefficient on interactive effects term in regression analysis

texts:
I have two continuous variables: $X1$ and $X2$, both have a positive correlation on the dependent variable $y$ (continuous). I found that the interaction term $(X1*X2)$ is statistically significant for $p\lt(0.00)$ when added to the model. The final model is the following:
$$y=aX_1+bX_2-c(X_1*X_2)$$
Where $a$, $b$, and $c$ are the regression coefficients. However, I am wondering why the regression coefficient for $c$ of the interaction term is negative? What does it mean?
If I only add the interaction term $(X1*X2)$ to the model why is the coefficient positive when if I add it with $(X1)$ and $(X2)$ the regression coefficient becomes negative? It doesn't make sense!


texts: Imbalanced classification or Regression? What is the best approach to my A/B testing related problem?

texts:
The context of the problem is A/B testing of two new versions of a game. I have a structured dataset (50000 rows x 22 columns) from the game designers that represents data with respect to two versions of the game: A and B. The rows represent individual players, and there are several categorical features (c*) and continuous features (n*). One of the important columns is the monetization (in-game purchases) obtained from the players. The end goal is to be able to recommend a version A or B to each player, such that the monetization would be maximized.
raw_df_columns: 'id', 'c1', 'c2', 'c3', 'c4', 'c5', 'c6', 'game_version', 'n1', 'n2',
'n3', 'n4', 'n5', 'n6', 'n7', 'n8', 'n9', 'n10', 'n11', 'n12', 'n13',
'monetization'

Issues:
The most important problem is that the version of the game (A or B) in the 'game_version' column is not the end label. It is simply the version that was issued to the person. I do not have the required labels directly to model it as a classification problem.
Therefore my initial approach to this was to attempt it as a regression problem. I did an EDA, eliminated some non-essential features and tried to predict 'monetization' using the rest of the features (except 'id' which has no correlation). The prediction accuracy is very good, but the problem is that the feature 'game_version' is more or less being ignored in this prediction. Or with some ML models it seems to have a very simplistic relationship, which is wrong.
Finally, in most cases A is better than B (has more monetization than the best of B), except for a small minority (~ 190 cases out of 50000) where B is better than A. This also makes it very difficult for the ML algorithms to zero in on such cases.
Ask:
What is the best approach that I should take for this problem? How do I obtain the recommended version of the game for a given player?
I can also provide additional details or clarifications if needed. Thanks in advance!


texts: How to justify the usage of 200 dimensions in word vectors instead of the 300 dimensions?

texts:
When employing machine learning methods in NLP, most of studies use 200 or 300 dimensional vectors. 300 dimensional embeddings carry more information and this, therefore, is considered to produce better performance results in general.

If you have unlimited computational resources and the training time is not a problem for you, when does it make sense to use 200 dimensional embeddings instead of 300 dimensional vectors in a classification problem (e.g. in sentiment analysis) and why?

I am assuming that you are using off-the-shelf word2vec, GloVe, or other pretrained vectors. That is, the vectors are not being learnt from scratch in your classification task.


texts: Can features passed into Conv1d layers be randomized?

texts: If I have input time series data shape as such, X.shape = (batch_size, 50, 5), it means that the data has 5 "features", each having 50 time steps. Does passing data like this into a Conv1d layer mean that the 5 features have some kind of spatial relationship with each other, like how a pixel in an image is more closely related to the pixels nearby around it than further away? Is there a way to remove this feature relationship when dealing with time series data like this? As in, even if the order of the features were randomized, the network would still learn from the patterns seen in 1d space of each feature?


texts: Time Series and Statistical Learning methods differences

texts:
I am trying to understand the Time Series and Forecasting methods.

I have this basic theoretical question about this:

Why "Time Series" is NOT included on the Contents of, for example, the book "The Elements of Statistical Learning" Tibshirani - Hastie or so many others similar books?

There are a lot of specific books for Time Series, but isn't it an Statistical Learning method?


texts: Linear discriminant analysis in R: how to choose the most suitable model?

texts:
The data set vaso in the robustbase library summarizes the vasoconstriction (or not) of subjects’ fingers along with their breathing volumes and rates.
> head(vaso)
 Volume  Rate Y
1   3.70 0.825 1
2   3.50 1.090 1
3   1.25 2.500 1
4   0.75 1.500 1
5   0.80 3.200 1
6   0.70 3.500 1

I want to perform a linear discriminant analysis in R to see how well these distinguish between the two
groups.
And I consider two cases:
ld <- lda(Y ~ ., data=vaso)

ld1 <- lda(Y ~ log(Volume)+log(Rate), data=vaso)

Please help me understand which model is better? What characteristics to look at?


texts:
I have a dataset and I'm trying to predict the label for my sample but I couldn't map it since that case never showed here is my sample (I'm using naïve Bayesian method)

X=(Age=middle, has_job=false, own_house=true, credit_rating=good)

and that's it the dataset

What I'm supposed to do to fix the problem?
I know that I should  avoid it but didn't know how


texts: Word embedding of a new word which was not in training

texts:
Let's say I trained a Skip-Gram model (Word2Vec) for my vocabulary of size 10,000.
The representation allows me to reduce the dimension from 10,000 (one-hot-encoding) to 100 (size of hidden layer of the neural network).

Now suppose I have a word in my test set which was not in my training vocabulary. What is a reasonable representation of the word in the space of dimension 100?
For me, it seems that I cannot use the neural network I trained to come up with the word embeddings.


texts:
I have a simple question. Please see the below screenshot :

It is from a midterm exam from a university : https://cedar.buffalo.edu/~srihari/CSE555/exams/midterm-solution-2006.pdf
My questions is how the means are postive ? I am asking because the class samples are all negative so I would expect that the mean is also negative ?


texts: Custom loss function with both min(y, p) and max(y,p)

texts:
I'm creating a neural network in tensorflow and need to minimize the following loss function:
$\frac{max(y,p)}{min(y,p)}$ Where $y$ represents the true value and $p$ the predicted value. Since the loss function is not differentiable it becomes a problem when using gradient descent.
Update, now I'm trying to implement:
$p > y \rightarrow loss = log(p)-log(y)$
$p < y \rightarrow loss = log(y)-log(p)$
$p = y \rightarrow loss = log(y)-log(y) = log(p)-log(p)$
This is my code:
def custom_loss(y_true, pred):
    if pred > y_true:
        custom_loss = K.log(pred) - K.log(y_true)
    elif pred < y_true:
        custom_loss = K.log(y_true) - K.log(pred)
    else:
        custom_loss = K.log(pred)-K.log(pred)

    return custom_loss

if __name__ == '__main__':
    run = 1

    X = np.load("vectors_normal_2way.npy")

    with open("target2.pickle", "rb") as file:
        target_dict = pickle.load(file)

    target_strings = [*target_dict]

    Y = np.array([])

    for target_value in target_strings:
        Y = np.append(Y, target_dict.get(target_value))

    for i in range(run):
        X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2)
        model = get_model()
        start = time.time()
        model.fit(X_train, Y_train, verbose = 1, epochs = 1, batch_size = 32)
        end_time = time.time()-start
        pred = model.predict(X_test)[:, 0]

But I get the following error:
tensorflow.python.framework.errors_impl.InvalidArgumentError:  The second input must be a scalar, but it has shape [32]
     [[{{node gradient_tape/custom_loss/cond/StatelessIf/gradient_tape/custom_loss/weighted_loss/Mul/_17}}]] [Op:__inference_train_function_1040]

The code can run when I have batch_size = 1


texts:
Powerful techniques are sometimes rediscovered by various disciplines at different points in time, because the particular scientific fields do not overlap or interact.

A couple of ML researchers have pointed out, that the back-propagation (Rumelhart et al, 1986) algorithm (a solution to credit-assignment) has been rediscovered by many fields.

In what other fields is backprop used? Under which names and in which context?


texts: Techniques for trend extraction from unbalanced panel data

texts:
My data set is formatted like this:

User-id | Threat_score
aaa       45
bbb       32
ccc       20

The list contains the top 100 users with the highest threat scores. I generate such a list monthly and store each month's list in its own file.

There are three things I would like to get from this data:

1. Users who are consistently showing up in this list

2. Users who are consistently showing up in this list with high threat scores

3. Users whose threat scores are increasing very quickly

I am thinking a visual summary would be nice; each month (somehow) decide which users I want to plot on a graph of historic threat scores.

Are there any known visualization techniques that deal with similar requirements?

How should I be transforming my current data to achieve what I am looking for?


texts: How/What Correlation for Position Ranking?

texts:
I'm newbie to Data Science. I'm trying to understand how to correlate the position of an app in the app store, e.g. 1-10, to the number of backlinks, e.g. #1 = 250 links, #8 = 50 links, to the app store listing page in Python.

I've manage to correlate both rows entirely, which obviously gives me a nonsense figure.

Could someone point me in the right direction to allow me to fill this knowledge gap. I'm not entirely sure of the correct terminology for this, making Googling it pretty impossible.


texts:
Imagine that I've the following dataset:

 Customer_ID    Product_Desc
    1   Jeans
    1   T-Shirt
    1   Food
    2   Jeans
    2   Food
    2   Nightdress
    2   T-Shirt
    2   Hat
    3   Jeans
    3   Food
    4   Food
    4   Water
    5   Water
    5   Food
    5   Beer

I need to make the consumer behaviour and predicte what products are associated. For do that I think that will a good strategy make the relationships first and then count the occurrences (don't know if anyone have a better idea).

The first step is to conclude this relationships:

Jeans-T-Shirt-Food
Jeans-Food-Nightdress-T-Shirt-Hat
Jeans-Food
Food-Water
Water-Food-Beer

How can do this? With Apache PIG or with Spark?

Many thanks!!!


texts: complete entity extraction from unstructured data

texts:
I understand there are many techniques/libraries/packages to extract named entities like people, places etc. from data.
Personally, for me an entity is something like:
first name: john
surname: smith
dob: 1/1/2000
shoesize: 6
address: ...

etc.
So an entity is a class having fields, to use object orientated terminology.
One would expect that these fields/attributes would occur close in a unstructured data (closeness could be defined by word distance). Are there techniques to extract, what I would call, complete entities? Of course this would no be too accurate but anything would be better than nothing. I did a few google searches without success.


texts:
I am trying to compute Silhouette with k-means. However I have the value really close to 0 and the clusters are very clearly separated. Do you know where can be the problem? This is the code:

n_samples, n_features = data.shape
n_digits=2
labels=data[:,-1]
data=data[:,:-1]
sample_size = 300

print("n_digits: %d, \t n_samples %d, \t n_features %d"
      % (n_digits, n_samples, n_features))
print(79 * '_')
print('% 9s' % 'init'
      '    time  inertia    homo   compl  v-meas     ARI AMI  silhouette')

def bench_k_means(estimator, name, data):
    t0 = time()
    estimator.fit(data)
    print('% 9s   %.2fs    %i   %.3f   %.3f   %.3f   %.3f   %.3f    %.3f'
          % (name, (time() - t0), estimator.inertia_,
             metrics.homogeneity_score(labels, estimator.labels_),
             metrics.completeness_score(labels, estimator.labels_),
             metrics.v_measure_score(labels, estimator.labels_),
             metrics.adjusted_rand_score(labels, estimator.labels_),
             metrics.adjusted_mutual_info_score(labels,  estimator.labels_),
             metrics.silhouette_score(data, estimator.labels_,
                                      metric='euclidean',
                                      sample_size=sample_size)))

pca = PCA(n_components=n_digits).fit(data)
bench_k_means(KMeans(init=pca.components_, n_clusters=n_digits, n_init=1),
              name="PCA-based",
              data=data)

print(79 * '_')

The obtained Silhouette is 0.052 and this  is the obtained k-means clustering.

Thanks,

Laia


texts:
I would like to determine whether a given string (eg in English) is pronounceable or not , e.g. using soundex or the like .  Has anyone run across a way to do this?

Examples of unpronounceable:
'lskjd','xqq','gtmm'

Examples of pronounceable:
'aabaa','kwerty','smorglebop'


texts: How accurate does my machine learning model need to be?

texts:
Is there any way to determine upfront how accurate a model should be before it's worth putting into production? Just aiming for the highest accuracy possible doesn't tell us anything about the money a company would save with a 75% versus 80% model.

Say we needed to predict the number of mechanical failures. If the national average is 100/year but our company is getting 125, how accurate does our model need to be (how many correct predictions does it need to make) to get our company back down the national average?


texts: Is not having overfitting more important than overall score (F1: 80-60-40% or 43-40-40)?

texts:
I've been trying to model a dataset using various classifiers. The response is highly imbalanced (binary) and I have both numerical and categorical variables, so I applied SMOTENC and Random oversampling methods on Training set. In addition, I used a Validation set to tune the models parameters by GridSearchCV(). As both precision and recall were important for me, I used f1 to find the best model.

I should note that I selected these three subsets by cluster analysis and extracting samples by stratified train_test_split() from each cluster; so I have more confident that the subsets have more similarity.

Due to complex nature of Decision Tree and Random Forest or boosting techniques, I usually get high fitting (high f1 score) on Training set, relatively high on Validation set, but moderate to low on Test set.

The general sign for overfitting is the high difference between Training and Test sets (or between Validation and Test sets in my problem); but I am confused how to select the best model in following cases:

Case A: Training fit is very high; but Validation and Test sets fit are low but close to each other

Case B: Training, Validation, and Test fits are similar; but much lower than Case A.

                            F1 Score
   Model             Train     Val    Test
----------------------------------------
A: SVC                80.1     60.3    37.5
B: MLPClassifier:     43.2     40.0    39.1

I know that Case A might be the best model,however there is no guarantee that is produces similar result for new data, but which model do you pick with regards to overfitting?
(assume that precision and recalls for both models are similar)


texts: sklearn.decomposition.PCA explained_variance_ratio_ attribute does not exist

texts:
When trying to identify the variance explained by the first two columns of my dataset using the explained_variance_ratio_ attribute of sklearn.decomposition.PCA I recieve the following error:

AttributeError: 'PCA' object has no attribute 'explained_variance_ratio_'

My code (condensed):

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

df = pd.read_csv('Input.csv')
df = df.dropna()
df_transform = StandardScaler().fit_transform(df)
pca = PCA(n_components=2).fit_transform(df_transform)
var_exp = pca.explained_variance_ratio_

When the last line is executed, I get the error:

AttributeError: 'PCA' object has no attribute 'explained_variance_ratio_'

I am using sklearn version 0.20.0

Edit

After examining the attributes of sklearn.decomposition.PCA, I see that the attribute does indeed not exist (as shown in image).


texts: How to use a loss functions closed-form gradient even when it does not have a closed-form itself

texts:
I have a loss function without a closed analytic form that I want to implement in tensorflow that has a nice clean gradient (i.e. i have some nice gradient function $f(x)$ that i want to use for backprop but $\int_x f(x) \ dx$ is with no closed form)

Ofcourse i wouldnt be able to evaluate the loss, but i want to use $f(x)$ in the optimization process. How could this be accomplished?


texts: NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?

texts:
I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text tokenizer. I have never seen a subword tokenizer before and don't know why or when it should be used as opposed to a word tokenizer.
The tutorial says The tokenizer encodes the string by breaking it into subwords if the word is not in its dictionary.
To get an idea of what the results can look like, the work Transformer gets broken down into index-subword pairs.
7915 ----> T
1248 ----> ran
7946 ----> s
7194 ----> former

Does anybody know what the advantages of breaking down words into subwords is and when somebody should use a subword tokenizer instead of the more standard word tokenizer? Is the subword tokenizer used because the translation is from Portuguese to English?
*The version of Tensorflow is 2.3 and this subword tokenizer belongs to tfds.deprecated.text


texts: The impact of Normalization/Data Transformations on Incremental Learning

texts:
Suppose we have a Neural Network (or any machine learning model), and the goal is to perform incremental learning as new data comes in on a regular schedule.

There is extensive literature showing improvement in performance or convergence speeds if the features are normalized to mean 0 and unit variance prior to training, such as Jayalakshmi and Santhakumaran (2011).

However, most feature pre-processing tends to be dependent on the distribution of the features itself. For instance, let $\{X_{t,k}\}$ be a features matrix. In the case of Z-score normalization, we have the transformed matrix $\{Z_{t,k}\}$ where $Z_{t,k} := \frac{X_{t,k}-\mu(X_{k})}{\sigma(X_{k})}$ where $\mu, \sigma$ are the mean and standard deviations of the $k$th column.

In an incremental learning approach, we may have a sequence of batches $\{X_{t,k}\}_{\tau\geq 0}$ so by performing any preprocessing or feature transformations individually on each batch $\tau$, there's the implicit assumption that each batch should be i.i.d. for the model to learn accurately.

The dataset I am working with is a matrix of multi-variate non-stationary time series, so the possibility of concept drift is quite likely. How can I address this in the model?

Some thoughts:

Fit the preprocessing only at the beginning $\tau = 0$, and apply that to all subsequent batches. In this case, if concept drift is severe enough or for large enough values of $\tau$, the transformed matrix may no longer have mean 0 and unit variance - which defeats the purpose of preprocessing in the first case and may make the model less robust.
Fit the preprocessing for every batch. Then, small differences in the distribution of the features matrix could lead to the model overfitting on that particular batch and not being able to generalize to future batches.
Don't do any preprocessing at all. Then due to concept drift, the model may be training on unseen data and also may lead to subpar results in the future.
Stationarize the data (i.e. I(k) differentiation). This is a potential solution so that each batch will be roughly i.i.d., but also wipes out memory of past values beyond the order k. I am using an LSTM NN since it can work with non-stationary time series.


texts: What is the best performance metric used in balancing dataset using SMOTE technique

texts:
I used smote technique to oversample my dataset and now I have a balanced dataset. The problem I faced is that the performance metrics; precision, recall, f1 measure, accuracy in the imbalanced dataset are better performed than with balanced dataset.

Which measurement can I use to show that balancing dataset may improve performance of the model?

NB: roc_auc_score is better in balanced datset than roc_auc_score with imbalanced dataset
Can it be considered as a good performance measurement?
after the explanation i implemented code and i got this results

import pandas as pd
import numpy as np
from sklearn import preprocessing
import matplotlib.pyplot as plt
plt.rc("font", size=14)
from sklearn.svm import LinearSVC
from sklearn.svm import SVC
from sklearn.cross_validation import train_test_split,StratifiedShuffleSplit,cross_val_score
import seaborn as sns
from scipy import interp
from time import *
from sklearn import metrics
X=dataCAD.iloc[:,0:71]
y= dataCAD['Cardio1']
# Split the dataset in two equal parts
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0)
print(y_test.value_counts())
model=SVC(C=0.001, kernel="rbf",gamma=0.01, probability=True)
t0 = time()
clf = model.fit(X_train,y_train)
y_pred = clf.predict(X_test)
t = time() - t0
print("=" * 52)
print("time cost: {}".format(t))
print()
print("confusion matrix\n", metrics.confusion_matrix( y_test, y_pred))
cf=metrics.confusion_matrix(y_test, y_pred)
accuracy=(cf.item((0,0))/50)+(cf.item((1,1))/14)
print("model accuracy \n",accuracy/2)
print()
print("\t\tprecision_score: {}".format(metrics.precision_score( y_test, y_pred, average='macro')))
print()
print("\t\trecall_score: {}".format(metrics.recall_score(y_test, y_pred, average='macro')))
print()
print("\t\tf1_score: {}".format(metrics.f1_score(y_test, y_pred, average='macro')))
print()
print("\t\troc_auc_score: {}".format(metrics.roc_auc_score( y_test, y_pred, average='macro')))

Results:

Name: Cardio1, dtype: int64
====================================================
time cost: 0.012008905410766602

confusion matrix
 [[50  0]
 [14  0]]
model accuracy
 0.5

        precision_score: 0.390625

        recall_score: 0.5

        f1_score: 0.43859649122807015

        roc_auc_score: 0.5

For balanced dataset

X_train1,y_train1 = sm.fit_sample(X_train, y_train.ravel())
df= pd.DataFrame({'Cardio1': y_train1})
df.groupby('Cardio1').Cardio1.count().plot.bar(ylim=0)
plt.show()
print(X_train1.shape)
print(y_train1.shape)
#model=SVC(C=0.001, kernel="rbf",gamma=0.01, probability=True)
model=SVC(C=10, kernel="sigmoid",gamma=0.001, probability=True)
t0 = time()
clf = model.fit(X_train1,y_train1)
y_pred = clf.predict(X_test)
t = time() - t0
print("=" * 52)
print("time cost: {}".format(t))
print()
print("confusion matrix\n", metrics.confusion_matrix(y_test, y_pred))
cf=metrics.confusion_matrix(y_test, y_pred)
accuracy=(cf.item((0,0))/50)+(cf.item((1,1))/14)
print("model accuracy \n",accuracy/2)
print()
#print("\t\taccuracy: {}".format(metrics.accuracy_score( y_test, y_pred)))
print()
print("\t\tprecision_score: {}".format(metrics.precision_score( y_test, y_pred, average='macro')))
print()
print("\t\trecall_score: {}".format(metrics.recall_score(y_test, y_pred, average='macro')))
print()
print("\t\tf1_score: {}".format(metrics.f1_score(y_test, y_pred, average='macro')))
print()
print("\t\troc_auc_score: {}".format(metrics.roc_auc_score( y_test, y_pred, average='macro')))

Results:

(246, 71)
(246,)
====================================================
time cost: 0.05353999137878418

confusion matrix
 [[ 0 50]
 [ 0 14]]
model accuracy
 0.5

        precision_score: 0.109375

        recall_score: 0.5

        f1_score: 0.1794871794871795

        roc_auc_score: 0.5

I found  no efficient results. Should i implement the model using cross validation?


texts: A trick used in Rademacher complexity related Theorem

texts:
I am currently working on the proof of Theorem 3.1 in the book "Foundations of Machine Learning" (page 35, First edition), and there is a key trick used in the proof (equation 3.10 and 3.11):

$$\begin{align*}
&E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m}  g(z'_i)-g(z_i)\right]=E_{\boldsymbol{\sigma},S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} \sigma_i(g(z'_i)-g(z_i))\right] \\
&\text{where } {\Bbb P}(\sigma_i=1)={\Bbb P}(\sigma_i=-1)=\frac{1}{2}
\end{align*}$$

It is also shown in the lecture pdf page 8 in this link:
https://cs.nyu.edu/~mohri/mls/lecture_3.pdf

This is possible because $z_i$ and $z'_i$ can be swapped.  My question is, why can we swap $z_i$ and $z'_i$?  In the book, it says this is possible because "we are taking the expectation over all possible $S$ and $S'$, it will not affect the overall expectation."  Unfortunately, I don't understand the meaning or intuition of this part.

Thank you very much for reading the question and for your time!


texts: How exactly does a validation data-set work work in machine learning?

texts:
With typical machine learning you would usually use a training data-set to create a model of some kind, and a testing data-set to then test the newly created model. For something like linear regression after the model is created with the training data you now have an equation that you would use to predict the outcome of the set of features in the testing data. You would then take the prediction that the model returned and compare that to the actual data in the testing set. How would a validation set be used here?

With nearest neighbor you would use the training data to create an n-dimensional space that has all the features of the training set. You would then use this space to classify the features in the testing data. Again you would compare these predictions to the actual value of the data. How would a validation set help here as well?


texts: Is it scientifically correct to derive conclusions unrelated to hypothesis from A/B test data

texts:
Consider a software A/B test with the hypothesis that "the addition of feature F is predicted to increase metric X".

At the end of the test, the data doesn't show any significant change in X, but it does show a significant increase in Y - something that wasn't expected or even considered at the beginning of the experiment.

At this point, is it scientifically valid to say that F increases Y, or should a new A/B test be designed and executed?


texts: LSTM: Converting to Bayesian Deep Neural Network

texts:
Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My model comprises of 2 LSTM layers, followed by a relu dense layer, followed by a softmax layer. A dropout layer is added after each of the layers.

I've found some implementations for measuring the uncertainty of a deep neural network (like this one here: https://fairyonice.github.io/Measure-the-uncertainty-in-deep-learning-models-using-dropout.html), but all of them seem to be applicable to dense layers rather than LSTM layers.

So in summary, what I am asking about is:

How can I turn on Dropout during testing for LSTM layers?
How can I measure my model's uncertainty?


texts: What is the difference between bootstrapping and sampling in reinforcement learning?

texts: I have come across a David Silver's slide which contains both the terms "bootstrapping" and "sampling". Is there any realistic example which helps me to understand the concepts better.


texts: Features selection with a lot of dummy variables in R

texts: I am performing features selection on 3849 dummy variable (one-hot encoding) using Boruta algorithm and the algorithm is taking forever to run. Is there a faster way I can perform features selection faster or I only have to reduce my number of dummy variables


texts: Logistic Regression - ROC curve plots Sensitivity vs Specificity instead of (1-Specificity)

texts:
I am new to Machine Learning and have been doing some practice on Logistic Regression. To evaluate the models, I've been trying to create some ROC plots.
The package that i used is pROC.

The model name is - 'model' dataset is 'data'.

The code I used in R is:

library(pROC)
predictionData <- predict(model, newdata = data, type = "response")
rocModel <- roc(data$y ~ as.numeric(predictionData>0.5))
plot(rocModel)

Ideally, from what i have learned, ROC should plot Sensitivity or TPR (True Positive Rate) vs 1 - Specificity. But as shown in the picture below, it shows Sensitivity vs Specificity.

Am I missing some obvious trick here or is something wrong with what I have done ?


texts: Cloudera recently added the spark-time series library to github. According to the user docs, it definitely can fit autoregressive integrated moving average (ARIMA) models, but I see no mention of ARIMAX, which takes into account explanatory variables. Does anyone with more experience with this library know whether ARIMAX is possible in the library's current implementation? Mathematically, I understand the difference between ARIMA and ARIMAX, and I know extending it myself wouldn't be terribly difficult, but I'm right now looking for an off-the-shelf solution in spark. Can anyone recommend an alternative spark implementation?


texts: How to print accuracy in each fold of Validation Dataset? and assign fold number to each row in the dataframe?

texts:
How to print the accuracy in each fold of a Validation Dataset ? And how can I assign the fold number to each row in the dataframe?

classifier = RandomForestRegressor(n_jobs = -1, criterion='mse')
clf = GridSearchCV(classifier, param_grid = tunedParameters,cv=10)
all_accuracies = cross_val_score(classifier, X=X_train, y=y_train, cv=10)
print(all_accuracies)


texts: Modern methods for reducing dimensions and feature engineering

texts:
I am training a binary classifier in Python to estimate the level of risk of credit applicants. I extracted a little over a thousand independent variables to model the observed behavior of four million people. My target is a binary column that tells me whether or not a person defaulted on a loan (1 for event, 0 for non event).
I am asking this question because I feel overwhelmed by the dimensionality of the problem. I want to know some common and modern ways used to:

drop features (dimensionality reduction)
create new features based on combinations of other features (feature engineering)

So far, I dropped features based on their Information Values and kept only the most relevant ones. From the remaining set of features, I calculated each pair's correlation coefficient and for every highly correlated pair, I kept the feature with the highest information value of the two.
I now want to make new features based on this subset of remaining variables, such as ratios and multiplications (i.e. Number of open accounts divided by number of closed accounts). However, I think that my elimination process can be improved. My current method is very old school, as I have mostly relied on univariate analysis to drop features (one variable against the target).


texts: Connection between the first pooling layer and the second convolutional layer

texts:
This is a follow-up question regarding this question. I managed to verify in python that the output of the first pooling layer will be a $14 \times 14\times 32$ tensor.

I interpret this as having 32 $14 \times 14$ images. Maybe my interpretation is mistaken. In the second convolutional layer we apply $64$ filters.

When I type in python conv2.shape I get

TensorShape([Dimension(55000), Dimension(14), Dimension(14), Dimension(64)]).

How that comes? I mean I am in interested in the outcome of the depth, which is $64.$


texts: How to automate left join of multiple data frames with single data frame one by one in R

texts:
i have 206 data frames named after country(FRANCE,GERMANY,HONGKONG...) with same 37 variables in these 206 files and another data frame name pf with 9 variables.

i want to do left join of these 206 data frames with PF data frame one by one selecting all variables and store in other data frame. left join is based on two condition( hcom_id=tnow_id and city=region). hcom_id and city are variables in 206 different data frames and tnow_id and region are in PF

i am using sqldf package to join:
a<-sqldf("select * from FRANCE left join PF on (FRANCE.hcom_id=pf.tnow_id and FRANCE.city=pf.region)")

I am struggling to automate it using loop rather than doing 206 times separately.

Any idea please suggest


texts: How to predict churn events that may happen within a period of time?

texts: I am trying to build a model that predicts churn events in the future. The business wants to be able to identify which customers are likely to terminate the services within a month. "Within a month" can mean the next day or the 30th day. The problem is some of the features are time-based, for example how many months into the current term, the number of tickets created in the last two weeks, etc. If the event date is floating, how do I calculate the values of these features? Should I make 30 copies of the same churned service and calculated these time-based features for each of them? Is there a better way to approach this?


texts:
I have a dataset with 4712 records. Label Yes - 1558 records and Label No - 3554 records.

I read online that 1:10 rule is based on the frequency of lower occurring class.

In my case, frequency of lower occurring class is 1558

According to 1:10 rule, am I right to understand that it is calculated like 1558/10 = 155.8 further equals 150 predictors

So In my logistic regression, I can use 150 variables/input features to the model without the risk of overfitting. Am I right?

By any chance do we also have to look at the frequency of the other (high occuring) class to determine the no of predictors that I can use? If yes, can you share me as to what has to be done to determine the predictor count?

I am aware that we could also use 1:20 or 1:50 rule. But my question is mainly on

1) Whether is there any other consideration for determining the number of predictors in logistic regression model?

2) How do people calculate min sample size required based on this?

Can someone help me with this?


texts: Performace of Fischer projection as dimension reduction compared to other LDA methods

texts:
How is the performance of Fischer projection compared to other LDA methods of dimension reduction? I thought that Fischer projection was a great method of dimension reduction by maximizing class separation, but when I looked at the LDA methods in scikit learn, Fischer projection wasn't even in the list. This got me thinking, is it any good compared to the other methods out there?

Edit (answer): My bad, they are the same. Fischer projection is a 2 class special case of LDA. Projection using LDA can be performed using the fit_transform and transform methods in sklearn


texts: SEM (Structural Equation Modelling) with Exploratory Factor Analysis

texts:
Problem Statement: I need to do some Structural Equation Modelling at work to get the main factors in a marketing survey data-set.

There are no assumed equations to perform SEM on so what would be the best exploratory way to create those equations out of the data
All variables are very highly correlated across all the variables so how can we deal with that

Please let me know if I can help you with any supplements.


texts: How to use multiple encoders(one-hot and numerical) together for PCA

texts: I want to implement PCA on a dataset(retail) but the data is categorical. One-hot encoding on some columns like Gender, Fabric, Brand makes sense but on other features like price range, size, I would like the encoded values to have some numeric significance, i.e. higher value actually means something. Any suggestions on implementing both these encodings together for PCA?


texts: Why are ordered lists considered one dimensional?

texts:
At Uni I am currently taking data visualization as a course.
While reading thru the script I stumbled upon two examples for one dimensional data.
Example one:
Bus stops with their name and order (current -> next bus stop).
Example two:
A list of values (not a set!)(the type of value does not matter for the sake of simplicity).
Now I am wondering how an ordered list can be two dimensional. To me it looks like two dimensional data.
The first dimension being the index, the second one being the value. For the first example the two dimensions could be the station name and the next stations name.
Since v(t) is considered as two dimensional data (according to my Prof) I am wondering why value(index) us not two dimensional data.
Representation wise I can understand why the data can be represented one dimensional, since they can be represented in a straight line.
But feature wise I don't understand how value(index) and v(t) are different.
Please enlighten my mind since the prof only told me to stick to the script...


texts:
I am learning how to implement Keras LSTM on a simple time series data.
The dataset I'm using has $12$ columns and $300k$ rows. Each group of $200$ rows represents one time-series cycle. Then, the time starts again from zero and run for the next $200$ rows. I want to make predictions for sixof the columns.
My questions are, how many (multiple of 200?) rows do I choose for train/test sets? Is my batch size $200$? What is the basis for choosing number of neurons?


texts:
I have two disks, of equal diameter. Both disks have a 'spatter' of holes, in random positions (from a two-dimensional - i.e., surface -  uniform distribution; in other words, holes close to the sides are as likely to happen as holes close to the center). The number of holes per disk is normally distributed; holes' diameters are normally distributed, too.

If I overlap the two disks, some holes will (even partially) overlap; in other words, if I have a source of light beneath them, I will see light coming through the overlaps (possibly none, of course); it goes without saying that that percentage will change if I rotate the disks with respect to each other, from a minimum to a maximum.

I'm interested in the statistical distribution of the percentage of overlapping holes with respect to the surface of one disk. In particular, what will be the probability of having less than a certain percentage of overlap if I take two disks at random, and I measure separately their total surface of holes?
To fix the idea: let's say disks A and B have holes accounting for 25% and 35% of their surfaces, respectively; clearly, the maximum overlap will be 25%. If I overlap them at a random angle, is there a theoretical way to predict that the overlap will be less than a given value, say, 10%?

I suspect that the answer comes from Gaussian Random Fields, but I'm definitely not sure.

many thanks in advance for your suggestions and insights.
Kind regards,
Luca


texts: I am new to xgboost, I get weird results when apply the code to a dataset with values that are below 0 (0.023-0.035....etcetera)

texts:
parameters = {
    'n_estimators': [100, 200, 300, 400],
    'learning_rate': [0.001, 0.005, 0.01, 0.05],
    'max_depth': [8, 10, 12, 15],
    'gamma': [0.001, 0.005, 0.01, 0.02],
    'random_state': [7]
}

eval_set = [(X_train, y_train), (X_valid, y_valid)]
model = xgb.XGBRegressor(eval_set=eval_set, objective='reg:squarederror', verbose=False)
clf = GridSearchCV(model, parameters)
clf.fit(X_train, y_train)


texts: Did I do the right thing in my CNN Keras (class imbalance - augmentation)

texts:
To implement my Binary CNN in keras, I had a dataset of ~~35000 images but only 700 is from one class and all the others are from the other class, so what I did: I get the 700 unique images from class 1 and get 700 images randomly from class 0 totalizing 1400 samples (700 each class). After I perform this augmentation in my set:

datagen = ImageDataGenerator(
    rotation_range=30,
    zoom_range=0.15,
    featurewise_std_normalization=True,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.15,
    horizontal_flip=True,
    fill_mode="nearest")

I did it right? or there's a better way to do?


texts: Why is this convolution equation easier to apply than it's commutative counterpart?

texts:
The convolution is an operation on two functions of a real- valued
argument.
The convolution operation is typically
denoted with an asterisk:
s(t) = (x ∗ w)(t)

It is a special kind of linear operation which is also commutative, so:
s(t) = (x ∗ w)(t) is also equal to s(t) = (w ∗ x)(t).
In convolutional neural networks we usually use convolutions over multiple axis. In a  two-dimensional image we want to use 2 convolutional operations over two axis at a time. We also use a 2-D kernel as follows:
$S(i,j) =(I*K)(i,j)= \sum\limits_{m} \sum\limits_nI(m,n)K(i-m,j-n)$
where I is the input, and K is the Kernel.
It's commutative equivalent is:
$S(i,j) =(K*I)(i,j)= \sum\limits_{m} \sum\limits_nI(i-m,j-n)K(m,n)$
As per www.deeplearningbook.org,

Usually the latter formula is more straightforward to implement in a machine
learning library, because there is less variation in the range of valid values of
m
and n.

I am struggling to comprehend this sentence and thus, my question is, why would there be less variation in the range of valid values of m and n when they are commutative equivalents?


texts: Need help in deriving Policy Evaluation (Prediction)

texts:
Policy Evaluation is computing the state-value function for an arbitary policy $\pi$.(suton & barto book).
Now

\begin{equation}
v_{\pi} = E_{\pi}[\;G_{t}\,|\;S_{t}=s] \qquad\qquad\qquad\qquad(4.1)\\\
        =  E_{\pi}[R_{t+1}+\gamma G_{t+1} |S_{t}=s]\qquad\qquad\quad(4.2)\\
        =  E_{\pi}[R_{t+1}+\gamma v_{\pi}(S_{t+1})| S_{t}=s]\qquad\qquad(4.3)\\
        = \sum_{a} \pi(a|s) \sum_{s^{'},r} p(s^{'},r|s,a)[r+\gamma v_{\pi}(s^{'})]\quad(4.4)
\end{equation}

Can someone help me to figure out from (4.3) to (4.4). And If equation (4.4) is bellman's equation for state value function, then how is it different from below equation
\begin{equation}
v_{\pi} = \sum_{a\in A} \pi(a|s)[R_{s}^{a}+\gamma \sum_{s'\in S} P_{ss^{'}}^{a}v_{\pi}({s^{'})}]
\end{equation}


texts: Resources for data science applications in finance/banking

texts:
What are good resources for data science applications in finance/banking ?

I want to know about the following:

Trends in finance/banking area
What creative companies do in finance/banking area with data science
Major open data sources in finance/banking in Canada

I subscribed to several data science news but none of them focus on finance/banking. I need daily/weekly news and reports for my internship. This is my job and I will be evaluated based on this so it is very important for me to get enough information in time.

Any suggestions? Thank you very much!


texts: UnboundLocalError: local variable 'result_table' referenced before assignment

texts:
# Define a result table as a DataFrame
result_table = pd.DataFrame(columns=['classifiers', 'fpr','tpr','auc'])

# Train the models and record the results
for cls in classifiers:
    model = cls.fit(X_train, y_train)
    yproba = model.predict_proba(X_test)[::,1]

    fpr, tpr, _ = roc_curve(y_test,  yproba)
    auc = roc_auc_score(y_test, yproba)

    result_table = result_table.append({'classifiers':cls.__class__.__name__,
                                        'fpr':fpr,
                                        'tpr':tpr,
                                        'auc':auc}, ignore_index=True)

What should I do.. I am setting the dataframe initially but still getting error... Pls help!


texts: Processing data in the right manner in data science

texts:
From what i have learned, people say that it is more correct if i preprocess data after splitting it to train/test dataset. My questions are

1.Does it mean we detect flaws of the data + preprocess it after the splitting? if yes/no, why?

2.Is it okay if i detect flaws of the data before splitting it and then preprocess the flaws after splitting it? if it is/ no it is not, why?


texts: What is the appropriate evaluation metric for RandomForest with probability in R?

texts:
In order to build a predict model with two categories (buy or not buy),I want to use RandomForest and predict with type='prob', so I can have a prob of someone buy or not buy. So, with this outcome I can clusterize and make groups, like this:

group A: costumer who has [100 to 80]% of buy.
group B: costumer who has [81 to 60]% of buy.
...

But I don't know the appropriate evaluation metric to measure the accuracy of this model. I guess that I can't use a confusion matrix.

Maybe I can use a ROC curve, and or measure KS between the buy group with the not buy group. But I'm not sure about this metrics.


texts: Algorithms that can determine whether a string is an English sentence?

texts:
In particular, I'm looking for something that can distinguish between complete sentences and sentence-like grammatical structures like clauses and phrases. It would also be very helpful if a library with this functionality was shared, but even a link to a paper would be appreciated.

For example- I would like "False" to be returned for strings like:

'Via The Daily Currant,'

“Waiting for the rain to stop”

'查看更多请返回网站主页。'

'*  *  *'

I would like "True" to be returned for things like:

"This is a complete sentence."

"I don't think there will ever be a Half Life 3, unfortunately."

Thank you!


texts: Which features of a data set can be used for market campaigning using propensity scores?

texts: A dataset contains so many fields in which there is both relevant and irrelevant field. If we want to do a market campaigning using propensity scoring, which fields of the data set are relevant? How can we find which data field should be selected and which drive to the desired propensity score?


texts: model.fit indefinite execution in Google Colab

texts:
I am working on image classification model. My model is build using CNN, tensorflow and Keras  libraries.

I am trying to execute my CNN multi-classification model in google Colab, and I am loading the data (image data) from my google drive (mount drive command). Everything is going perfectly fine, up to model.fit, at this point the model execution is indefinite (never-ending) in Google Colab. However the same model is working perfectly in Jupiter Notebook.

So, can anybody tell me what is happening?


texts:
I'm wondering if there is literature or studies done on how to model organic attribution from paid user acquisition.

So the context is, on our mobile app, we have paid installs that we purchase and organic installs, but we believe organic installs mostly come from either featuring or referrals from current users. Thus, for every paid user we buy, we actually can attribute some organic installs to that paid user.

My problem then is what this coefficient is. My first instinct is to use a granger like model regressing on the previous values of organic and paid installs with a dummy variable for featuring like this:

$y_t=\beta_0+\beta_1 y_{t-1}+ ... + \beta_q y_{t-q} + \gamma_1 x_{t-1} + ... + \gamma_q x_{t-q} +\delta_0 z_t+\varepsilon_t$

where $y_t$ is organic installs, $x_t$ is paid installs, and $z_t$ is just a dummy variable indicating whether the game was featured on date $t$

Is this model valid and what would be the best way to determine the order for this model?

Anyways, would love to hear everyone's thoughts or suggestions if there's a well-defined model for this!


texts: Using Gini index, how to calculate the probability of correctly classifying a new randomly selected case to the highest probability class?

texts:
I have the following binary Decision Tree:

Can you please explain how can I report this tree to a person who only understands probabilities?

If ca=1 and cp_4.0=1, what’s the probability of Yes HD?


texts: Determining whether a sentence is "cliche" using NLP

texts:
I have a collection of essays from students. Each essay is about the same topic and of the same word length. My goal is to develop a machine learning algorithm that pinpoints "cliche" sentences (i.e., sentences that are "unoriginal" and similar to what other students have written).
Let document X be the essay we're trying to analyze. Let S be the set of all essays. Here is the rudimentary approach I've thought of:

Split all essays (from set S - X) into individual sentences. Call this collection of sentences Y. Use this solution to compare each sentence of X to each sentence in Y. This will yield an array of scores Z for each sentence in X.
For each Z: weight each score in Z based on position. If a sentence in X and a sentence in Y are in similar locations in their respective essays (e.g., both at the intro of the essay) the weight on the respective score in Z will be higher than if these two sentences were farther apart.
Average all of the Z arrays for all of the sentences. The sentence with the highest average is the "most cliche."

Is there a better approach to this problem?


texts: Linear Regression to predict a growing variable with time

texts: Can we use Multiple Linear Regression to predict a dependent variable that is growing exponentially with time?


texts: Text similarity with sentence embeddings

texts:
I'm trying to calculate similarity between texts with various lengths. My current approach is following:

Using Universal Sentence Encoder, I convert text to a set of vectors.
I average these vectors to create the final feature vector.
I compare feature vectors using cosine similarity.

This gives me pretty good results for texts with roughly same sizes, but I was wondering if there is a better approach for the step #2 if texts have different lengths.


texts: Machine Learning Model Alert/Notification system - Python

texts:
Let's say, I have a Machine Learning model built and does predictions for the future. If I have some predicted values that exceed a threshold, can I set up some type of notification system? It can be in email or a text message. Is it possible to do it in python?

I have been searching online, could not find a valid answer. Any thoughts or resources would be appreciated.


texts: When/how should I use the validation set for hyper-parameter sweeps for neural networks?

texts:
I know similar questions have been asked so many times, but I couldn't find the answer to this one particularly, at least not in a way that satisfied me.

I am very confused about how to use validation sets. I know they are used to perform hyper parameter sweeps but I'm not quite sure in what way. For example, suppose I am trying to decide between a neural network that has 1 hidden layer and one which has 2, and I have a 50/25/25 train/validation/test split of my data. I will not be performing k-fold CV for reasons which are inherent to my problem.

Should I be using the validation set during training (to prevent overfitting) and the same validation set after the fact (once when the models are completely trained) to determine which model is better? When do I use the validation set to compare models?

And then once I get to the test set, I know I am supposed to train on the training data and validation set together. But should I actually keep the validation set separate from the training set during training of the finalized model to prevent overfitting? Why would I use the separate validation set during training of the individual models to prevent overfitting, but not use it for that purpose (and make it more training data) on the final model?


texts: test_train_split with stratify integer overflow

texts:
I'm trying to do a stratified split for a skewed dataset with target variable 'b'. The target variable is a bit value (either 0 or 1). Here's an example:
df = pd.DataFrame(data={'a': np.random.rand(100000), 'b': 0})
df.loc[np.random.randint(0, 100000, 1000), 'b'] = 1
tr, ts = train_test_split(df, test_size=.2, stratify=df['b'])
print(tr.shape, ts.shape)

This code returns the following:
(93105, 2) (38, 2)

My problem is that the returned train/test arrays do not meet the set split ratio of 20%.
My setup:

Python 3.7.0 (32bit)
Sklearn 0.20.3
Pandas 0.23.4

I discovered that the problem is resulting from an integer overflow in the underlying split function.
How can I resolve this issue and is this a known bug? I couldn't find anything helpful.


texts: Learning conditional statements from natural language

texts:
Natural language text in emails might have conditional statements in them. Are there any technical papers and methods that explore converting unstructured text (eg. emails) into structured conditional statements?
For example:
Natural Language -
".. Hey, come home tonight if you are free. Otherwise, can we meet tomorrow?"
Output -
<Conditional_IF> You are free </Conditional_IF>
      Come home tonight
<Conditional_ELSE>
      Can we meet tommorow?


texts: How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)

texts:
Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly there are not nearly as many older people, etc..).

How could I draw a sample from the old data set with respect to that shifted variable so that the distributions would mimick the new data distributions?


texts:
I am trying to understand how you marginalise a joint distribution.

In my case I have a fair coin,  $P(C) = \frac12$ and a fair dice $P(D) = \frac16$. I am told I win a prize if I flip the coin and it lands on Tails and if the outcome of the dice $= 1$. I am told at least one of them is correct.

$$Q = (\text{Coin = Tails or Dice} = 1)$$

$$W = (\text{Coin = Tails and Dice} = 1)$$

So if I wanted to work out the probability $W =$ True $| Q =$ True I can use marginalisation to work this out given the joint distribution:

$$P(C), P(D), P(Q|C,D), P(W|C,D)$$

I am just not sure where to start any help would be really appreciated. I am pretty new to this.

Thanks in advance.


texts: Correlation between (average daily) impressions and user response ratio in social media

texts:
I'm doing a preliminary study on data from a (niche) social media platform. Studying the correlations between impressions of an object (aka views), interactions (aka likes, comments, ...) and their ratios, it seems that the interacting audience (people who leave a like or a comment) is "capped" after a while. A phenomenon that is best described by this quote from an (old) article about popularity on YouTube:

  In a counter-intuitive way, the user response ratio (i.e. number of comments per thousand views) decreases as a function of the number of viewcounts. In other words, as a video is seen by an increasing number of people, it elicits less acute reaction.

I believe that this phenomenon might be more general, but I could not find many references in academic papers or systematic studies of those indices. Several naive questions are quite natural: are there similarities between the ratios in different social media? Are the ratios spurious? ...

Maybe this kind of phenomena is so classic that it is not useful to remark it in research papers, but I would like to find reference on this phenomenon.


texts: Convert Neural network to Keras Classifier

texts:
I am training a Neural Network for Multi-Class classification. After succesfully training it and validating the model through cross validation, I would like to use this network inside a voting Classifier. In order to perform cross validation on my trained network I convert it to a Keras Classifier and then calculate its validation score. However when I parse the same exact "Keras Classifier" in the Voting Classifier method, I get the following error:
ValueError: The estimator KerasClassifier should be a classifier

The code can be seen below:
import random
random.seed(42)

from keras.layers import Dense
from keras.models import Sequential
from keras.wrappers.scikit_learn import KerasClassifier
import tensorflow as tf
from sklearn.ensemble import VotingClassifier

def NeuralNetwork():
    model = Sequential()
    # define first hidden layer and visible layer
    model.add(Dense(600, input_dim=k, activation='relu'))
    model.add(Dense(20, activation='relu'))
    # define output layer
    model.add(Dense(3, activation='softmax'))
    # define loss and optimizer
    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=["accuracy"])
    return model

NN = KerasClassifier(build_fn=NeuralNetwork , epochs=100, batch_size=100, verbose=0)
cross_val_score(NN, X_new_train,y_train_no_id, cv=3)

votingC = VotingClassifier(estimators=[ ('LR1', LR1),('LR2', LR2), ('XGB',XGB),('NN',NN)], voting='hard', n_jobs=4)
#LR1 and LR2 are some other Logistic Regression estimators defined in another section
votingC = votingC.fit(X_new_train, y_train_no_id)
votingC.predict(X_new_test)


texts: It is a completely theoretical question. What if the training data and the prediction data are the same? Will it impact the results if I use the same data for model predictions that I have used for training?


texts: Dealing with extreme values in softmax cross entropy?

texts:
I am dealing with numerical overflows and underflows with softmax and cross entropy function for multi-class classification using neural networks. Given logits, we can subtract the maximum logit for dealing with overflow but if the values of the logits are quite apart then one logit is going to be zero and others large negative numbers resulting in 100% probability for a single class and 0% for others. When loss is calculated as cross-entropy then if our NN predicts 0% probability for that class then the loss is NaN ($\infty$) which is correct theoretically since the surprise and the adjustment needed to make the network adapt is theoretically infinite.

I know this can be dealt with normalizing the data and choosing weights and biases from standard normal distribution but this is a what-if scenario where the data is mean preprocessed but not standard deviation processed, I believe this can also occur even after preprocessing both mean and stddev.


texts: What morphological operations can be used to vectorise road surface binary rasters?

texts:
We are currently developing a Deep Convolutional Neural Network to extract road surfaces from aerial orthophoto's. Our current workflow involves using an existing road centreline to mask a large enough area to determine which of the pixels are road surface. The probability binary raster output are reasonably accurate. The final step within our workflow is that we use the original road centreline and average pixel width to determine the road width to buffer the road centreline as the output the road surface area. The problem with the following approach is that collector and arterial roads are not uniform in road width as well as the road bellmouth is lost at road intersections. Currently the team that has developed the following model is using Matlab, which I'm not familiar with, hence the following question to find out what morphological operations could be used to extract a better representation of the road surface by approximating road edges (straight lines), varying road widths and bellmouths at road intersections from the probability raster output from the CNN model.

CNN: Current Workflow

CNN: Red (Manually Derived Road Surface \ Training Dataset) Yellow (Vectorised Road Surface Output)


texts: Heat map and correlation among variables

texts:
I would have a question on heat map and correlation among variables.
I created this heat map, looking at possible correlation among variables and target. I got very small values.
I wanted to set a small threshold, e.g., 0.05, for selecting features.
Do you think it makes sense, or should I exclude all of them?


texts: memory error while converting images into an array

texts:
I am working on a facial recognition use case. I have 57k jpg images and am converting them into an array. While executing the program, I am getting a  memory error.

The function I am using:

def image_array(l):
    features = []
    for pgm in l:
        pic = image.load_img(pgm, target_size=(224, 224))
        x = image.img_to_array(pic)
        x = np.expand_dims(x, axis=0)
        x = preprocess_input(x)
        features.append(x)
    npfeatures = np.array(features)
    img_dt = np.rollaxis(npfeatures, 1, 0)
    return img_dt[0]

The input for this function is a list which looks like:

l =['/home/user/image1.jpg','/home/user/image2.jpg','/home/user/image3.jpg'......]

The error that I am getting:

Traceback (most recent call last):
  File "FR.py", line 145, in <module>
    vec_image1 = image_array(final_df['image1'].values.tolist())
  File "FR.py", line 140, in image_array
    npfeatures = np.array(features)
MemoryError

The imports that I used for above function are:

> from keras.preprocessing import image
> from keras.applications.vgg16 import preprocess_input


texts: Which is better: Cross validation or a validation set for hyperparameter optimization?

texts:
For hyperparameter optimization I see two approaches:

Splitting the dataset into train, validation and test, and optimize the hyperparameters based on the results of training on the train dataset and evaluating on the validation dataset, leaving the test set untouched for final performance estimation.

Splitting the dataset into train and test, and optimize the hyperparameters using crossvalidation on the train set, leaving the test set untouched for final performance estimation.

So which approach is better?


texts: How to chose training sample for neural network in fraud detection?

texts:
I am working on a personal project sponsored by a data scientists, he offered some data sets to me.

And I have built a neural network system based on my own theory.

But when I tried to train it, I always can't get a ideal result. I think carefully chose the training data set samples may help.

So I would like to know if there are some principles or theories things which can guide researchers choosing samples.

Any books, papers or blogs would help.

by the way, I am working on a fraud detection project.

UPDATE

Ok, I think there are some misunderstandings in the question.

What I am looking for is a systematical theory which explains how to organize the training set.

For a naive example:

I have example data set from US. There are tons of data, so it is not possible to train the machine with all the example set. Here comes the question, how should I pick carefully the training data set from the example data set? how to do that is a reasonable way? and a logical way?

For a further example:

Choose the training set according to states evenly? or choose training set according to their population as weight?

Thanks, and I will provide more information if it is needed.


texts: Spatial clustering based on response to inputs and building a reduced model

texts:
Stats version: I have a few measurements of a function that takes three inputs and produces a few 2D fields of outputs: $f_i(a,b,c;x,y)$, with $f$ being a vector of several quantities. I would like to cluster the points in $(x,y)$-space into regions that have a similar response to the input variables $(a,b,c)$. If I treat the points independently, applying a clustering algorithm is fairly easy. (I can elaborate on which one I'm using if you think it's relevant, but I'm open to suggestions on your favorites.) But there's no guarantee that the clusters are near each other spatially. Having contiguous clusters of roughly equal size is definitely a plus. What are some ways to modify the approach to add in that constraint?

Science version: I'm running an atmospheric chemistry model and varying emissions over the US. In that setup, $(a,b,c)$ are the national-total emissions of NOx, SO2, and NH3; $(x,y)$ are grid points over the US; and $f_i$ is a vector of the various chemical species I'm tracking. I've varied $(a,b,c)$ in a number of simulations. Now I want to divide the country into regions that have broadly similar responses to the emissions changes, and create simplified models for a manageable number of regions. I'm particularly interested in nonlinearities/mixed-effects/second-order derivatives in the chemistry response. It's possible that different regions of the country do respond similarly to emissions even if they're not near each other, but the different regions should have some sort of spatial agglomeration. Thoughts?

Side question: I originally played around with this a few years ago with Matlab code but have since migrated to Python. Suggestions on how to efficiently use Python stats packages for this sort of problem?


texts: Is data augmentation changing the train/test sets distribution?

texts:
I am thinking a simple question which just came to my mind.
There are many people using data augmentation on their image data to train deep CNN.
When I learn from Andrew Ng's DL courses, he mentioned that to train a better model, you generally need to have your train/test data come from the same distribution. Like, if you train you CNN with a lot of images of cars and human, it's (usually) not a good idea to use it to classify a cat or a dog.
After the augmentation, my training data is kind of boosted, then can I still say my train/test data set comes from the same distribution? Am I violating what he said? Why? Or, why not?


texts: PyTorchs ConvTranspose2d padding parameter

texts:
Im confused about what PyTorchs padding parameter does when using torch.nn.ConvTranspose2d.
The docs say that:

"The padding argument effectively adds dilation * (kernel_size - 1) -
padding amount of zero padding to both sizes of the input".

So my guess was that the dimensions of the feature maps increase when applying padding. However running test they decrease:
inp = torch.ones((1, 1, 2, 2))
conv_no_pad = nn.ConvTranspose2d(1, 1, kernel_size=(3, 3), stride=2, padding=0)
conv_pad = nn.ConvTranspose2d(1, 1, kernel_size=(3, 3), stride=2, padding=1)
print(conv_no_pad(inp).shape)
# => (1, 1, 5, 5)
print(conv_pad(inp).shape)
# => (1, 1, 3, 3)

Can somebody explain how the padding works?


texts: Padding sequences for neural sequence models (RNNs)

texts:
I am padding sequences for a GRU based classifier that I am building in Keras.  I'm wondering if there's any accepted best practice for padding the leading or trailing side of the sequence.

E.g.

sequence = [1,2,3,4]
leading_pad = [0,0,0,1,2,3,4]
trailing_pad = [1,2,3,4,0,0,0]

In other projects I have generally padded the leading end of the sequence as a convention imprinted on me by various blogs, course work and other examples.

My question: is there any research that shows whether or not it matters which side of the sequence you pad?  Or am I completely over thinking this?

Any advice would be much appreciated.


texts: Item Based Collaborative Filtering with No Ratings

texts:
I am building a recommender for web pages. For each web page in our data set, we wish to generate a list of web pages that other users have also visited.

Our data only shows that a user has either visited a page, or they have not. Users do not provide any ratings of our web pages. This is a good task for item based recommendation. However, most of the algorithms (such as the one in Mahout) requires rating data.

The first solution I came up with was to use a graph database and write a query which does the following:

For each page we want recommendations for, we search for all the users who have viewed that page. Then, for each of those users, we look up all other pages they have viewed. We then count the number of users which have viewed each page in this data set, and use those with the highest count as our recommendations.

While this works pretty well, our data set has grown substantially and scaling the graph database is difficult. The queries become slower as the number of page views in our data set increases. We would like to consider a different implementation before we commit to moving to a distributed graph database.

In a more traditional item-based recommender (like Mahout's), is there a good way to 'fake' the ranking data, or is there a popular open source implementation which does not requires the ranking data?


texts:
I am looking at time series security attack data where a given IP can either be labeled as (1) attack or (0) no attack.
In total we will have thousands of IPs and roughly an equal number of attacks and non attacks. The data is rather noisy and every time series sequence can have a different length.

I am looking for advise on state of the art approaches to time series classification. I am past the stage of simple things like moving averages and I am looking for ways to improve my current methods or new things to try.

I have currently implemented a few different techniques:

K-nearest neighbor with DTW. I am successfully using http://www.cs.ucr.edu/~eamonn/UCRsuite.html which provides state of the art performance.
Logical shapelets (http://www.cs.ucr.edu/~mueen/LogicalShapelet/). This seems promising but have not been able to get any existing code base to work.

Can anyone suggest different technique to try? I have seen papers about discords and motifs but still need to investigate if they are relevant for my problem.


texts:
I'm currently working with a dataset with a wide range of document lengths -- anywhere from a single word to a full page of text.  In addition, the grammatical structure and use of punctuation varies wildly from document to document.  The goal is to classify those documents into one of about 10-15 categories.  I'm currently using ridge regression and logistic regression for the task, and CV for the alpha values of ridge.  The feature vectors are tf-idf ngrams.

Recently I've noticed that longer documents are much less likely to be categorized. Why might this be the case, and how can one "normalize" for this kind of variation?  As a more general question, how does one typically deal with diverse data sets?  Should documents be grouped based off of metrics like document length, use of punctuation, grammatical rigor, etc. and then fed through different classifiers?