texts: Extracting text from few areas on product label

texts:
I'm trying to achieve algorithm that will extract text from few areas(marked with red color) on label(similar to attached image) and QR code on a single photo taken with mobile camera so label may be in a little bit different position. I'm struggling with choosing and executing good enough solution to accomplish it. I was wondering about labeling images with marked boxes around my area of intersts and then put it into some CNN. I've encounter some interesting paper on text spotting but I'm not sure if it would be good solution for my problem.
I will be more than thankful for any help and suggestions. Thanks in advance :)


texts: How to define the input channel of a CNN model in Pytorch?

texts:
In pytorch, we use:
nn.conv2d(input_channel, output_channel, kernel_size)

in order to define the convolutional layers.
I understand that if the input is an image which has size $\text{width} \times \text{height} \times 3$ we would set the input_channel = 3. I am confused, however, what if I have a data set that has dimension: $3 \times 3 \times 30$ or $30 \times 4 \times 5$?
Which number should I use to define the input_channel for these?
Thanks in advance.


texts: 1x1 convolutions, equivalence with fully connected layer

texts:
I'm confused by the concept of equating a 1x1 convolution with a fully connected layer. Take the following simple example of a 1x1 convolution of 2 input channels each of size 2x2, and a single output channel.

The only way I can relate this to fully connected layers is to say that there are 4 fully connected layers, one for each location in the input feature map (inputs and outputs colour coded).

From what I can understand my interpretation is consistent with the Network in Network paper[Lin et al. 2013] which describe the 1x1 as being equivalent as cross channel parametric pooling

  The cross channel parametric pooling layer is also equivalent to a
  convolution layer with 1x1 con- volution kernel.

I have seen
 this one from Yann LeCunn equating 1x1 convolutions to a fully connected layer. And I have read this answer and I'm just not seeing the equivalence between a 1x1 convolution over an input volume and a single fully connected layer...

Any insight would be appreciated, if you can please relate back to the example above. Thanks!


texts: one hot encoding target variable in tree and non tree (knn) methods

texts:
I am learning about label encoders, one hot encoding etc applied to datasets for classification via KNN and XGBoost type trees. However, I am a bit confused as to whether the target variable should be one hot encoded or not.
In my case, I have a dataframe which has 500x10 size and the last column is the target (for prediction) country column while the rest of the 499 columns contain floating point values. So, an example row:
feature1 ..... fetaure499 target
1.1  ......... 40.1       Germany
6.1  ......... 265.7       USA
8.1  ......... 98,7      Singapore

In this case, what is the optimal way to encode the target for:
[1] KNN
[2] tree based methods.
My relatively naive understanding says that one hot encoding is useful for the feature space while for the target such as this, i.e list of countries labelencoding should be enough. Is this true for this case? Are there any situations I need to aware of when the target has to be label encoded?
Thank you.


texts:
Is there any way now to use TensorFlow with Intel GPUs? If yes, please point me in the right direction.
If not, please let me know which framework, if any, (Keras, Theano, etc) can I use for my Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller.


texts: How do I interpret loss in a neural network?

texts:
I am studying how to evaluate the performances of a convolutional neural network, and in particular I have seen that we have to look both at accuracy and loss. I don't understand why do we have to look also at the loss, and honestly I haven' t understand really clearly what the loss is. I have understood that it is something that we want to minimize and the lower the loss, the better the performances are.

But also I have seen that if the loss is to low we are overfitting, and also that there is a sort of relationship between loss and accuracy, but I have not clear what it really means.

To me this loss concept and how to interpret it both alone and with respect to the accuracy seems an abstract concept at the moment.

Can somebody help me clarify this concepts? Thanks in advance.

[EDIT] For example, I obtain this plot :

How do I interpret this?

From what I can see, it seems that the loss decreases both for train set and test set at the same rate, and we have that if this happens the model is good, otherwise we encouter overfitting or also underfitting. But I don't understand, why we have this? So why if the are close and decrease at same pace we are doing well, otherwise not?

Thanks.


texts: Supervised learning: classifying/learning a single discrete value

texts: Assume that I have a single feature dataset X comprising of say a 1000 samples.  The feature is an integer that may take up any value upto 2^32. Among these, one particular value alone is considered malicious. The target/lable vector flags out the malicious value as 1 and others as 0.  What would be the right approach to train a model that could learn this single value and classify it as 1 and all else as 0?


texts: Convert a repetitive list into Pandas Dataframe

texts:
How do you convert something like this:

A: 1
B: 2
C: 3
###
A: 5
B: 5
C: 6
###
A: 2
B: 5
C: 7

into a dataset where the first row would be the first section with

  A as column-1 B as column-2 and C as column-3

so we get this:

 A B C
 1 2 3
 5 5 6
 2 5 7


texts: How are big models like GPT-3 developed?

texts:
For models trained with a moderate amount of data, I would expect a the development workflow to include extensive hyper-parameter optimization through extensive retraining.
But how is this handled for large models, like GPT-3? I read that it costs tens of thousands of dollar to train the model. I would assume that during development, a smaller amount of training data is used to optimize the hyper-parameters, but I assume what is good for less training data is not necessarily good for higher amounts of training data. For example, the more parameters a model has, the more it might(?) suffer from overfitting for smaller data sets, but might achieve better performance than a smaller model when more data is involved.
But if the training gets very expensive and time-intensive, how can experimentation still happen?


texts:
How can I transfer two column features into pivot table on the following dataset

I have tried the aggfunc function but this fill the value either 0 or 1. I want to transfer the row as cell value.

Here is the dataset

content  Users
22       1196
23       1196
23       1216
16       880
20       880
20       1224
22       1245
23       1122
22       872

I want to transfer this dataset into this

Users   1    2
1196    22  23
1216    23  NaN
880     16  20
1224    20  NaN
1245    22  NaN
1122    23  NaN
872     22  NaN

I have tried it by using

df.pivot_table(index=["city"], columns="cuisine", aggfunc=lambda x: 1, fill_value=0)

  df.pivot_table(index=["users"], columns="content", aggfunc=lambda x: 1, fill_value=0)

But this fills the values either 0 or 1


texts: Can feature representation acquired by a same model but trained on different corpus be used on the same classification model?

texts:
For example, if I wanna do document classification with doc2vec embeddings, first I train the training set to get doc2vec embeddings, and fit the embeddings to a classification model; later on when I want to classify a new doc, do I have to use the previous doc2vec model to infer vectors to use on the classification model, or I can train a new doc2vec model to get vectors, since the dimension is the same?

From my point of view, I think the same doc2vec model has to be used, since even if we can get vectors of the same dimension via a newly trained doc2vec model,  they are not on the same feature space, here the doc2vec model would be a feature function and we has to use the same function to get features in a consistent feature space.

But doc2vec, or word2vec is based on distributed representation, so even if a new doc2vec model is trained on new documents, theoretically sentences with same semantics tend to be represented the same regardless of the model, so I'm confused.

Any suggestions or corrections would be appreciated!


texts: Random Forest prediction fails due to unseen Features

texts:
I have trained a Random Forest Model on some dataset and like to predict outcomes on other data which were not seen in training. When doing this, I get

ValueError: Number of features of the model must match the input. Model n_features is 12 and input n_features is 13

The problem is that there are some variables from the training data not existent in my prediction set. E.g. I capture the count of some feature via dummy variables D_0, D_1, D_2, D_3 indicating the number of occurences of D. I might have no D_2 in my training data but D_2 in my prediction data set.

What's best practice in such a case? I am planning to use this estimator repeatedly on future data and I can't know which features will be existent. Should I rather check for inconsistencies between both feature lists and manually correct those which do not overlap? In the above example, I'd code all occurences of D_2 to D_3 in order to align feature lists.


texts: Which part of the data should I use for Linear Regression?

texts:
I have a dataset composed of Carbon Monoxide level per 5 seconds (up to 20 seconds) as my independent variables/features and mg/dl (Glucose) as dependent variable. I want to use Regression to make a model for the CO level in breath vs Glucose in blood. Should I use the average of four instances (5s, 10s, 15s, 20s), should I have them all as features? Or should I manipulate it first?

My data goes like this

CO(5s) | CO(10s) | CO(15s) | CO(20s) | mg/dl

161      168       165       170       8.6


texts: Is it a good idea to use the mean and standard deviation of coefficients from other models as my prior in Bayesian Regression?

texts:
I have a dataset that I’ve been playing around with for school I have gotten very good results with a bunch of methods (Ridge, Lasso, ElasticNet, SVM, Bagging, Stacking and NN even)
Now I’m having a range of different coefficients of my predictors, is it a good idea to use them as my priors (I did so, I think the result has been ok) or should I use noninformative priors instead.
If it is a bad idea, could you explain why?
Or if different use cases, when does it make sense to use one or the other? I'm using Pymc3's sampler and GLM.
Thanks in advance for any pointers and explanations.


texts: chatbot encoder/decoder: why do we need to use chatbot answer as the decoder inputs?

texts:
I am looking into the chatbot tutorial at:

https://medium.com/predict/creating-a-chatbot-from-scratch-using-keras-and-tensorflow-59e8fc76be79

It uses sequence to sequence model with encoder/decoder to solve the problem:

encoder_inputs = tf.keras.layers.Input(shape=( None , ))
encoder_embedding = tf.keras.layers.Embedding( VOCAB_SIZE, 200 , mask_zero=True ) (encoder_inputs)
encoder_outputs , state_h , state_c = tf.keras.layers.LSTM( 200 , return_state=True )( encoder_embedding )
encoder_states = [ state_h , state_c ]

decoder_inputs = tf.keras.layers.Input(shape=( None ,  ))
decoder_embedding = tf.keras.layers.Embedding( VOCAB_SIZE, 200 , mask_zero=True) (decoder_inputs)
decoder_lstm = tf.keras.layers.LSTM( 200 , return_state=True , return_sequences=True )
decoder_outputs , _ , _ = decoder_lstm ( decoder_embedding , initial_state=encoder_states )
decoder_dense = tf.keras.layers.Dense( VOCAB_SIZE , activation=tf.keras.activations.softmax )
output = decoder_dense ( decoder_outputs )

model = tf.keras.models.Model([encoder_inputs, decoder_inputs], output )
model.compile(optimizer=tf.keras.optimizers.RMSprop(), loss='categorical_crossentropy')
model.summary()

I understand that the "chatbot question" needs to be the input of the encoder, and I also understand that "chatbot answer" needs to be the output of the decoder. However, I do not understand why the "chatbot answer" (decoder_inputs) has to be the decoder (as well as the entire model) input:

model = tf.keras.models.Model([encoder_inputs, decoder_inputs], output )

Could anyone please share their thoughts? Also, any paper related to such approach? What's the intuition behind using the "chatbot answer" as the inputs to decoder? Thanks!


texts: interpret diagnostics plots from classifierplots package in R

texts:
I have used a "adabag"(boosting + bagging) model on imbalanced dataset (6% positive), I have tried to maximized the sensitivity while keeping the accuracy above 70% and the best result I got where: ROC= 0.711 SENS=0.94 SPEC=0.21.

I have used classifierplots package in R for a diagnostic plot.
this is the plot:

I don't really understand how to interrupt the plots of positive instances per decile, prediction density and calibration. I have tried to find a comprehensive explanation with no success.

can you help me understand what the graphs mentioned above represent? I really want to improve the results.


texts: how to split available data into training and testing (Information security)

texts:
I was advised to ask my question here.

Recently, I made a post about finding suitable dataset for SIEM (Security Information and Event Management) systems. The goal was to work on classification and correlation to detect security attacks.

I decided to use the dataset from the Honeynet Project Challenges. The problem is this: I don't know if I should use the whole dataset set for my project, because if you look, for example, at the KDD99 dataset, it is devised into two parts: 10% for training and 90% for testing.

I have seen some researcher use dataset A for training and dataset B for testing, do you have any other ideas? I am really stuck at the part of training vs. testing

If my question is too broad. I don't mind some reading materials that will help me deal with my dataset.

Bests,


texts: Purpose of backpropagation in neural networks

texts:
I've just finished conceptually studying linear and logistic regression functions and their optimization as preparation for neural networks.

For example, say we are performing binary classification with logistic regression, let's define variables:
$x$ - vector containing all inputs.
$y$ - vector containing all outputs.
$w_{0}$ - bias weight variable.
$W=(w_1,...,w_{2})$ - vector containing all weight variables.
$f(x_i)=w_{0}+\sum_{i=1}x_{i}w_{i}=w_{0}+x^{T}W$ - summation of all weight variables.
$p(x_{i})=\frac{1}{1+e^{-f(x_i)}}$ - logistic activation function (sigmoid), representing conditional probability that $y_i$ will be 1 given observed values in $x_i$.
$L=-\frac{1}{N} \sum^{N}_{i=0} y_i*ln(p(x_i))+(1-y_i)*ln(1-p(x_i))$ - binary cross entropy loss function (Kullback-Leibler divergence of Bernoulli random variables plus entropy of activation function representing probability)
$L$ is multi-dimensional function, so it must be differentiated with partial derivative, being:
$$\frac{\partial{L}}{\partial{w}}$$
Then, the chain rule gives:
$$\frac{\partial{L}}{\partial{w_1}}=\frac{\partial{L}}{\partial{p_i}} \frac{\partial{p_i}}{\partial{w_1}}$$
After doing few calculations, derivative of the loss function is:
$$(y_i-p_i)*x_i$$

So we got derivative of the loss function, and all weights are trained separately with gradient descent.
What does backpropagation have to do with this? To be more precise, what's the point of automatic differentiation when we could simply plug in variables and calculate gradient on every step, correct?
In short
We already have derivative calculated, so what's the point of calculating them on every step when we can just plug in the variables?
Is backpropagation just fancy term for weights being optimized on every iteration?


texts: How to select features when performing classification with a dataframe of multiple columns?

texts:
I have a dataframe of 50000 observations and I want to perform a classification task. But I'm struggling with features selection. I have 89 columns, which after getting rid of some redundant features, I get down to 58 columns. Some of them are categorical, some numerical.

Any suggestions would be much appreciated.


texts: Finding the usage percent to perform predictive analysis for new users

texts:
Problem Statement-
I have to find the average feature usage all the users and the usage of user X to suggest if he should use the feature.

Example - On google home page out of all the user's avg 85% of times uses the search button.
If a user X comes on the home page and based on his activity we calculate that if only 35% of the time he click on the search button. We want to notify him about the benefits of the search button.

Data We have-

User | Landed on home page | used search button
 1        1000                  100
 2        100                   10
 3        1                     1
 4        10                    10
 5        10000                 1

Issues-

How to eliminate user 5 as  this is exceptionally making the data skewed. Median might be a solution for the use case. Is there any better suggestion?
How to find the Average usage, I mean 1/1 (User 3) and 10/10 (User 4) are not the same i.e. 10/10 (User 4) should have more value than 1/1 (User 3)
If these users(USER 1-5) stopped coming on the home page from 1 month, still the usage average would be the same- which is wrong. Since it was getting used earlier but not recently so the usage average should get decayed.

So my question in addition to above is that, am I approaching in the right direction? Is there any build-in algorithm or tool available for the problem statement? Any new approach is most welcome.


texts: Stateful LSTM for time-series prediciton - should each input sequence be shifted by 1 time step or by `sequenceLength` time steps

texts:
I am building an LSTM to attempt to learn the historic trend of some time-series data set (e.g. the daily share price of a company). When training my network, I am taking batches of size 1, each consisting of 25 sequential daily closing prices, where it then makes a prediction for the price on the 26th day.

Say the first sequence in an epoch used to train begins at $t=0$ and ends at $t=24$. I am using a stateful LSTM and hence taking the output state of one batch and inputting it to the state of the next batch, does this mean that my 2nd batch must be $[t=25, t=49]$?

Can I instead "slide" each batch by 1 time step so that the 2nd batch is $[t=1, t=25]$, or does this defeat the purpose of passing the state between batches?


texts: how to encode labels for relationship extraction

texts:
I am trying to extract relationship from text. So, lets take the following text
"He went to movie. but, they went to school"

So, here the relationship's are "He and Movie", "they and school".
How can i possibly encode these as labels. my initial thought was to mark all pronouns as 1's and nouns as 2's. But, is there a better way to do this?.


texts: How to validate clusters after calculating Gower distances and Ward's clustering in R

texts:
I am trying to apply Ward's clustering on a mixed types dataset, and wanna explain what I did (maybe helpful to others), and I have some questions regarding this analysis, mainly how to validate my clusters.

So, let me explain what I did in detail:

I started with a dataset containing 53 variables. These variables are either numerical or binary. The first variable contains the participant number, so will not be used in the clustering. I directly coded the categorical data into binary variables so that I didn't have to transform these variables.

Then I checked for normality of the variables for all numerical variables, using the Shapiro-Wilk test. Significance values below 0.05 will be log-transformed in the next step. Binary variables don't need a normality check.

I calculated distance using the daisy library. Here, I log-transformed the variables with a value lower than 0.05 in the second step with 'logratio', the other variables were appointed as asymmetric binary and symmetric binary. I read that Gower distance is an appropriate metric for mixed data types, and that's why I carried out this step this way.

gower_dist <- daisy(mydata[, -1], metric = "gower", type = list(logratio = c(4,5,6,7,8,9,10,11,12,13,17,24,25,26,44,45,49,50,51,52), asymm = c(14,15,16,18,19,20,21,23,27,28,29,30,31,32,34,35,36,37,38,39,40,41,42,43,46,47,48), symm = c(1,3)))

Then I used Ward's method for clustering, and I plotted a dendogram with the results.

hc1 <- hclust(gower_dist, method = "ward" )
plot(hc1) # display dendogram

This resulted in the following output:

I think that I am in the right direction with my analysis, but have some problems with finding the right number of clusters. I wanted to use pvclust() since it provides p-values for hierarchical clustering, and that is what I am interested in. However, it seems that this package is not usable when using Gower distances. Does anyone know another way to find p-values in R for Ward's clusters using the Gower distance?

Next to this, I have some small doubts on the correctness of my analysis. These are the following:

I read that using binary variables can be problematic in https://www.researchgate.net/publication/223532418_Hierarchical_clustering_of_mixed_data_based_on_distance_hierarchy. I assumed that with a high dimensional data set, this would not be a large problem, is that true?
According to https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4120293/ log transforming data does not help make data less variable or more normal and may, in some circumstances, make data more variable and more skewed. Is it still wise to apply log transformation to the variables? Also, I think that using Ward's method is wise to use for high dimensional datasets, is that true?
I wanted to use 'Partial data cluster analysis' to deal with missing values in the data (as explained in https://www.displayr.com/5-ways-deal-missing-data-cluster-analysis/), but I am not sure how I dealt with missing data the way I carried out the analysis now. This also doesn't seem to be documented.


texts: I created a boosting tree and got the probability for each tuple in my testing set. But I'm confused on how to combine each probability. Can someone tell me how to combine the probabilities?


texts: Developing KNN with custom distance matrices

texts:
I am trying to predict the extent of suitability (0.5 to 3.5) for a section of non-fiction books based on a few parameters. This is for a set of subscribers that we want to predict books suitability: based on score I plan to display star ratings: most suitable(2.5 to 3.5), mildly suitable(1.5 to 2.5), not relevant (0.5 to 1.5).
Example dimensions:

price: continuous

subject: categorical (humor, adventure, mystery, sports, non-fiction)

hardbound: binary

recency of book: categorical

We initially start with a default profile for all user and after getting feedback from him (as 1, 2 or 3 stars), build a customized profile.
Given that we will initially start with small volume of user data, looking to do a k-NN to identify nearest neighbors on the dimensions, and calculate suitability score based on weighted suitability of those neighbors.
Given the categorical data, I wanted to create custom distance matrices so that I could provide the relative distance between categories. (e.g. to indicate that humor and adventure are close preferences; sports and nonfiction are far apart)
Two questions:

Is my current approach valid- any feedback or pointers?
What specific k-NN implementation (package) would allow me to provide my custom distance matrices?


texts:
I am trying to fit a keras classifier on a data matrix X_train

dummy_y = np_utils.to_categorical(y_train)
dummy_y_val = np_utils.to_categorical(y_val)

def baseline_model(optimizer='rmsprop', init='normal', dropout_rate =0.0):
    model = Sequential()
    model.add(Dense(100, input_dim = X_train.shape[1], activation='relu', init=init))
    model.add(Dropout(dropout_rate))
    model.add(Dense(50, activation = 'relu', init = init))
    model.add(Dropout(dropout_rate))
    model.add(Dense(15, activation = 'sigmoid', init = init))
    model.compile(loss = 'binary_crossentropy', optimizer = optimizer, metrics =['accuracy'])
    return model

   estimator = KerasClassifier(build_fn=baseline_model, nb_epoch=200, batch_size=5, verbose=0)
   model = baseline_model()

model.fit(X_train, dummy_y, batch_size=32,  verbose=1, callbacks=None, \
     validation_data=(X_val, dummy_y_val), shuffle=True, class_weight=None, \
    sample_weight=None, initial_epoch=0)

That's what I get as an error :

  Train on 582 samples, validate on 290 samples Epoch 1/10

  ValueErrorTraceback (most recent call last)
   in ()
        1 model = baseline_model()
  ----> 2 model.fit(X_train, dummy_y, batch_size=32,  verbose=1,

  ....

  ValueError: Cannot feed value of shape (582, 32) for Tensor
  u'dense_input_28:0', which has shape '(?, 18760)'

NOTE : X_train.shape = (580,18760) and y_train has 15 classes


texts: Which algorithm to use for transactional data

texts:
I'm given a Dataset of transactions and asked to find insights for businesses. I'm extremely new to ML / Data science and have only been experiencing with KMeans. The dataset has the following features

merchant ID
Transaction date
Military time
Amount
card amount paid
merchant name
Town
area code
client ID
age band
gender code
province
average income 3 months
card value spending
card tapped

Ignoring NULL data, what type of analysis can I do on this data? I have done KMeans on if a customer will spend excessive amount (more than median amount).


texts: General programs/libraries for studying user search behavior?

texts:
Are there any general open-source programs or libraries (e.g., a Python library) for analyzing user search behavior?  By "search behavior", I mean a user's interaction with a search engine, such as querying, clicking relevant results, and spending time on those results.  I'd like something with the following properties - it doesn't have to be all of them, but the more the merrier:

Models individual user behavior (aggregate and time-based)
Models group user behavior
Simulates individual user behavior, given a model
Is easily extensible (to accept data input formats, user models, document models, etc., that end-users define)

Links are a plus!


texts: Can I use xgboost on a dataset with 1000 rows for classification problem?

texts:
I have used all types of classification algorithms on my dataset yet I couldn't improve my score no matter how I try.

So I've read about Xgboost classifier. So I was wondering is it practical to use xgboost on a dataset with around 1000 rows.

Please let me know.


texts:
I'm aware the LSTM cell uses both sigmoid and tanh activation functions internally, however when creating a stacked LSTM architecture does it make sense to pass their outputs through an activation function (e.g. ReLU)?

So do we prefer this:

model = LSTM(100, activation="relu", return_sequences=True, input_shape(timesteps, n_features))
model = LSTM(50, activation="relu", return_sequences=True)(model)
...

over this?

model = LSTM(100, return_sequences=True, input_shape(timesteps, n_features))
model = LSTM(50, return_sequences=True)(model)
...

From my empirical results when creating an LSTM-autoencoder I've found them to be quite similar.


texts: LSTM/RNN model fails on new test data - TFLearn

texts:
i'm fairly new to ML and at the moment i'm trying to develop a model that can classify spoken digits (0-9) by extracting mfcc features from audio files.

I trained the model on a data set that consists of 15 speakers and 2400 training examples (240 audio examples for each digit). After 3000 epochs the model has achieved 97% accuracy.

The problem is, when i record my own digit wavs, the model fails to classify them correctly. Why is this happening and what could i do to fix this issue? Is this an example of overfitting?

Extracting mfcc features

  wave, sr = librosa.load(wav_file, mono=True)
  mfcc = librosa.feature.mfcc(wave, sr)
  mfcc = np.pad(mfcc,((0,0),(0,80-len(mfcc[0]))), mode='constant', constant_values=0)

Parameters

learning_rate = 0.0001
training_iters = 150
batch_size = 64

width = 20  # mfcc features
height = 80  # (max) length of utterance
classes = 10  # digits

Network building

net = tflearn.input_data([None, width, height])
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, classes, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=learning_rate,
      loss='categorical_crossentropy')

Training

model = tflearn.DNN(net, tensorboard_verbose=0)

EPOCHS = 20
for i in range(training_iters):
    model.fit(X_train, y_train, n_epoch=EPOCHS, validation_set=(X_val, y_val),
          show_metric=True, batch_size=batch_size)


texts: Average of importance gain for a categorical variable

texts:
Suppose I have a set of M categorical variables, some of them with a different number of categories (for instance, var1 has five categories, var2 has three, etc).
I train an XGBoost model on a numeric target Y after having performed one-hot encoding on the M categorical variables, thus creating a set of dummy inputs.

When looking at the model results, I get a table of importance gain for the categories of each feature, meaning how important they are in the model. A toy result would look like this:

feature | category               gain
       var1 | cat3                 25
       var2 | cat1                 20
       var1 | cat5                 12
       var5 | cat6                 11
       var4 | cat1                  8
           ...                    ...

The main question I'm asking is the following:

In order to get an idea of how important a variable is overall rather than just one of its categories (for instance, how much var1 is important overall rather than just category cat3 of var1), does it make sense to take the average of all the importance gains for each feature as an importance indicator?

Probably the sum of such gains would not be correct as the features may have a different number of categories, but I'm wondering if the average of such gains might serve as an indicator of the importance of a particular feature overall.

I already looked at some questions like this without gaining much insight about this topic.


texts:
I am trying to train a neural network to find a mapping(embedding) to a lower dimensional space. I would like for my dataset, once mapped to the lower dimensional space, to appear gaussian-like distributed.

Does anyone have any idea what form of regularization I could use. I have been reading-up on KL-divergence but that seems to require a parametrized gaussian (rather than samples). Any help would be tremendously appreciated!


texts: If mean absolute loss is not differentiable, how it can be used in neural networks? which majorly are trained using back-propagation

texts:
If Mean Absolute Error (MAE) loss is not differentiable, how can it be used in neural networks? which majorly are trained using back-propagation

I am wondering if MAE is not differentiable how they can be used as a loss function.


texts: How to create an roc plot and calculate AUC for an svm (that does not return probabilities)?

texts:
I have some SVM classifier outputting final classifications for every sample in the test set, something like

1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1

and so on.

The "truth" labels is also something like

1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1

I would like to run that svm with some parameters, and generate points for the roc curve, and calculate auc.

I could do this by myself, but I am sure someone did it before me for cases like this.

Unfortunately, everything I can find is for cases where the classifier returns probabilities, rather than hard estimations, like here or here

I thought this would work, but from sklearn.metrics import plot_roc_curve is not found!

anything online that fits my case?

Thanks


texts: Understandable and explainable machine learning model

texts:
I want to find formula for best financial portfolio.

Inputs: Historical fundamental data for last 15 years. For 3000 companies for every quatal we have things like market cap, cash flow, liabilities, assets, revenue etc, every one of these inputs is a time series with data points for every quartal (i.e. 4 times in a year).

Goal: I want to find a formula that given the 3000 companies would allow me to rank (sort) it from best to worst. And then I can pick let's say top 10 of those as the best portfolio.

Example, how human can solve this task: study those historical financial data for couple of years, and came up with something. Like this very simple formula rank = revenue[last]/total-assets[last] + revenue[last]/market-cap[last] (looking for highly profitable and not too much overpriced companies).

Machine Learning way to solve it: feed those data points to Neural Net, run it and hope it will learn something after a while.

The problem is that we can't analyse the the neural network result because it's a black box. And I want to see the final formula in more or less compact form and be able to understand what it does.

What other technics other than Neural Nets can be used? The model has to be sophisticated enough to be able to work with the time series as inputs.


texts: How to programmatically differentiate between extractive and abstractive summarization in NLP?

texts:
We are using different pre-trained models in python transformers library, to generate summaries(both extractive and abstractive). So is there a programmatic way, based on the output summary, we can classify it as abstractive or extractive?
One method I think of is using the rouge python library to compute rouge score with respect to original input text(not human reference summary), which will have rouge(specifically LCS) precision score as 1.0 for extractive(since all the words present in summary will be present in original text). Whereas for abstractive summary some words present in summary will not be in original text and hence precision score will be less than 1.0. Please let me know if this is the right approach? Or is there any other way to differentiate between extractive and abstractive summaries?


texts:
I'm using Google Cloud NLP Entity Recognition to recognize Persons/People, when I paste a chunk of text, it correctly provides PERSON information:

In the example:

"Google, headquartered in Mountain View, unveiled the new Android phone at the Consumer Electronic Show.  Sundar Pichai said in his keynote that users love their new Android phones."

Sundar Pichai is correctly identified as a person, but also the word users.
How can I differentiate real names vs words which refer to persons?
I have seen that for popular people there is metadata like Knowledge Graph mid or Wikipedia articles, but for others there is no reference, (Example: from "Susan Fowler" recent Uber scandal)
Any ideas/pointers will be greatly appreciated.


texts:
My understanding is that a connection between two neurons has a weight, but a neuron itself does not have a weight.

If connection c connects neurons A to B, then c has a weight w, but A and B don't have a weight. w determines if A has a strong influence on B or a weak influence on B.

But this Wikipedia article says:
"The connections between artificial neurons are called ‘edges’.
Artificial neurons and edges typically have a weight that adjusts as learning proceeds."

https://en.wikipedia.org/wiki/Artificial_neural_network

The 2nd sentence seems to imply that a neuron has a weight, too.

Does a neuron really have a weight?


texts:
My Model is defined as below:
## defining the model
batch_size = 1

def my_model():

   input_x = Input(batch_shape=(batch_size, look_back, 4), name='input')
   drop = Dropout(0.5)

   lstm_1 = LSTM(100, return_sequences=True, batch_input_shape=(batch_size, look_back, 4), name='3dLSTM', stateful=True)(input_x)
   lstm_1_drop = drop(lstm_1)
   lstm_2 = LSTM(100, batch_input_shape=(batch_size, look_back, 4), name='2dLSTM', stateful=True)(lstm_1_drop)
   lstm_2_drop = drop(lstm_2)

   y1 = Dense(1, activation='relu', name='op1')(lstm_2_drop)
   y2 = Dense(1, activation='relu', name='op2')(lstm_2_drop)

   model = Model(inputs=input_x, outputs=[y1,y2])
   model.compile(loss='mse', optimizer='adam',metrics=['mse'])
   model.summary()
   return model

model = my_model()

It is a Stateful LSTM model with batch size =1. My model.fit looks like this :
# Train the model
history = model.fit(
    x_train,
    [y_11_train,y_22_train],
    epochs=1,
    batch_size=batch_size,
    verbose=0,
    shuffle=False)

model.reset_states()

My model runs well and outputs results. But I am unable to increase or alter the batch_size as flexibly as we could do when the model is in stateless condition.
As in for bigger size of dataset the model seems to be training forever as the batch_size here is just 1. And as we know we can't randomly put any batch_size value for stateful LSTM as it needs to be a divisible factor.
I have gone through some blogs which describes changing batch_size by using different batch sizes for training and predicting using the get_weights() and set_weights() functions in the Keras API ref: https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/, still it seems to be that the batch_size that is being used here is less.
My question is : Cant we use batch size like 32, 64, 128 in stateful LSTMs? If yes then how do I implement it in my above given model, if not then what are the alternatives?
Looking for valuable suggestions.
Post Edit
In stateful LSTM model.reset_states() should be after every epoch, Hence I set the resetting of the states after each epoch in the following way:
for i in range(100):
    start = time.time()
    history = model.fit(x_train, [y_11_train,y_22_train], epochs=1, batch_size=batch_size, verbose=0, shuffle=False)
    model.reset_states()

    print("Epoch",i, time.time()-start,"s")


texts: Simmultainiously calculting loss from target interdependend metric

texts:
Is there a way to incorporate multiple targets into one loss?

Currently, I work with the Sequential() API, I guess this won't be sufficient....

I work with area predictions as targets. Each sample of the Dataset has a finite area to allocate. my regression often overestimates the total sample area... I want to implement an area restriction into a loss. This would be the sum of all true target values for a sample. I want to separately penalize the overestimation of the area.

I know Keras does automatically average losses like this :

1.trained on batch -->

2.for all targets: loss (target) -->

3.mean loss over all targets

But i need something like this:

1.trained on batch -->

2.loss(sum all predicted areas, sum of true areas)& for all targets: loss (target)-->

3.mean loss overall targets and areasumloss

The current solution was a feature engineering approach :

total area per sample got added to the feature list but did not enhance the result. The sum of individual areas is still overestimated

Is there a way to do this with the Sequential() API or do i need to use the Functional API?


texts: Subdivide a numerical vector with a normal distribution

texts:
I have a numerical array of prices values. I'd like to do classification on this parameter, so I'd like to create a certain number of classes with the same granularity. I'd like to create a generalized function that given the array and the number of classes i'd like to have it automatically return the price intervals for those classes.
Currently this is my price attribute:

df_cleaned.price.describe()
>>>
count    122668.000000
mean      11253.349594
std        7856.513917
min        1010.000000
25%        4995.000000
50%        8995.000000
75%       15965.000000
max       34991.000000
Name: price, dtype: float64

And I created a function manually to create 6 classes and it looks like this:

def normalize_price(df):
    cond = [
        (df['price'] >= 1000) & (df['price'] <= 4999),
        (df['price'] >= 5000) & (df['price'] <= 8999),
        (df['price'] >= 9000) & (df['price'] <= 15999),
        (df['price'] >= 16000) & (df['price'] <= 24999),
        (df['price'] >= 25000) & (df['price'] <= 34000),
        (df['price'] >= 34001) & (df['price'] <= 40000)
    ]

    choice = [
        1000,
        5000,
        9000,
        16000,
        25000,
        35000,
    ]

    df['price'] = np.select(cond, choice, df['price'])
    return df

Can anyone help me improvint this function? pls thanks


texts: I know that clustering can be used for unsupervised learning and some people told me for many more techniques, but I was left with no answer, when I asked for what else clustering is used.


texts:
I am trying to do the following as shown below.

    Input       Output
Letter Number    A B C
A      1         1 1 1
A      2         2 2 2
B      1           3 3
B      2             4
B      3
C      1
C      2
C      3
C      4

I have wrote the following code which works just fine.

import pandas as pd

df = pd.read_excel('Test.xlsx')
df = df.pivot(columns='Letter', values='Number')

list = []
for col in df.columns:
    col = df[col].sort_values()
    col.index = range(len(col))
    list.append(col)
ndf = pd.concat(list, axis=1, sort=False)
ndf = ndf.dropna(axis=0, how='all')
print(ndf)

Is there any other alternative way of doing this without using loops? Any help would be very much appreciated.


texts: What is done in practice to speed up batch computation for deep learning by standard software packages?

texts:
Something that confuses me with machine learning in practice is that when you send in a batch of examples, you are computing the value of each example independently.
And then when you backprop, you are computing the gradient also independently.
That seems to be a great deal of wasted resources. But obvious in practice, when training a network on Pytorch or Tensorflow, this is not done sequentially, and it scales well even if you change your batch to a relatively high number in the hundreds. How is this process sped up in practice?
I don't know too much about "parallel computation" so I am not confident to say that this is some magical parallel computing. I know that computation can be done in parallel on multi-thread CPUs, multi-core CPUs, hyperthreading, GPUs and FPGAs, or some non-trivial good old linear alg. Which one is actually used in practice?


texts: How to do scatterplot Visualization of Text Classification

texts:
I am working on my first ever ds problem on classification. I have got end to end process but not much visualization except bar charts and pie charts.

I want to see how well classes are distinguished in sample dataset itself to find anomaly in sample data or to see how many outliers sample have.

I see many nice scatterplot clustering done in many articles, pappers and books which are mostly based on numerical features. Something like following taken from scikit learn : http://scikit-learn.org/stable/auto_examples/datasets/plot_random_dataset.html

However, most of those are for numeric features. How can I draw similar plot for textual features?

One way I can do it is converting all features into vectors; plot all the indexes(token indexes) on x-axis and their respective metric (TF or TF-IDF) on y-axis? But then there are so many features!?

Also, how can I analyze raw sample data in such clustering format without transforming them into vectors. I can do tokenization and some basic normalization.

For example  my sample data is
text, category

"chiense restarant near me", chinesefoodlover
"japanese food near me", japanesefoodlover

I can break it down to

[chiense, restarant, near, me], chinesefoodlover
[japanese, food, near, me], japanesefoodlover

But then how to draw clusters of classes from there.


texts:
I am trying to learn LSTM's, and struggling a bit with the structure and the inputs/outputs of LSTM layers. Say I have a network definition like this:

timesteps = 50
features = 10
inp = Input(shape=(timesteps, features,), name="input_layer")
x = LSTM(128, return_sequences=True, name="lstm1")(inp)
x = LSTM(64, return_sequences=False, name="lstm2")(x)
x = Dense(128, activation="relu", name="dense1")(x)
x = Dense(64, activation="relu", name="dense2")(x)
x = Dense(1, activation='sigmoid', name="output")(x)
model = Model(inputs=inp, outputs=x)
model.compile(loss='binary_crossentropy', optimizer='adam')

I can find plenty of resources on the internet on the internal structure of the LSTM cell. For example, this has the figure with one input and some outputs.
But I would first like to understand at a higher level, what inputs and outputs an LSTM layer gives in different configurations.

Lets say I have input data shape of (1000,50,10).
So 1000 observations, each with 50 timesteps for 10 features.

Questions:

My understand is, the first layer ("lstm1") would take the input as shape (50, 10).
One at a time for each of the 1000 observations.
How are the 10 features input into the LSTM layer?
Is it similar to a Dense layer, so each input feature goes into each LSTM cell with some weight?
This answer seems to indicate so, but I do not quite understand what are the multiple inputs for each LSTM "unit" in that answer (perhaps even what "unit" is).

And the internal workings.
Does "lstm1" here process all the 50 timesteps at once and then produce an output?
After it is done for all 50 timesteps the next layer gets its input from this output?
And what is the shape of that output? Is it shape=(50, 128)?
So is the set of output "features" in this case the number of units in the LSTM layer?
And because of "return_sequences=True", it also outputs the intermediate value at each timestep to produce the shape=(50,XXX) outputs?

This leads to my question on the "lstm2" layer.
Using "return_sequences=False", does it now output a single array of 64 values? So shape=(64)?
No value for every timestep but rather only one per unit at the end of the "unrolled" cell loop?


texts: Why we add a constant value column in our DataFrame sometimes?

texts:
Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1.

I need to know why we do so. And also what will happen if we don't use it.

Thank you.


texts: the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?


texts: Very low probability in naive Bayes classifier

texts:
I have designed NB classifier from scratch in python for binary classification problem. There are total 220 records out of which 85 records belongs to 'Yes' class and 135 to 'No' class. My classifier is giving 88% accuracy.

So, whenever I calculate the posterior probability of one sample that belongs to 'Yes' class it is very low in terms of numbers. For eg., I'm making a prediction that whether the batsman is rising star (i.e. the probability of sample belongs to 'Yes' class ).

Here The posterior probability of being rising star i.e. P(RS) is very low in numbers something like 2.33E-8. But the posterior probability of being not rising star is also very low similar in the range of E-8 to E-16. Some features I'm using to calculate posterior probability are also small values in the range of 0.1 to 0.01.

My question is how to represent posterior probability i.e. P(RS) in terms of percentage. Like P(RS)=90%.

PS: I googled this problem and tried log method which returns a negative value.


texts: When should I use Cox propotional hazard model, KAPLAN-MEIER’S ESTIMATOR or NELSON-AALEN ESTIMATOR for survival analysis?

texts: I know there is a lot of journal and article about this I have read about it and I don't understand a lot of thing but mainly I don't understand the difference and when to use it.


texts: Is there any data set containing list of windows software and the category each one of them belongs to?

texts: I wanted a dataset which lists many windows softwares according to their categories  for a project. eg: anaconda: software development, roblox: gaming, VLC: video player etc. I searched but could not find anything similar to this. Please link me to one of them if you are familiar to any such dataset.


texts: Data scaling for large dynamic range in neural networks

texts: The usual strategy in neural networks today is to use min-max scaling to scale the input feature vector from 0 to 1. I want to know if the same principle holds true if our inputs have a large dynamic range (for example, there may be some very large values and some very small values). Isn't it better to use logarithmic scaling in such cases?


texts: How do I determine which variables contribute to the 1st PC in PCA?

texts: Given the coefficients of PC1 as follows for each variable (0.30, 0.31, 0.42, 0.37, 0.13, -0.43, 0.29, -0.42, -0.11) which variables contributes most to this PC? Does the sign(+/-) matters or considering the absolute value is enough?


texts:
I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term "weight" being used a lot, although it is a little confusing.

I know, that before the inputs are summed and passed to activation functions, they are separately weighted, after some research, i found out that the purpose of the weight function was to:

ensure orthonormality
avoid data loss

I know that two inputs are only orthonormal if they are orthogonal unit vectors, if they are orthogonal then their dot product is always 0.
Dot product:

where theta is angle between two unit vectors and absolute value of a is norm.

For example, if two unit vectors are perpendicular to each other, then they are orthonormal and their dot product is 0.

But what does this have to do with data loss? I've also heard that sometimes the value of input might be zero, and we know that multiplication by zero outputs zero.

So what is the real purpose of the weight in neural networks? and what does it have to do with orthonormality? For example, what would be the purpose of weights in linear regression?


texts: Skip Gram Negative Sampling with Logistics Regression

texts:
Given a training sentences as follow form document:
... lemon, a tablespoon of apricot jam a pinch ...

Word apricot choose to be target word as t with windows size 2

Training sample with both negative and positive samples looks like as follow
Positive samples:
apricot tablespoon
apricot of
apricot preserves
apricot or
Negative samples: (1 positive sample with 2 corresponding negative samples)
apricot aardvark  apricot twelve
apricot puddle  apricot hello
apricot where  apricot dear
apricot coaxial  apricot forever
The likelihood function (single word):
$$log\frac{1}{1+e^{-c\cdot t}}+\sum_{i=1}^k log\frac{1}{1+e^{n_i\cdot t}}$$
1.K is 2 since we have 2 negative sample for each positive sample
2.t is a vector of words apricot
3.c is a vector of words within the windows size such as apricot tablespoon in positive sample
4.$n_i$ is a vector of words in negative sample for each positive sample
Questions:
Here 's my questions:
1. How to fit the negative and positive sample into vector $c$ vector $n_i$ and vector $t$?In deep learning version it is one hot encoding
but how about in this version?
2. Any workable example with small datasets?
3. How do I know my training result of vector t is correct? Since I prefer to study this method with only a very small datasets and this method need a lot of training sample and training time like a weekHowever my aim is to learn about this method not for the word embedding
It will be great for anyone offer help to my question I am not only asking for help but also share what I learn


texts: How to downsample a dataset with constraints in Python?

texts:
I have a dataset that has columns a, b, and others. Both a and b are numerical values, for example,
       a b c d e ...
row_0  2 5 .....
row_1  6 3 .....
...

I am hoping to downsample the dataset from total rows n to n_d, while changing the ratio of sum(a)/sum(b) to a given value, e.g., sum(a_d)/sum(b_d) = 1.5. Is there a package or method I could use to achieve this goal?


texts: Pass data to CNN with multiple outputs in keras

texts:
For some reason am I getting an uexpected output dimension for my classification network.

The network has 18 inputs of shape (45,5,3) - data formatted as (samples,45,5,3)

And the output is a vector of length 15 - one class for each third of 45.  The extracted classes comes from a pool of 145 classes. - data formatted as (samples,15)

My network looks like this:

#stride = 2
#dim = 40
#window_height = 5
#splits = ((40-5)+1)/2 = 18

kernel_number = int(math.ceil(splits))
list_of_input = [Input(shape = (45,5,3)) for i in range(splits)]
list_of_conv_output = []
list_of_max_out = []
for i in range(splits):
    list_of_conv_output.append(Conv2D(filters = kernel_number , kernel_size = (int(splits-3),3))(list_of_input[i]))
    list_of_max_out.append((MaxPooling2D(pool_size=((2,2)))(list_of_conv_output[i])))

merge = keras.layers.concatenate(list_of_max_out)
print merge.shape
reshape = Reshape((15,324))(merge)

dense1 = Dense(units = 1000, activation = 'relu',    name = "dense_1")(reshape)
dense2 = Dense(units = 1000, activation = 'relu',    name = "dense_2")(dense1)
dense3 = Dense(units = 145 , activation = 'softmax', name = "dense_3")(dense2)
model = Model(inputs = list_of_input ,outputs = dense3)

But for some reason am I getting an error when I passing my output data.
It is currently stored as numpy.ndarray of shape (16828,15) and I get an value error stating:

Error when checking model target: expected dense_3 to have 3 dimensions, but got array with shape (16828, 15)

Why is it expecting 3 dim rather than 2 dim?

The model summary clearly indicates that the output dim is (15,145) as I also would expect?  15 classes from a pool of 145 classes. right?

Model summary:
https://pastebin.com/27YTQW2m


texts: How do I build a model to improve CTR on campaign?

texts:
I am trying to build a propensity model for a client to increase the CTR.
Client has the list of people who clicked in the previous campaigns but doesn't have the data on the list of people who didn't (neither they have the complete list of who all were the user to whom the campaign was sent). In short, I only have positive labels.
I am currently thinking of modelling this as one-class classification and try Algos like OneSVM, Isolation forest, etc.
Questions:

Has anyone tried Propensity modeling via OneClass Classification? If yes how were the results?
What are some of the cons of modeling it this way?


texts: Plots of data using DBSCAN algorithm not making sense

texts:
I am using clustering for my data. Since the DBSCAN algorithm will also tell me an estimate of clusters that I can use, I have used DBSCAN. I have tried for the eps=[0.123,1,2] and min_smaples=[2,10,...60]. The print satatement in the below code prints 714 which is equal to the number of data-samples(rows). The code looks like:

dbscan = DBSCAN(eps=1, min_samples = 4)
clusters = dbscan.fit_predict(df)
print(len(clusters))
plt.scatter(df.iloc[:, 0], df.iloc[:, 1], c=clusters, cmap="plasma")
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")

Sample plots for some of the different min_samples is shown below:

The parameters for the above plots are given below in the same pattern.

EPS=1, MIN_SAMPLES=2, EPS=1, MIN_SAMPLES=10
EPS=1, MIN_SAMPLES=20, EPS=1, MIN_SAMPLES=40

None of these plots are making sense to me from a clustering perspective. From this, I am forced to conclude that I cannot use clustering for the given data or I am doing it wrong. So, I need help with insights into the weird appearance of the above plots.

Any help is appreciated


texts: Why are bigger embedding vectors not necessarily better?

texts: I'm wondering why increasing the dimension of a word dimension vector in NLP doesn't necessarily lead to a better result. For instance, on examples I run, I see sometimes that using a pre-trained 100d GloVe vector performs better than a 300d one. Why is this the case? Intuitively a larger dimension should become almost like a one-hot encoding and be in some more "accurate," no?


texts:
Could someone please explain to me how and why can we go from equation $4.3$ to equation $4.4$:

$$\hat{c}= \arg\max_{c \in \mathcal{C}}P(c|d) = \arg\max_{c \in \mathcal{C}}\frac{P(d|c)P(c)}{P(d)}\tag{4.3}$$

$$\hat{c}= \arg\max_{c \in \mathcal{C}}P(c|d) = \arg\max_{c \in \mathcal{C}}P(d|c)P(c)\tag{4.4}$$


texts: How to divide 7 non-continuous result values into 5 legend categories?

texts:
I'm making a map which displays score results for river reaches. The score results are integers from 1 to 7.

Now I want to make a legend and color code my results. However, I don't want to use too many color shades as the human eye/brain is not really able to distinguish clearly between more than +/- 5 shades.

So my question is: how do I divide my 7 (non-continuous) score results into 5 categories for my legend?

I thought of the following options:

legend 1:

result = 1 - 2
result = 3
result = 4
result = 5
result = 6 - 7

legend 2:

result <= 1.4
1.4 < result <= 2.8
2.8 < result <= 4.2
4.2 < result <= 5.6
result > 5.6

My issue with the first legend is that I would be visually tricking my audience into thinking that the extreme values are more prevalent than they actually are.
My issue with the second legend is that my result values are not continuous, so in reality the second legend looks like this:

result = 1
result = 2
result = 3 - 4
result = 5
result = 6 - 7

So again, my data will be visually skewed and deceiving my audience.

What are proper ways to handle this type of situation? Or any similar situation for that matter (for instance for 9 or 11 or 13 result values)?

Thanks!


texts: What should be the requirement for training data in order to obtain a good regression model using neural network?

texts:
I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet.

Data:- I have 3000 samples with 7 input features (consisting of 5 normalized continuous variables and 1 binary variable recoded using one-hot encoding). I have 2 outputs per sample.

So I have trained my model and I am getting around 95% accuracy. I feel it's too good to be true. Is it?
I have normalized my training data and used the mean and standard deviation of the training data to normalize the test data. Right approach, right?
I have read that neural network is after all a method to do interpolation, so how would it behave on the data with features with values which are not within the range while it's being trained?

I ask this because when I take a $80\%$ random sample out of the available data, it's possible that for a given feature (continuous variable) $X_1$ out of $X_1, X_2, ..., X_7$, $X_1 \in [a,b]$, where $a,b$ are some scalars. However, the other $20\%$ test data might have $X_1$ values $<a$ or $>b$. Is my trained model expected to give correct results then?

Should I treat each output independently and create models, $i.e.$ weights separately (2 in my case) or should I find weights with the joint output? (The outputs may have different scales, so should I normalize them too in this case?)

How different would be the behaviours of both the models?

I believe the questions are preliminary but it would be really helpful to get my basics clear.


texts:
I am new for data science subject and I am trying to learn on my own. Please bear with me if this is a stupid question. I have a dataset that has no classes. The data set comprises people's activity on Office 365 and my goal is to predict whether the person is experiencing pressure or not using machine learning algorithms, but the dataset does not have classes as you all would expect. Can you please help me how I can solve this problem? Is there any possible way to create classes prior to applying ML models on the data?

Thanks in advance.


texts: Dimension of output in Dense layer Keras

texts:
I have the sample following model
from tensorflow.keras import models
from tensorflow.keras import layers
sample_model = models.Sequential()
sample_model.add(layers.Dense(32, input_shape=(4,)))
sample_model.add(layers.Dense(16, input_shape = (44,)))

sample_model.compile(loss="binary_crossentropy", optimizer="adam", metrics = ["accuracy"])

IP for the model:
sam_x = np.random.rand(10,4)
sam_y = np.array([0,1,1,0,1,0,0,1,0,1,])
sample_model.fit(sam_x,sam_y)

The confusion is the fit should have thrown an error of shape mismatch as the input_shape for the 2nd Dense Layer is given as (None,44) but the output for the 1st Dense Layer (which is the input of the 2nd Dense Layer) will be of shape (None,32). But it ran successfully.
I dont understand why there was no error. Any clarifications will be helpful


texts:
I'm trying to create a longitudinal data graph using the raincloud R package. My df consists of a repeated measure design. I want to plot how each participant changed from time 1 to time 2 to time 3 by using the line feature (see image below but only 3 time points.
I followed the tutorial online but it seems like I am not able to plot participants. Here is the code that I follow:
 raincloud_2x3_vertical <- raincloud_2x3_repmes(
  data = df_2x3,
  colors = (c('dodgerblue', 'darkorange', 'dodgerblue',
              'darkorange', 'dodgerblue', 'darkorange')),
  fills = (c('dodgerblue', 'darkorange', 'dodgerblue',
             'darkorange', 'dodgerblue', 'darkorange')),
  size = 1,
  alpha = .6,
  ort = 'v') +

scale_x_continuous(breaks=c(1,2,3), labels=c("T-1", "T-2", "T-3"), limits=c(0, 4)) +
  xlab("Time") +
  ylab("Score") +
  theme_classic()

raincloud_2x3_vertical

This gives me the following:
So, how can I make a figure that shows the participants as the points and their scores in time 1, 2, and 3 connected with lines?
Thanks!


texts: Structure of NN for input data with drop out

texts:
In financial markets, there is a simple problem of trading calendars varying across different countries. For example, Sweden observes Sweden National Day and Norway has Whit Monday. Typically, what happens then is that a time-series in the equity market that was closed for a holiday 'catches up' the next day when the market is open again.
For example:
+-----------------+------------+------------+------------+------------+
| Date            | SEK 1      | SEK 2      | NOK 1      | NOK 2      |
+-----------------+------------+------------+------------+------------+
| Date 1          | + 0.2%     | +0.4%      | +0.3%      | +0.6%      |
| Date 2          | + 1.1%     | +0.7%      | +0.3%      | +2.1%      |
| Date 3          | -3.2%      | -2.9%      | NaN        | NaN        |
| Date 4          | +0.1%      | -0.2%      | -2.8%      | -1.6%      |
+-----------------+------------+------------+------------+------------+

The objective of my model is to adjust the NOK returns for dates 3 and 4 since they have been distorted by the NOK holiday on Date 3. To do this I will use as many good dates as I have, such as date 1 and build a large dataset by randomly dropping out some data, adjusting the following the date and using the dropped out adjustments as the input and obviously I know the supervised output as the real data.
I feel that this would be well suited for a neural network, but I have never constructed one before with dropped out input.
Would it be appropriate to just build network where input data-points that are missing have no impact on the weights for that round? Are there any neural networks (or all) that do this by default?


texts: Predictive modeling on big data set that can't fit into memory

texts: I am trying to build a Decision-Tree model on top of a dataset that is about 10G in size on my local computer. However, I only have 8G memory. What I am doing now is just random sampling certain subset of the data, trying different model parameters and checking the prediction performance with the scikit-learn package. Still I need a final model based on the model parameters from those random sampling results, but obviously I cannot build a model on top of the full dataset. What can I do with this problem? I am a newbie of machine learning, any suggestions are welcome. Thanks!


texts: Which classifier is efficient in dealing with test query which belongs to no trained class?

texts:
Suppose classifier trained with 5 class, and input query content does not belong to any of the trained class data.

Naive bayes provides and random class as a result here. Which classifier deals best in such scenario?


texts: How can I replace NaN values with other values in the CustomerID column if they have a same InvoiceNo?


texts: How to interpret the value of categorical cross entropy?

texts:
Categorical cross-entropy loss is usually used in settings where the target in one-hot encoded. Suppose I have a problem where there are 300 possible outcomes, and thus my final fully connected layer will have 300 neurons and the output (after softmax) is expected to be probabilities for each class such that their sum is equal to one.
My question is: how to interpret the value of the categorical cross-entropy loss? What does it mean to have a categorical cross-entropy loss of 5 or 0.9 or 0.1?
Does the interpretation of the loss get affected with the number of possible classes?


texts: How to compute the prediction interval for unseen data?

texts: I want to calculate the prediction interval of individual predictions without knowing what it's target value is gonna be. For example, I am doing a blood glucose prediction of individual patients and I want to predict what his/her blood glucose value is going to be after 1 hour. I want to compute where that value is gonna lie. (Predicted value: 250, lies within : 220 - 270 or something) And I don't know what his/her actual value is going to be in next hour. So is it possible to achieve this?


texts:
I'm confused with the functionality of tensorflow classes tf.distribute and tf.train.ClusterSpec.
As per my understanding, tf.distibute has many classes to facilitate distributed computing  either within a machine or using multiple machines.
Similarly, we can achieve this thing with tf.train.ClusterSpec.
Please provide me more insight in terms of, how they are different, what is the relevance of each classes, when should we use a particular one and so on.


texts: Extracting domain specific terms from a huge hard-coded list from a text

texts:
I know. The title sounds like I haven't googled my problem, but trust me, I did. Maybe my problem has a name and I haven't found it yet. Hoping you can help me wrap my head around it.

What I want to do is, given a text, extract all terms from a specific domain. For simplicity let's say, given a list of hard-coded animals, I want my model to extract from an input text all of the animals that are present in the list. Why would I use IA for this? Well e.g. I want to distinguish negations, so I don't want to extract "lion" if the text says "no it a'int a lion m'am", also I want it to extract "lion" if the text says "lioness", without having to make a huge list of synonyms.

I think my problem is kind of like NER, but not exactly, right? I don't just want to say "lioness" = animal, I want it to map it to "lion" also. So this is basically a classification problem, right? I want it to label all of the animals found in the text, basically label "lioness" = lion. But the problem is I may have 100k+ thousands of possible labels, so should I just train a classification model with 100k+ possible outputs? Doesn't seem right.

Thanks in advance for any feedback.


texts: Classification loss function: how to implement individual weights for each observation and class

texts:
The problem I have to solve is a classification problem. The costs of a misclassification are very different (but known) for the various observations, so I plan to include them by assigning weights to each observation accordingly. My issue is that additionally, the costs of misclassification are different for different classes (and these differences depend on the observations). So in theory, I would need to incorporate into the loss function weights $w_{ij}$ for each observation $i$ and class $j$. But I have no idea how to do this for example for a neural network in keras (or with any other classifier I didn't build from scratch myself).

Any ideas on how to do this would be greatly appreciated!


texts:
I am new to Reinforcement Learning, and I am having trouble understanding the difference between Value function and Q-value

Here is my current understanding: Q-value is just a value for a particular action taken, and the Value function of a state sums over the Q-value of every action possible in that state.

Would this description be accurate?

Thanks!


texts: What came first? Backpropagation or Sigmoid?

texts:
Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function.

When did the sigmoid function become so popular in NNs?


texts: Combining textual and numeric features into pre-trained Transformer BERT

texts:
I have a dataset with 3 columns:

Text
Meta-data (intending to extract features from it, then use those i.e., numerical features)
Target label

Question 1: How can I use a pre-trained BERT instance on more than the text?
One theoretical solution suggests having BERT fed the text and another neural network with the numerical features fed into this one, then aggregating their output, into another neural network.

Is that the most efficient approach?

Question 2: How can you connect neural networks?

You get the output from each, but then what?

You get classification output from BERT, you get classification
output from MLP based on numerical features.

You concatenate these and feed them to another MLP, and you get the
final prediction? Wouldn't that last prediction be less robust ?

In other words, does the last MLP encapsulate the other 2 networks?

If so, what happens if BERT predicts on 90%, but the first MLP just
50%, will we get a lesser outcome?

Question 3: Any tips on how to implement this in pytorch?


texts: Does pruning a decision tree always make it more general?

texts:
If I prune a decision tree, does that make the resulting decision tree always more general than the original decision tree?

Are there examples where this is not the case?


texts: deep learning output data in keras fit method

texts:
if I have following data set and I want to predict all three feature How should I prepare the output data ?

 a      b    c
2265.0  8   21.3515625
2264.5  8   21.3515625
2268.0  8   21.3046875
2266.5  8   21.328125
2260.5  8   21.328125
2259.0  8   21.328125

I know Keras takes 3d input. Therefore the input needs to be reshape. There is a timestamps as well.So the output/label would be all except the time stamp in the dataset ?


texts: Detect non-digitized university degree's authenticity by Deep Learning

texts:
How can we detect by the look and feel of non-digitized university certificates’ authenticity through machine learning?

I am working in this project right now, where I have to detect the authenticity of non-digitized university certificate, I know how can we do it with a digital certificate.

The main problem here is that I don't have the dataset. I am thinking to follow the following approaches:

Extract the embossed seal on degree certificate and compare with the real one.
Extract the signature and feed it into forgery detection neural network.
Watermark detection.

Anyone can suggest me any other approach then that would be very helpful. TIA.


texts: To learn machine learning which one is good?

texts:
Until now I have implemented linear and logistic regression myself. I have not used any library other than numpy and matplotlib. But in the internet every example is solved using libraries such as sklearn, pandas and etc. My question is, which one is good to learn for machine learning, implementing algorithm yourself or using libraries (sklearn or ...)?

Thanks.


texts: dynamic addition and variety of neural network outputs

texts:
After studying some academic papers about neural network recognition, plus trying some of them out myself, I do understand how you can train a network to give you a certain output on a defined set of output neurons depending on the input.

However I struggle to understand, how I can dynamically add output neurons. I've seen neural network applications, which where recognizing objects from images, however they set of classes to be recognized could always be increased. From my understanding this should not be possible, as the size of output neurons is fixed.

Lets assume I want to build a application in which user can make a photograph, upload it to the database and use if subsequent for image recognition.
I see only two possible solutions:

Retraining

Retrain the whole neural network every time some adds a new image. However training is very expensive and this would require a lot of resources so users can add all their objects.

"Dead" Outputs

as suggested in https://datascience.stackexchange.com/a/9551/18744

We could add a lot of neurons to the output-layer which are always trained to 0, unless we need more outputs. Once a new object appears, we start training the new output. However I believe the performance of such procedure would suffer in recognition terms, as we leave important training feature such as randomization out.


texts: why positive loss is going up in binary classification problem?

texts:
I am training a model for object detection with Lidar using binary cross entropy. But my positive_class_loss(foreground) is going up, whereas negative_class_loss is going down(background).
But overall_classification loss(positive_class_loss + negative_class_loss) is going down,because negative _class_loss is going down. Using two hyper-parameters alpha and beta two balance between negative and positive class losses.
I tried to find a reason on internet, but I didn't find any article or solution even on stack-overflow.
Below one can see, how the model is doing:

could someone explain the reason for this?
Thanks in advance.


texts: PELU activation: how does it work, and how to implement?

texts:
I have encountered PELU (Parametric Exponential Linear Unit) in the literature, but I can't find practical applications of it.

Moreover, I have some questions about how it works:

Are its parameters learned, or are they hyperparameters?
Is a Keras Custom layer required for a TensorFlow implementation?
In the original paper, it was applied to convolutional models only. Could you share links to applications outside Computer Vision?
Computational burden aside, is its performance proven (outside original paper) to be superior to more classical activations from the ReLU family?


texts:
I am new to the field of data science and trying to figure out ways to handle data quality issues before performing any modeling.

I am working on a house rental price data set. In this data there is feature called Total Squarefeet. The issue I am facing here is that out of 12000 records 200 have a range, for e.g., 1200 - 1800, OR 850 - 855, OR even and these ranges also have a random difference between them. Rest other are simple numbers. Is there a way to correctly handle this kind of data. Can anyone help me or guide me to a place from where i can learn and use techniques to handle such data. Thanks in advance.


texts: Training a sound localization neural network

texts:
I am trying to train a neural network, to estimate the location (in degrees from 0 to 180) a sound is coming from.

I am using tensorflow keras in python to train the model.

The input data are two binaural cues, specifically the ILD (Interaural Level Difference) and the ITD (Interaural Time Difference), each vector, consisting of the two above described features, is of dimensions [1,71276]. I have a total of 2639 measurments, 10% of which is used as validation data, and another 10% as test data.

The output should be an angle in the range [0,180].

I have normalized the data in the range [-1, 1] and the best loss I've been able to achieve is MSE = 16.

The model that achieved the highest MSE is the following:

model = tf.keras.Sequential(([
    tf.keras.layers.Input(shape=(71276,), name='input'),

    tf.keras.layers.Dense(units=900,activation='relu', name='dense_1'),
    tf.keras.layers.Dense(units=360,activation='relu', name='dense_2'),
    tf.keras.layers.Dense(units=180,activation='relu', name='dense_3'),

    tf.keras.layers.Dense(units=1,activation='linear', name='output')
]))

model.compile(loss='mse',
              optimizer=tf.keras.optimizers.Adam(lr=0.001),
              metrics=['mae'])

EPOCHS = 500
BATCH_SIZE = 32

callbacks = [
    tf.keras.callbacks.EarlyStopping(monitor='val_loss', mode='min', min_delta=0.5, patience=100, verbose=1),
    tf.keras.callbacks.ModelCheckpoint('best_model.h5', monitor='val_loss', mode='min', save_best_only=True),
    tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=50, verbose=1, mode='min', min_delta=2, cooldown=0, min_lr=0.000001)
]

history = model.fit(X_train, y_train, validation_data=(X_val,y_val), shuffle=True,
                    batch_size=BATCH_SIZE,
                    epochs=EPOCHS, verbose=1,
                    callbacks=callbacks)

Since this is the first neural network I've trained using my own data, I'm wondering whether there is anything obvious I've missed that could reduce the loss function, and if not, any suggestion is welcome!

I should note that I'm using google colaboratory and I've already tried adding another hidden layer but I got a ran out of memory error. I've also tried increasing/reducing the number of neurons in each layer but I haven't gotten better results and I tried using a CNN architecture as well, with little success as it didn't even converge after 300 epochs.

Thanks in advance!


texts: What is the problem with: the condition has length > 1 and only the first element will be used?

texts:
newbie to r. i have:

> > age <- function(Age) {
> +     answer<-NULL
> +     if(Age<15) answer<-"under 15"
> +     else if(15<=Age&&Age<=50) answer<-"15 to 50"
> +     else answer<-"over 50"
> +     answer
> + }
> > titanic_3 <- titanic %>%
> +     select(Survived, Pclass, Age, Sex) %>%
> +     filter(!is.na(Age)) %>%
> +     mutate(agecat=age(Age)) Warning message: In if (Age < 15) answer <- "under 15" else if (15 <= Age && Age <=  :   the condition has
> length > 1 and only the first element will be used
> >

if i use one & instead of &&, i get two warnings. i know that one of these is because of the shortcut thing.

how do i write this function so as to get rid of the warning message?

thanks

edit: i tried:

age <- function(Age) {
    answer<-NULL
    if(Age<15) answer<-"under 15"
    else if(Age<=50) answer<-"15 to 50"
    else answer<-"over 50"
    answer
}

and got even more warnings:

Warning messages:
1: In if (Age < 15) answer <- "under 15" else if (Age <= 50) answer <- "15 to 50" else answer <- "over 50" :
  the condition has length > 1 and only the first element will be used
2: In if (Age <= 50) answer <- "15 to 50" else answer <- "over 50" :
  the condition has length > 1 and only the first element will be used

this one seems to work, but they want us to use cut.

age2 <- function(Age) {
    ifelse(Age<15,"under 15",ifelse(Age<=50,"15 to 50","over 50"))
}


texts:
The dataset consists of

1) a set of objects and

2) a set of labels, which are used to describe the objects.

For the moment, for simplicity sake, each label can be marked as either true or false (In a more complex setup, each label will have a value of 1-10).

But, not all the labels are actually applied to all the objects (in principle, all the labels can and should be applied across all the objects, but in practice, they just are not). Also, when a label isn't applied to an object, one cannot simply assume that the label's value for that particular is false. Therefore, the missing labels will be ignored in the model.

I need to cluster the objects based on their labels.

Any tips on how and what algorithms to use will be appreciated.


texts: How do two perceptrons produce different linear decision boundaries?

texts:
I'm trying to visualize how two perceptrons converge to two different decision boundaries (which is ultimately used to create the classifier for the non-linearly separable data).

 Source: https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html

I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron? I know the weights can be initialized differently but does this second perceptron classify something else? Shouldn't the decision boundaries be converge to be the the same ultimately?

 Source: https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html


texts: What do I use as initial token? TSP using pointer networks

texts:
I'm new to learning neural nets and trying to recreate stuff in order to get an understanding of how things work. I'd like to replicate a pointer network to train on the travelling salesman problem.

While I'm able to understand most of what the paper by Vinyals et al 2015. The point of confusion is the use of the special starting token (=>)

I understand that for text you could create such a token. But how does it translate to co-ordinate data?

Would be really useful if someone could expand on this or point to a resource.

Thanks!


texts: How to set a class_weight Dictionary for Random Forest?

texts:
I'm dealing with an unbalanced dataset, so I decided to use a weight dictionary for classification.
Documentation says that a weight dict must be defined as shown below:
https://imbalanced-learn.org/stable/generated/imblearn.ensemble.BalancedRandomForestClassifier.html
     weight_dict = [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}]

So, since I want to predict 12 classes which are located in the last column.
I assume that the setting would be like:
weight_dict = [{0: 1, 1: 5.77390289e-01}, {0: 1, 1: 6.48317326e-01},
               {0: 1, 1: 1.35324885e-01}, {0: 1, 1: 2.92665797e+00},
               {0: 1, 1: 5.77858906e+01}, {0: 1, 1: 1.73193507e+00},
               {0: 1, 1: 9.27828244e+00}, {0: 1, 1: 1.18766082e+01},
               {0: 1, 1: 8.99009985e+01}, {0: 1, 1: 6.39833279e+00},
               {0: 1, 1: 2.55347077e+01}, {0: 1, 1: 9.47015372e+02}]

Honestly, I don't clearly understand the notation of the first indicators, I mean the:
      0:1 of {0: 1, 1: 1}

or the:
 1: value.

Do they represent column position, label order?
What is the right way to set it?
I'll be grateful for your insights.


texts: Using attributes to classify/cluster user profiles

texts:
I have a dataset of users purchasing products from a website.

The attributes I have are user id, region(state) of the user, the categories id of product, keywords id of product, keywords id of website, and sales amount spent of the product.

The goal is to use the information of a product and website to identity who the users are, such as "male young gamer" or "stay at home mom".

I attached a sample picture as below:

There are all together 1940 unique categories and 13845 unique keywords for products. For the website, there are 13063 unique keywords. The whole dataset is huge as that's the daily logging data.

I am thinking of clustering, as those are unsupervised, but those id are ordered number having no numeric meaning. Then I don't know how to apply the algorithm. I am also thinking of classification. If I add a column of class based on the sales amount of product purchased. I think clustering is more preferred. I don't know what algorithm I should use for this case as the dimensions of the keywords id could be more than 10000 (each product could have many keywords, so does website). I need to use Spark for this project.

Can anyone help me out with some ideas or suggestions?

Thank you so much!


texts: Tensorflow ArgumentError: argument --model_dir: conflicting option string: --model_dir

texts:
I am trying to execute this Tensorflow tutorial https://github.com/tensorflow/tensorflow/blob/r0.12/tensorflow/examples/learn/wide_n_deep_tutorial.py but I have the following error:

ArgumentError: argument --model_dir: conflicting option string: --model_dir

that appears in line:

flags.DEFINE_string("model_dir", "", "Base directory for output models.")

Do you know how I can solve it?

Thanks in advance,

Keira


texts: Tensorflow regression predicting 1 for all inputs

texts:
I'm trying to predict a UPDRS score (regression) from the parkinsons dataset. I wanted to start with a fairly basic structure and the add in more layers/techniques to improve performance but no matter what I do the network constantly predicts 1 for every input.

I've been banging my head against this problem for a couple of days now with no success. Originally I was using RELU and thought it was a case of dead neurons but using leaky RELU and monitoring the outputs I can see them firing.

Changing the network structure or training steps always results with the same horrible MSE of 896.39105. This leads me to believe the mistake is somewhere in the normalization but when I check the intermediate values they all seem fine.

Can anyone tell me where I've messed up?

import tensorflow as tf
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import normalize
from sklearn.preprocessing import StandardScaler

def leaky_relu(x, alpha):
  return tf.nn.relu(x) - alpha * tf.nn.relu(-x)

raw = pd.read_csv('parkinsons_updrs.data')
print("CSV loaded successfully")

#Split into features and targets
features = raw.drop(['subject#','test_time','motor_UPDRS','total_UPDRS'], axis = 1).as_matrix()
targets = raw['total_UPDRS'].get_values()

scale = StandardScaler(with_mean=0, with_std=1)
scale.fit(features, targets)
scaled_feats = scale.transform(features)

print("pre-processing complete")

#Split data into test/train
data_train, data_test, targets_train, targets_test = train_test_split(scaled_feats,targets,test_size = 0.2,random_state=123)

print("Train/Test split complete")

#Need to reshape to match placeholder shape
targets_train = targets_train.reshape(targets_train.shape[0],-1)
targets_test = targets_test.reshape(targets_test.shape[0],-1)

#Network Structure
X = tf.placeholder(tf.float32, [None,features.shape[1]])
y = tf.placeholder(tf.float32, [None,1])
prob = tf.placeholder(tf.float32)

w1 = tf.Variable(tf.random_normal([features.shape[1],100]))
w2 = tf.Variable(tf.random_normal([100,1]))

b1 = tf.Variable(tf.random_normal([100]))
b2 = tf.Variable(tf.random_normal([1]))

layer1 = tf.nn.dropout(leaky_relu(tf.add(tf.matmul(X,w1),b1),0.2),prob)
out = tf.nn.dropout(tf.nn.tanh(tf.add(tf.matmul(layer1,w2),b2)),prob)

loss = tf.reduce_mean(tf.square(out - y))
train_step = tf.train.AdamOptimizer(0.001).minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

print("Training...")

minibatch = False
batchsize = 512

for i in range(1000):
    if minibatch:
        for x in range(0,targets_train.shape[0],batchsize):
            sess.run(train_step, feed_dict={X:data_train[x:x+batchsize],y:targets_train[x:x+batchsize],prob:0.5})
    else:
        sess.run(train_step, feed_dict={X:data_train,y:targets_train,prob:0.5})
    if i%25==0:
        acc = sess.run(loss, feed_dict={X: data_train, y: targets_train,prob:1})
        print ("Training accuracy: %.5f" % (acc))
        #print(sess.run(layer1,feed_dict={X: data_train, y: targets_train,prob:1})[0])

test_acc = sess.run(loss, feed_dict={X: data_test, y: targets_test,prob:1})
print ("Test accuracy: %.5f" % (test_acc))

#Check if still predicting 1
output = sess.run(out,feed_dict={X: data_test, y: targets_test,prob:1})
print(sum(output)==len(output))


texts: Analytics term for turning row values into column names and count its assigned values

texts: Do we have a data mining/analysis term for turning row values into column names and count its assigned values?


texts:
I've got data representing a sequence of events at irregular intervals and an outcome. There are a fixed number of event types, about 10 of them. The outcome being modeled is binary.

Arguably the following are important to incorporate into the model:

1) Which events occur

2) The order in which the events occur

3) The time lag between events

I.e:
User has event A at start, 5 months later event C, and outcome is positive.
Another user has event B at start, 3 days later event A, 2 days later event D, and outcome is no.

I've also got some static (non sequence data) for each user which I'd like to incorporate.

What is the best approach to model this?

Thanks in advance.


texts: Any consideration for image classification when one class has generally one particular color

texts:
I am trying to predict rooftop orientations among: horizontal, vertical, flat, unknown.

The particularity of horizontal and vertical rooftops, is that it's color generally is reddish, the reason of tiles. The flat ones, I would say, most are blue to grey. This is just an eye perception.

Is there a intelligent consideration to put the third dimension which is color on focus ?

The third dimension goes for the RGB channel.

My model is like

from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(96, 96, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

# the model so far outputs 3D feature maps (height, width, features)

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))
# model.add(Dropout(0.5))
model.add(Dense(4))

model.add(Activation('softmax'))

model.compile(optimizer=tf.train.AdamOptimizer(),
              loss='categorical_crossentropy',
              metrics=['accuracy'])


texts: How can I know if my conv1D model is overfitted or underfitted from loss curve?

texts:
I am working on classification of time series multivariate data. By doing PCA, I converted multivariate to uni-variate and fed it into a conv1d in keras.

However, I am getting a very high accuracy and low loss both in validation and in training. How can I justify this?

I have tried cross validation, but the results are not much different.
I am using adam optimizer (learning rate:0.0001). With 0.001, my model fails to converge.

I have made sure that I am not mixing the training and validation datasets.
I have shuffle both datasets independent of each other. I trained on 3728 samples and validated on 610 samples.

Can we expect such a high accuracy with binary classification?


texts: On-line random forests by adding more single Decisions Trees

texts:
A Random Forest (RF) is created by an ensemble of Decision Trees's (DT). By using bagging, each DT is trained in a different data subset. Hence, is there any way of implementing an on-line random forest by adding more decision tress on new data?

For example, we have 10K samples and train 10 DT's. Then we get 1K samples, and instead of training again the full RF, we add a new DT. The prediction is done now by the Bayesian average of 10+1 DT's.

In addition, if we keep all the previous data, the new DT's can be trained mainly in the new data, where the probability of picking a sample is weighted depending how many times have been already picked.


texts: Multi target regression when some targets have correct NA labels

texts:
I'm working on a Keras neural net that does key point prediction of body parts (left foot, left knee, left hip, etc.). For each image (X), the target (Y) is a list of coordinates for the keypoints (left_foot_x, left_foot_y, left_knee_x, left_knee_y), etc.). Whenever a keypoint isn't visible the X and Y coordinate equal = 0.

I suspect that leaving the data in this state will lead to bad results because 0 really means NA (i.e. the keypoint isn't visible). Leaving the X and Y coordinates 0 would suggest the keypoint is actually in the corner which is false. What is the right way to structure data to do multi target regressions when the correct output is sometimes NA?


texts: How do I perform Sentiment Analysis on Tweets in the following pattern:

texts:
I have tweets obtained based on matches (football) before the match begins. I have tweets which specify a team will win 3-1 and so on which are easily analyzed using regular expressions. I am facing difficulties when it comes to tweets where two team names are specified with win / loss comparison such as:

Manchester United will win the match tonight against Spurs.
(or)
Spurs will find it difficult to break down United defence.

I have taken several such tweets and replaced the team names as TEAMONE and TEAMTWO and placed them into two files (teamone winning tweets and team two winning tweets). Using SVM, the model couldn't clearly classify tweets between the two classes (files mentioned above). Is it due to class imbalance? Should I stick to a rule based classification in this case or would any different method work? I'm a newbie in ML and any suggestions would be appreciated.


texts: What is a good approach for embedding both textual and spatial features for document classification?

texts: I am working on a document classifier that can perform the classification based on the document structure as well. My plan is to get the word embedding as well as the word coordinates and somehow combine the two features and pass it through a Graph Convolutional Network (GCN) to generate a graph embedding which I can then use to train a classifier. I was referencing this paper and they do data extraction by first getting the text embedding and image embedding and then combining them using elementwise addition. I was wondering what would be a good solution to combine both text features and positional features (x,y,height and width).


texts: Random seed in Machine learning model comparison

texts:
I would like to ask a question about the random population generation gin splitting the dataset in machine learning classification models.
For example, I used seed = 1 and got accuracy of 0.7 and seed = 5 and got accuracy of 0.8 and seed= 2000 and got accuracy of 0.89 using Adaboost.
I found a research paper using the same dataset i used and accuracy achieved is 0.94 using xgboost model without specifying the seed used in developing the model.
Same exists for other research papers as well.
Which results I need to pick to compare my model with other models proposed in the literature? Meanwhile I implemented all the models proposed in literature with the different seed and found that results are different when compared to the paper. And sometimes without using my result with adaboost is better.
I need help to compare my proposal with other works.


texts: Testing Environment for Data Science Models in Industries?

texts:
In Software Engineering, when an application gets developed it would be pushed into the testing environment to make sure that the application is reliable and doesn't have any bugs. Mostly this will be done by another User other than the Developer.

How does it work in Data Science?

Do we have a testing Environment in industries as well before the model goes into Production?

If yes, what will a sample test case look like and will there be a separate tester or is it the developer doing different tests like Gini Test or the T-test and evaluate his results against AUC, Recall and Precision ?


texts: Gradient Checking LSTM - how to get change in Cost across timesteps?

texts:
I am performing gradient check for my LSTM which has 4 timesteps. The LSTM looks as follows:

   01       01       01       01
   ^        ^        ^        ^
  LSTM --> LSTM --> LSTM --> LSTM
   ^        ^        ^        ^
   11       11       11       11

So, at every timestep we are feeding in vector {1,1} and expect {0,1} at the output.

Assume I perturb the weight inside LSTM, then perform 4 forward props - one for each timestep - how do I now get delta of the cost function that this single perturbation has caused?

Am I allowed to simply add-up the change in Cost from all 4 timesteps to treat it as derivative estimate?

Also, should I perform it as follows for LSTM:

perturb a single weight upwards
forward prop 4 timesteps
perturb the weight downwards
forward prop 4 timesteps
get 4 deltas
sum the 4 deltas to get a total change in Cost

or

Set N=0
perturb the weight upwards
foward prop at a particular timestep N
perturb the weight downwards
forward prop at a particular timestep N
get single delta, store it away
increment N
until N not equal 4 return to step 2)
sum the 4 deltas to get a total change in Cost

The second approach somehow seems more correct, because LSTM will have a hidden state ..Is this correct intuition or it won't matter?


texts: How to feed LSTM with different input array sizes?

texts:
If I like to write a LSTM network and feed it by different input array sizes, how is it possible?

For example I want to get voice messages or text messages in a different language and translate them. So the first input maybe is "hello" but the second is "how are you doing". How can I design a LSTM that can handle different input array sizes?

I am using Keras implementation of LSTM.


texts: Random Forest Classifier - KFold CV Tunes Very Deep Trees --> Overfitting?

texts:
I'm tuning a random forest in python and am wondering if/why my model is overfit. The dataset is described below:

1700 Positive Cases / 54000 total cases ~ 3.2% (unbalanced)
50 Numerical Features,~450 label/hot encoded features (post data
reduction)
10Fold CV using 85% of data, with 15% hold out for final test
Classification Metrics = AUC or F1 (as data is imbalanced)

The results I get tend to suggest using very deep trees i.e depth 18 with no restriction on number of samples per split = 2(default). In this case, Train AUC was 99.9% , Max Test AUC was 84%. My scores are also almost monotonically increasing in max depth of trees. Given the results and how deep the trees are - I suspect the model is overfit? If this is the case then why would I not observe some sort of out of sample reduction in AUC as depth and min_samples_split typically constrain the random forest? Or have I overlooked anything in tuning?

My ranges in CV Grid Search are more or less:

n_estimates : range(100,1000,by=100)
max_features : {sqrt(p),0.3,0.4,0.5}
max_depth : range(2,20,by=1)
min_samples_split : range(2,50,by=1)
class_weights : {balanced,None}

Thanks