texts: How can I know if my NN TensorFlow model is overfitted or not?

texts:
I am new with TensorFlow (Python) and I can not juge my obtained results in terms of training and testing accuracy

I am using the GradientDescentOptimizer with a learning coeff equal to 10^(-4) and I have executed the following code :

for gg  in range (1501):
    session.run(optimizer, feed_dict=train_data)
    train_accuracy = session.run(accuracy, feed_dict=train_data)
    if gg % 100 ==0 :
        #print(train_accuracy)
        session.run(optimizer, feed_dict=test_data)
        test_accuracy = session.run(accuracy, feed_dict=test_data)
        print(test_accuracy)

One time I have commented the testing accuracy to print only the training accuracy and one time I did the contrary.
I had stored them in the following table:

My problem is that I am not able to detect overfitting if it exists or not
Can you help me please!


texts: Calculating saliency maps for text classification

texts:
I'm following the text classification with movie reviews TensorFlow tutorial, and wanted to extend the project by looking, for a certain input, which words influenced the classification the most.

I understand this is called a saliency map, but I'm having trouble calculating it. I believe that I need to calculate the gradients of the output with respect to the input. I tried to implement code similar to the code in this answer to no avail. A confounding issue is that the model uses an embedding layer, which doesn't propagate the gradient, so I think one needs to calculate the gradients with the input being the output of the embedding layer.

It's probably wrong for all sorts of reasons, but this is the closest I've gotten with the Python code:

# Create the saliency function
input_tensors = [model.layers[1].input, keras.backend.learning_phase()]
model_input = model.layers[1].input
model_output = model.output
gradients = model.optimizer.get_gradients(model_output, model_input)
compute_gradients = keras.backend.function(inputs=input_tensors, outputs=gradients)

# Word encoding
idx = 0 # Calculate the saliency for the first training example
embeddings = model.layers[0].get_weights()[0]
embedded_training_data = embeddings[train_data[idx]]
matrix = compute_gradients([embedded_training_data.reshape(sum([(1,), embedded_training_data.shape], ())), train_labels[idx]])

But the final matrix is the same row repeated and I'm not sure how to interpret it. Any help would be greatly appreciated. Thankfully, as this is extending a tutorial, there is a complete working example of the code!


texts: How can we apply PCA to reduce dimensionality of a heterogenous dataset?

texts:
I have a dataset containing insurance Claims with quantitative and qualitative variables but PCA refuses to convert or work with "string" type variables.

This is the code I used :

from sklearn.decomposition import PCA
claims=pd.read_csv('./insurance_claims.csv',sep=',',header=0)
X=claims.ix[:,1:].values
pca=PCA(n_components=12)
pca.fit(X)

I'm trying to reduce dimensionality to cluster the dataset and detect fraudulous claims. If there is any alternative to PCA for heterogenous data much appreciated.


texts: Prediction for not completely well classified data

texts:
I have a DataFrame of users, some of them are "bots" and they are identified with a bit equal to 1 in the "is_bot" column, if the bit is 0, the user is considered as "human".

The problem is that some users may be misclassified as "humans" instead of "bots" since the "bots" have been identified on the basis on an incomplete list in the gathering data phase.

I will train and test my model on this partially correct data, but when I test it, I will find that my model correctly predicts some users as "bots" even if in the original dataset they are "humans".

Correctly predicts means that, in reality, the users are bots because I checked some of them manually, but I can't do this for my entire dataset of 1 mln users.

This would result in a model with low accuracy, even if the predictions may be right.

How do I handle this problem?


texts: Multi Class Classification on large dataset with over 600 classes

texts:
I'm trying to train a text data for multi class classification which comprises of 1 Million rows. After cleaning the data, I'm using a sparse matrix of Word2Vec features (Feature size is 300)

The data which I have is 1. ID 2. Dictionary 3. Label

Dictionary size varies from 10 keys to 900 keys

Steps I followed on Dictionary columns are:

Converted Dictionary to String
Getting only good tokens from the string
Removing Stopwords
Stemming of words
Word2Vec Model training with feature size 300.
Word2Vec feaure generation
Label Encoding
Converting Feature Vectors to Numpy Array
Converting Numpy Array to Sparse Matrix of (1114220, 300)
Tried OneVsRest model for training
onevsrest = OneVsRestClassifier(SVC(probability=True) , n_jobs=-1)

onevsrest.fit(sparse_matrix , df.labels)

I was running this model for nearly two days and it got killed automatically

I also tried Logistic Regression

lr = LogisticRegression(penalty ='l1' , C=1 ,dual=False , solver='saga' , n_jobs=-1)

lr.fit(sparse_matrix , df.labels)

Still I faced the same issue ( Model keeps training for 2 days and gets killed)

Am I doing something wrong? Or is there any better way to do this type of problem?


texts: Identifying features which drives recommendations

texts: I'm want to a recommendation engine for one of the clients to recommend products to his customers. One of the requirement is that recommendations should be explainable. i.e. Why does the recommendation engine recommend certain products to a given customer, based on which features? I'm thinking of using either hybrid models (content based and collaborative filtering) or deep and wide models. However, I'm not sure whether there is any way to identify features which drive recommendations. Are there any other models in which results are more interpretable?


texts: Validating performance of panel data based models

texts:
I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). Specifically, I'm curious whether the final accuracy measure used (e.g. explained variance) should be an average of such measure for individual cross sections (perhaps to accounting for distribution of the accuracy measure itself) or of the full data sample?

To give an example, when dealing with factor models in quantitative finance, you have cross sections of variable number of stocks with multiple attributes, which you use to forecast the return of the stock for some lagged period.

To evaluate the in- and out- sample performance, I have seen both methods applied, for example:

IC Score (cross-sectional correlation averaged over the sample period)
R2/Explained Variance on the full sample

For fixed-effect models (e.g. Linear Regression), I understand the logic behind using the full sample, but I also find that it masks the presence of shocks/reversals in the system that may be occurring in specific periods that is unaccounted for in the model?

Also, when it come to tuning parameters for learning algos, should I adjust generic evaluations when it comes to cross-validation so that cross-sections are evaluated together?


texts:
My dataset has around 1000 features and 30k rows.

All the feautres have value either 1 or 0. My target variable is Size which 3 classes : Small, Medium and Large. I have around 5k "small" data points, 9k "Medium" data points and 20K "Large" data points. I want to find feature importance. I am not sure which algorithm will work best for features with only 1s and 0s. I tried Chi2 but accuracy is very low:

#find best scored 10 features
select_feature = SelectKBest(chi2, k=10).fit(X_train, Y_train)

X_train_2 = select_feature.transform(X_train)
X_test_2 = select_feature.transform(X_test)

clf_rf_2 = RandomForestClassifier(n_estimators=1000, n_jobs=-1, random_state = 100)
clr_rf_2 = clf_rf_2.fit(X_train_2,Y_train)
ac_2 = accuracy_score(Y_test,clf_rf_2.predict(X_test_2))
print('Accuracy is: ',ac_2)
cm_2 = confusion_matrix(Y_test,clf_rf_2.predict(X_test_2))

My accuracy is only 58% and the model is only good in predicting "Large" data points. Below is the Confusion Matrix

      Small   Medium   Large
Small  6      17       3000
Medium 9      9        2000
Large  31     32       7500

I mostly have 2 questions:

I am not sure which algorithm will work for best features with only 1s and 0s
How do I get higher accuracy using chi2 (predict Small and Medium class better)

Thanks! I would really appreciate any input on this.


texts:
I need to use the answers from a questionnaire for training a classifier.
I discovered that some questions can have nested sub-questions..
Let's say (just an example) that I want to predict whether a person is going to buy a house based on the following questions:

1) What is your gender?
[] male
[x] female
[] I prefer not to answer

in the case the answer is female (as in the example above) a sub-question is ansked

1_female) are you pregnant?
[x] yes
[] no

Then the questionnaire continues..

How should I use these features to train my model?

Option 1)
Treat them separately and transform them with one-hot-encoding
I will have then the feature vector

gender_male - gender_female - gender_not_answered - pregnant_empty - pregnant_yes - pregnant_no
     0      -        1      -         0           -        0       -       1      -        0

Obviously the feature pregnant_empty will be coded with 1 for all the males

Option 2)
Merge the 2 answers and encoding the concatenation

gender_female_pregnant_yes - gender_female_pregnant_not - gender_male - gender_not_answered
     1                     -                 0          -      0      -     0

Other options?

Please treat this just as an example... the problem is that in a real scenario

the nested question could appear with 2 or more answers
expanding the features as in option 2 will make my feature vector explode..

I hope my question was clear enough


texts:
Could someone please explain to me why the autoencoder is not converging? To me the results of the two networks below should be the same. However, the autoencoder below is not converging, whereas, the network beneath it is.

autoencoder implementation, does not converge

autoencoder = Sequential()
encoder = containers.Sequential([Dense(32,16,activation='tanh')])
decoder = containers.Sequential([Dense(16,32)])
autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder,
                        output_reconstruction=True))
rms = RMSprop()
autoencoder.compile(loss='mean_squared_error', optimizer=rms)

autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64,
            validation_data=(testData, testData), show_accuracy=False)

non-autoencoder implementation, converges

model = Sequential()
model.add(Dense(32,16,activation='tanh'))
model.add(Dense(16,32))
model.compile(loss='mean_squared_error', optimizer=rms)

model.fit(trainData,trainData, nb_epoch=numEpochs, batch_size=batch_size,
            validation_data=(testData, testData), show_accuracy=False)


texts: Space between an object and the ground truth bounding box

texts:
Which way is better for drawing the ground truth boxes for object detection?

Drawing as tight as the sides of the box touch the border of the object, or

Make a little space between the box and the object?


texts:
I understand that from mathematical point of view, only difference between Convolution and Cross-Correlation is that Convolution is commutative, while Cross-Correlation is now.

In many articles Cross-Correlation is explained as measuring similarity between two signals, and Convolution is explained as calculating relationship between two signals.

My question is this, what is the point of commutative property? How does it make Convolution different from Cross-Correlation in practice? Or is it just there for "better" mathematics and easier proofs? Also, what are, if any, benefits of implementing CNN with Convolution over Cross-Correlation?


texts: How to maintain accuracy with more classes?

texts:
I'm working on a CNN model (implemented by keras) that detects landmarks from images. Every landmark (for example Acropolis, Big Ben, Eiffel Tower etc) is as a separate class and is divided into own directory.
The training dataset contains 1600 images, validation 400 images and test 500 images. The training process has 80 iterations.

I noticed that with two or three classes the accuracy is quite good (3 classes => accuracy: 0.8353) but by every next class it's getting to decrease.

Is it so that adding more classes increases features and complexity? What could be the solution for that? Adding more layers or data? I have tried to add more layers but it's rather worsened the result than made it better. Is it because of vanishing gradient? Adding more data is big problem because I just can't find more unique images.
I also changed other config parameters (optimization algorithm, dropout rate etc) but with no result.

The used config parameters are:

NUMBER_OF_CLASSES = 4
BATCH_SIZE = 32
EPOCHS = 80
DROPOUT_RATE = 0.3
LOSS = 'categorical_crossentropy'
OPTIMIZER = 'rmsprop'
METRICS = ['accuracy']
CLASS_MODE = 'categorical'

And constructed model:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(64, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(128, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(256, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Flatten())

model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(NUMBER_OF_CLASSES))
model.add(Activation('softmax'))

--------------
Total params: 721,604
Trainable params: 719,620
Non-trainable params: 1,984


texts: Does multi-head attention remove the need for self-attention?

texts: The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the encoder and decoder still be necessary?


texts: Opposite results using statsmodels and seaborn regplot

texts:
I use regplot using the following code:

sns.regplot(x = "Year", y = "Data_Value", data = NOAA_TMAX_s );

and I obtain the following figure:

showing clearly that the trend is negative.

As seaborn does not provide the equation I calculate it by the following code:

import statsmodels.api as sm
X = NOAA_TMAX_s["Year"]
y = NOAA_TMAX_s["Data_Value"]
model = sm.OLS(y, X)
results = model.fit()
print(results.summary())

and this is the output:

LS Regression Results
==============================================================================
Dep. Variable:             Data_Value   R-squared:                       0.982
Model:                            OLS   Adj. R-squared:                  0.982
Method:                 Least Squares   F-statistic:                 1.141e+06
Date:                Mon, 30 Mar 2020   Prob (F-statistic):               0.00
Time:                        20:13:08   Log-Likelihood:            -1.0576e+05
Df Residuals:                   20961   BIC:                         2.115e+05
Df Model:                           1
Covariance Type:            nonrobust
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
Year           0.1380      0.000   1068.387      0.000       0.138       0.138
==============================================================================
Omnibus:                      331.301   Durbin-Watson:                   0.456
Prob(Omnibus):                  0.000   Jarque-Bera (JB):              387.614
Skew:                          -0.255   Prob(JB):                     6.77e-85
Kurtosis:                       3.428   Cond. No.                         1.00
==============================================================================

As I understand it, the result yields a positive highly significative coeficient.
I need some help to understand why both results are not coherent.


texts: Which approach for user classification on chat text (classifier, representation, features)?

texts:
I'm trying to train a classifier to classify text from a chat between 2 users so later on I can predict who of the two users is more likely to say X sentence/word. To get there I mined the text from the chat log and ended up with two arrays of words, UserA_words and UserB_words.

Which classifier should I use fot this purpose and what structure the training data should have? I've researched for the bag of words structure but dont know exactly how to train a classifier with data in that format.

To clarify this last point, for now I have the data in a dict like {"hello":34, "how":12} and so on, being the terms word:frequency of each user. As far as I know, there is no way to use this two dicts as a classifier fit input. So, how do I transform this 2 dicts into an array that I can use to train a classifier (let's say I want to use a gaussian Naive Bayes just for the sake of the example)


texts: Is there any way to get samples in under each leaf of a decision tree in Sklearn ?

texts:
I have trained decision tree . I also have a graph of the tree (
) . Now i want to see which samples (red circled ones ) are under which leafs . I am using sklearn's implantation .  Is there any way to get the samples under a leaf ? ?


texts:
I analyse log files, very often there are strings that only differ on few places and are same every where else, I am trying to find the generic string that they most likely belong to e.g. :

UHGUYGUYGUYGUY id = U1234 UYAG*&^T*#@G*(&G@ id2 = 8767 ib97y79yh0978
UHGUYGUYGUYGUY id = Z1D#34 UYAG*&^T*#@G*(&G@ id2 = 98h ib97y79yh0978
Sss3ug87g87g78ghs837g8 obj { 876t7g }937hs937hs973h97sh397 jh7897y98h
Sss3ug87g87g78ghs837g8 obj { 98u2 }937hs937hs973h97sh397 ZN7897y98h

to me the only difference is between the two the value of ids, so a generic forms/groupings will be

UHGUYGUYGUYGUY id = * UYAG*&^T*#@G*(&G@ id2 = * ib97y79yh0978
Sss3ug87g87g78ghs837g8 obj { * }937hs937hs973h97sh397 *7897y98h

I am not sure what in machine learning I should be looking up for this problem, or even if this problem has a name.

of course this is a very simplified example, the number, location of id's can vary for different generic cases, that is why I can not write old fashioned code for this, too many things can change.

Is there anything in machine learning to help find groups for such strings? if yes, what is it called?


texts: How to interpret metrics of a model after scaling the data

texts:
I have a GradientBoostingRegressor from scikit-learn which I trained. Afterwards, I obviously would like to know how good the model is. So, on a non-scaled dataset I would just use the mean_squared_error function from scikit and it would output a certain value that made sense (regarding the dataset).

Now, I scaled (/ transform) the targets in my dataset using the scikit QuantileTransformer(output_distribution='uniform'). The target is now scaled between 0 - 1. This is fine during training etc.

After training the model, I ran the following code to get a few metrics:

test_pred = gb.predict(X_test)
mse_test = mean_squared_error(y_test, test_pred)
print("RMSE on Test:", np.sqrt(mse_test))
print("MSE on Test:", mse_test)

mae_test = mean_absolute_error(y_test, test_pred)
print("MAE on Test:", mae_test)

Because the target values are scaled, the output is something similar to this:

RMSE on Test: 0.23563730007705744
MSE on Test: 0.05552493718760521
MAE on Test: 0.19235478752773819

I assumed that I could get the 'actual' non-scaled metrics back by applying the QuantileTransformer.inverse_transform function to the output.

So then I got:

RMSE on Test: 2231.21330222
MSE on Test: 807.28588575
MAE on Test: 1888.23406628

Which doesn't seem very correct to me. Normally, the RMSE would be smaller than the MSE, but that isn't the case. If you get that the sqrt of a (MSE) value under 1. Also, the MAE should probably be smaller than the MSE.

My question is, how do you interpret those scaled metric values? Is the inverse_transform output correct? How do I get correct, non-scaled values for the metrics?

I'd appreciate some help on this.

Thanks.

Edit:
The QuantileTransformer is only an example. The question also applies to the MinMaxScaler and other scalers in general.


texts:
I am trying to predict clients comportement from market rates.

The value of the products depends on the actual rate but this is not enough. The comportement of the client also depends on their awareness wich depends on the evolution of rates. I've added this in model using past 6 month rates as features in polynomial regression.

In fact media coverage of rate mostly depends on rate variations and I wanted to add that in my model.  The idea would be to add a derivative/variation of rate as a feature. But I anticipated something wrong, example with only two month , my variation will be of the form $x_n - x_{n-1}$ that is a simple linear combination of actual and past rates. So for a 1d polynomial regression i will have:

$$ x_{n+1} = a * x_{n} +  b * x_{n-1} + c * (x_{n} - x_{n-1})$$

instead of:

$$ x_{n+1} = a_0 * x_{n} +  b_0 * x_{n-1}$$

wich is strictly equivalent with $ a + c = a_0 $ and $b-c= b_0$. Higher polynomial degree results in a more or less equivalent result.

I am thinking about a way to include derivative information but it seems not possible. So I am wondering if all the information is included in my curve. Is this a general idea ? all information is somewhat directly contained in data and modifications of features will result in higher order objective function ?


texts: Using feature vectors from imagenet to train xgboost (vs a standard Conv net)?

texts:
I am planning to use feature vectors generated from imagenet to train an xgboost model. This is as opposed to training a standard convolutional network with the same image set.

This is because we use xgboost internally across several of our modeling tasks and is just a matter of personal preference. Has anyone noticed any issues - deterioration of accuracy or performance? Anything we should be looking out for?


texts: Does feature normalization improve performance of Hidden Markov Models?

texts:
For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be:

Normalize to 0-mean and unit-variance
Scale to [-1, 1] interval
Scale to [0, 1] interval

With neural networks, the rationale behind scaling is to get an "un-squished" error surface that is easier to navigate in.

HMMs use the Baum-Welch algorithm, which is a variation on the Expectation Maximization (EM) algorithm, to learn parameters.

Is EM sensitive to scale of features? Is there some motivation for normalization for HMMs?


texts:
I have time series data which looks like the graph mentioned below.
I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are that they do not account for the neighboring data points.
For example, in the data show below I do not want to remove the values which are simply maximum or standard deviation away from the mean. I want to remove the points which are circled in the red. The other extreme values are common in the area should not be detected as outlier as these data points have similar nearby data points.
Is there any method to remove these points or is there any python library I can use to remove these points. Normal standard deviation and median filters do not work well for these as they also remove the points which are not circled.


texts: Differences and similarities between nonnegative PCA and nonnegative matrix factorization

texts: I have seen references in the literature to nonnegative principal component analysis (nPCA) and nonnegative matrix factorization (NMF). I have tried reading the papers on both of them but it is not clear to me what the differences and similarities between them are. By similarity, I mean I am also interested in knowing when the nPCA and NMF method will give the same solution. Can someone clarify this?


texts:
I am trying to understand PCA intuitively. Here it goes: After finding the eigenvectors and eigenvalues of the covariance matrix of the dataset, the eigenvalues will represent how spread out the dataset is, and the eigenvector will represent the direction of the eigenvalue. And what we are trying to do is to order the eigenvector from highest eigenvalue to lowest (highest meaning the features are more spread out hence more independent of each other and less redundant) and project the original dataset onto the new set of "axis" (the eigenvectors). For example, if we have a 3-D dataset, then there will be three eigenvectors each orthogonal to each other (so they will form a 3-D space). Next, we project the original dataset onto this new set of "axis" space, and if we want to reduce to 2-D, we would take out the least significant "axis" (smallest eigenvalue)

I am familiar with the technical steps of performing PCA, but I am struggling to understand the actual intuition behind the algorithm. Were there any mistakes or flaws in my explanation above, especially the part of explaining dimension reduction with the use of the word 'axis'? Thanks a lot in advance!


texts: Random Forest Target/prediction maximum minimum

texts:
This is my current situation, I currently have a regression random forest that is targeting a continuous variable (sales amount) is there any configuration for the classifier that allowed me to set a maximum and a minimum for the target.

Example: Predicted sales_amount should be between 10000$ and 20000$

thank you!!


texts: Clustering a variable based on another variable or set of variables

texts:
df11[['COMPONENT_ID','FIRMWARE','SERIAL','CRP0_VDDN']].head()

Consider I have these four columns to analyse. I want to form say 3-5 clusters of COMPONENT_IDs with similar characters. I want this to happen based on the remaining features or just CRPO_VDNN in relation with COMPONENT_IDs. How can I do this ?


texts: In supervised learning, how to get info from correlation?

texts:
I am trying to build a classification model so I have tried to check the correlation between the features.

Here Loan_Status is my target variable.

I just don't know how to extract information from this? Please help.
I have questions like.
Is -0.0047 corelation of ApplicantIncome with Loan Status useful?


texts: Interpreting rawPrediction from Spark ML LinearSVC

texts:
I am using Spark ML's LinearSVC in a binary classification model. The transform method creates two columns, prediction and rawPrediction. Spark's docs don't provide any way of interpreting the rawPrediction column for this particular classifier. This question has been asked and answered for other classifiers, but not specifically for LinearSVC.

For a binary classification problem, the LinearSVC column contains pairs of values that range well below 0 and well above 1. Clearly this isn't simply the probability of belonging to each class. What is it?

Here's a model built on a subset of features in the original dataset. Fitting any data with Spark's LinearSVC will produce this column.

var df = sqlContext
  .read
  .format("csv")
  .option("header", "true")
  .option("inferSchema", "true")
  .load("/FileStore/tables/full_frame_20180716.csv")

var assembler = new VectorAssembler()
  .setInputCols(Array("oy_length", "ah_length", "ey_length", "vay_length", "oh_length", "longest_word_length", "total_words", "repeated_exact_words", "repeated_bigrams", "repeated_lemmatized_words", "repeated_lemma_bigrams"))
  .setOutputCol("features")

df = assembler.transform(df)

var Array(train, test) = df.randomSplit(Array(.8,.2), 42)

var supvec = new LinearSVC()
  .setLabelCol("written_before_2004")
  .setMaxIter(10)
  .setRegParam(0.001)

var supvecModel = supvec.fit(train)

var predictions = supvecModel.transform(test)

predictions.select("rawPrediction").show(20, false)

Output:

+----------------------------------------+
|rawPrediction |
+----------------------------------------+
|[1.1502868455791242,-1.1502868455791242]|
|[0.853488887006264,-0.853488887006264] |
|[0.8064994501574174,-0.8064994501574174]|
|[0.7919862003563363,-0.7919862003563363]|
|[0.847418035176922,-0.847418035176922] |
|[0.9157433788236442,-0.9157433788236442]|
|[1.6290888181913814,-1.6290888181913814]|
|[0.9402461917731906,-0.9402461917731906]|
|[0.9744052798627367,-0.9744052798627367]|
|[0.787542624053347,-0.787542624053347] |
|[0.8750602657901001,-0.8750602657901001]|
|[0.7949414037722276,-0.7949414037722276]|
|[0.9163545832998052,-0.9163545832998052]|
|[0.9875454213431247,-0.9875454213431247]|
|[0.9193015302646135,-0.9193015302646135]|
|[0.9828623328048487,-0.9828623328048487]|
|[0.9175976004208621,-0.9175976004208621]|
|[0.9608750388820302,-0.9608750388820302]|
|[1.029326217566756,-1.029326217566756] |
|[1.0190290910146256,-1.0190290910146256]|
+----------------------------------------+

only showing top 20 rows


texts: How to give importance to recent data in a large dataset

texts: I am solving a binary classification problem which is also partly a time-series problem. This means that the recent data is more relevant than the old data. But how can I assign more importance to the recent data as compared to the old one?


texts: Sort dataframe by date column stored as string

texts:
I have dataframe named df1.

I want to sort data frame month column according to month (Jan, Feb, March..).

For that I used code:

sorted_df = df1.sort_values(by='month')
print(sorted_df)

but the output is sorted by alphabetic order of month column.

I think the reason for that in this month column data type is object so month column is sorted by alphabetic order.

The question is: how sort values in column month in correct order (according to order of months in year)?

My dataframe:


texts: Google Earth Pro Satellite image segmentation using clustering

texts: I have downloaded a satellite image from Google Earth Pro software corresponding to a particular date for a selected area around a place. I want to specifically segment the road lanes from the image using Kmeans algorithm. How can this be achieved?


texts: How do you deal with variable input sizes with an encoder-decoder net with skip connections in Keras?

texts:
I am currently getting into image segmentation with Keras, and I am using an encoder-decoder type as in the image below.

My problem is that applying a MaxPooling2D layer of size (2,2) cannot be reliably reversed by using an UpSampling2D layer of size (2,2), as the input and output sizes do not match if the input dimensions are odd. The same problem arises with a Conv2D layer with a stride >1 which I want to "reverse" with a Conv2DTranspose layer. This leads to the problem that the skip connections sometimes connect layers of differing sizes such that I cannot concatenate these layers.
Is any canonical way to deal with this problem in Keras, while maintaining skip connections and allowing arbitrary input sizes?


texts: What does it mean when the shape of input images is (600,64,64,3)?

texts:
While attempting an assignment, I found that shape of the input image was (600,64,64,3).
I thought 3 stood for the number of channels but it's listed as the 4th dimension.
What does this mean?
This is in reference to convolutional neural networks.


texts:
Suppose that I have an input image with 224 * 224 * 3 dimensions. I pass it through a convolution layer with 64 filters and same convolution operation. Suppose that I want to reconstruct the original image using the output of the convolution layer which has the size 224 * 224 * 64. I have the following questions:

Should I use deconvolution? If so, how is the arrangement of deconvolution layer (number of filters and the value of weights. Also when should the activation be applied)?
Are the number of filters and weights in forward pass equal to the backward pass?
Is there any technique other than deconvolution?
Is there any available Keras code for my need?

I've seen here and also here but they represent a high level abstraction and don't contain appropriate detailed answer.


texts: How the Support Vector Machine will perform if the bias b = 0 in the equation of hyperplane?

texts:
We have a soft margin linear SVM and the equation is as follows :

How the SVM will perform if b = 0, means the hyperplane is passing through the origin ?


texts:
For detecting an outlier in a vector I have tested different well known outlier detection methods. Finally, I used combination of different methods and an agreement between those methods. Now, a person asks why did you choose this combination and algorithms!? You can reach different combinations and use other algorithms and they may yield better results. What should I answer? I cannot just say based on tests as there are many other algorithms that I haven't tested (cannot test all algorithms). It is not a logical response, I think.

I'm looking for tests to justify my selected methods and combination and say why I have selected these methods.

Please let me know your suggestions.


texts: migrating to python from R: specific questions

texts:
I have been using R and RStudio for prototyping and model building and due to some persisting problems (which would only be applicable to the environment that I am using in) we have decided to use python. I am looking to know:

is there a python development environment like "RStudio Server"?
how easy/hard to enable multi-threading/multi-core processing in
Python for decision trees/gradient boosting?
can data persist in python dev environment server? Meaning: Can I save the dev environment with code and data frames and come back the
next
    day to access/pick up where I left?
Can you connect to Oracle database from Python?
Is there a way I can import .RData to python development
    environment?

Why ask these questions here?

While looking up on search engines, I am mostly getting unreliable results, training insitutes promotions, outdated blogs and whitepapers from industry giants. I need a reliable answer.


texts:
I'm trying to solve the cocktail party problem (Independent Component Analysis) in a real application. So let's say there are two speakers and two microphones. What I'd to know is if it's mandatory that the two microphones start to record exactly at the same time.
Thanks for your time.


texts: A doubt about the GridSearchCV function in Sklearn?

texts:
When creating different hyperparameter combinations, does the function evaluate combination 1 on the same fold as combination 2? As in, are the folds the same across combinations? I understand that for example for cv = 3, a third of the dataset will be held out for evaluation.  There will be three such splits, each containing a test fold. Will the split be different for another combination? I also think that it is unnecessary computation to split again but does it happen?

I could not find strong reference to this fact in the documentation except for a part about cv_results_ or it is just assumed that, this is the way it is.


texts: Impact of unlabelled documents for label prediction via SVM

texts:
I have a corpus of text documents, some of which are labelled by analysts with label L. I am using this data to train an SVM for predicting if a new document should have label L.

So far it's straight-forward, but there is an issue: The analysts have not evaluated all documents in the training set, so there are in fact three classes of documents:

Documents labeled L
Documents the analysts have looked at, and chosen not label L (so you could say they're labelled not-L)
Documents the analysts have not looked at.

Unfortunately, at training time, I have no way to separate documents in 2. and 3, or not-L and unlabelled documents. I believe this is a problem, because a not-L label gives information to the SVM, but an unlabelled document is more "neutral".

How can I estimate the impact of this issue on predicting if a new document should have label L?


texts: Using machine learning algorithm to approximate a matrix?

texts:
I don't know if this is a good question. I would like to use machine learning to approximate a matrix. For example, a very simple one as

\begin{align}
A=\begin{pmatrix}
1 &0 \\
 0& 2
\end{pmatrix}.
\end{align}

I can easily generate many input-output pairs by left-multiplying the input with $A$. The input are random vectors of dimension $2\times 1$. I use the following neural network to do the job

input layer : 2 nodes;

hidden layer : 1 node;

output layer : 2 nodes.

A very simple Matlab code is written as follows, but it doesn't work. The testing results simply blow up, showing NaN.

m=1000;                % number of input-output pairs
alp=0.001;             % learning rate

inp=rand(2,m);         % random input
oup=[1 0;0 2]*inp;     % constructing output

W1=rand(1,2);          % initialise the weights
W2=rand(2,1);

for epoch=1:1000       % running for 1000 epochs
    for i=1:m          % do the forward propagation and backpropagation
       a1 = inp(:,i);
       z2 = W1*a1;    a2 = actf(z2);
       a3 = actf(W2*a2);

       delta2 = a3 - oup(:,i);
       delta1 = actfd(z2).*(W2'*delta2);

       W2 = W2 + alp*delta2*a2';
       W1 = W1 + alp*delta1*a1';
    end
end

actf here is the activation function $f(x)=\frac{1}{1+e^{-x}}$ and actfd is its derivative $f(x)=x(1-x)$. I see no problem of overfitting since in $A$ matrix there are 4 entries and the NN also has 4 weights in total. But the code seems to be wrong.

I'm wondering if people here know how to solve this problem (approximating a matrix using ML)? Did I do anything wrong in the code above?

Thanks in advance for your help. Any comments will be appreciated.


texts: Time-series prediction: Model & data assumptions in AI/ML models vs conventional models

texts:
I was wondering if there was a good paper out there that informs about model and data assumptions in AI/ML approaches.

For example, if you look at Time Series Modelling (Estimation or Prediction) with Linear models or (G)ARCH/ARMA processes, there are a lot of data assumptions that have to be satisfied to meet the underlying model assumptions:

Linear Regression

No Autocorrelation in your observations, often when dealt with level data (--> ACF)
Stationarity (Unit-Roots --> Spurious Regressions)
Homoscedasticity
Assumptions about error term distribution "Normaldist" (mean = 0, and some finite variance)
etc.

Autoregressive Models

stationarity
squared error autocorrelation
...

When dealing with ML/AI approaches, it feels like you can throw whatever you like as an input (my subjective perception). You are satisfied with the result as long as some prediction/estimation error measurement is good enough (similar to a high, but often misleading R²).

What assumptions have to be satisfied for an RNN, CNN or LSTM model that find application in time-series prediction?

Any thoughts?

ADDED

Good Article describing my question/thoughts.
Medium Article discussing model assumptions + tests, but not in the context of more advanced models
I read the 100-page ML Book- Unfortunately almost no content about model assumptions or how to test for them.


texts: Logistic Regression Model for categorical features with multiple values in each category

texts:
I am working on an insurance use case to build a logistic regression classifier to predict if a policy will lapse or not. The dataset has more than 20 categorical features for a policy. Each categorical feature in itself can have multiple values out of which only one will be applicable to a policy at a time. e.g., there could be multiple ways of premium payment (like annually, half-yearly, quarterly or monthly). The policy will have either of these value attached to it. As part of feature reduction, I used chi-square test against my target feature (if a policy will lapse or not) and removed a few features. After this, I created dummy variables (0 or 1) for all of them and tried to find a correlation using .corr() function in pandas. This operation helped to identify some dummy features have a high correlation with target feature e.g a specific value of product type, renewal type, source of business. Using these dummy features variable I trained my logistic regression classifier which is giving an accuracy of 94%, precision of 81%, recall of 98% and area under AUC value of 0.98, which I believe is a case of model overfitting.

I believe, if I use this model on a policy which does not have categorical feature values on which it is trained, it will fail miserably.

Please guide me if it is correct to use individual values of categorical feature to train the model? And if this value is not present/applicable to a new data then the model will not be able to predict with a high level of accuracy. Please guide how is feature selection done in such a case and how should the model be trained. Will appreciate if link to similar implementations could be provided.


texts: Are all classifiers linear in some high dimensional space?

texts:
Of all possible classifiers (including SVMs, locally weighted regression, softmax regression, lots others I'm sure I don't know about, etc.), are they all linear in some high dimensional space?

E.g. An SVM with a Gaussian kernel is linear in infinite-dimensional space.


texts: Understanding dropout and gradient descent

texts: I am looking at how to implement dropout on deep neural networks and found something counter intuitive. In the forward phase dropout mask activations with a random tensor of 1s and 0s to force net to learn the average of the weights. This help the net to generalize better. But during the update phase of the gradient descent the activations are not masked. This to me seems counter intuitive. If I mask connections activations with dropout, why I should not mask the gradient descent phase?


texts: Class weighting during validation in Keras

texts: I would like to know if the class weighting is also used in evaluating the loss in the validation data during the training. If not, is there a way to adjust the fit() function so that it takes into account the class weighting during the validation loss calculation.


texts:
I have a dataset which contains data related to the exchange rate in a certain time period(2013-2015). The dataset has a column date with YYYY/MM/DD format and USD/EUR which contains the exchange rate.
I would like to know what are the preprocessing steps need to follow and how to predict the values using neuralnet package and compare with test data?


texts:
I am training an Xgboost using 60% of my data and use 40% for testing.

In the 60% of data, I use 5-fold validation to find the best number of trees. I find that the optimal number of trees is around 150.

I evaluate my model in the 40% of the data that I have left and I am happy with the performance, so now I can deploy my model. However, before that, I want to do a final retrain to make use of all my data. I wonder if the number of trees obtained in cross-validation is going to be optimal when I train with the full dataset as well. My intuition says that I should use more than 150 trees as I have more data and I can overfit less.

Are there any sound decisions regarding the number of trees of the retrained model?

In my mind I have at least 3:

Use the full data to cross-validate and obtain the optimal number of trees (this is ok for sure but it is the slowest).
Use the same number of trees (this is the most conservative, we might be underfitting the data).
Use a heuristic, like final_trees = 150*100/60. I am very interested in a legit heuristic where I would not underfit and where I don't need to train models on cross-validation again.

Have you heard of any heuristic like that?

Note: this is not only specific to Xgboost, as any model with a parameter that controls regularization can also have the same issues.


texts: In generative adversarial models (GANs), why should we solve min-max problem and not max-min?

texts: I know that in GANs model, there is min-max game between generator and discriminator which discriminator tries to maximize the loss function and the goal of generator is to minimize it. But why we write the loss function as min-max problem and not max-min? As I know, the loss function is not convex, so there is difference between these approaches.


texts: RMSE is higher for bigger values of target variable - how to decrease

texts:
I am solving a problem with machine learning and I have some data with two integer type independent variables and a continuous dependent variable. I am optimising to RMSE. I had fairly large RMSE value on my validation data. I learned that my model didn't do good on larger values of target; so, I have tried removing rows with larger values and that didn't help. So, now in the process of understanding the mistakes, I calculated RMSE for each ground truth value and it's prediction from validation set and plotted it to understand where big mistakes had happened. Apparently, my model still doesn't do good at larger values of target.
Here is the plot:

And here is the plot showing relationship between ground truth values and predictions:

As you can see, my model's predictions got worse as the values got large. How do I prevent this?
Some information about my data(only what I can reveal):

There is absolutely no linear relationship between independent variables and target. So, I have used a tree based model and random forest is giving me relatively good results.
I can even went to say that both of my independent variables can be called categorical variables with very high categories.
Also, there are a lot of values in independent variables, that occur only once.
all the variables are highly skewed to the right.(range of IV_1: 0 to 3,700; IV_2: 0 to 40; target variable: 0 to 39,000)

How do I decrease my RMSE or doing what will decrease it?


texts:
First of all, I hope I'm in the right StackExchange here. If not, apologies!

I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million possibly). They contain some linguistic/syntactic features and their values are all strings.

Because most classifiers do not handle string data as values, I convert them to binary frequency values, so an attribute looks like this:

@attribute 'feature#value' numeric

And per row, the value is either 1 or it is absent (so note it's a sparse ARFF file).

The thing is, with 250K rows, there are over 500K attributes and so, most algorithms have a hard time with this.

There are a lot of algorithms. I'm really curious as to what you would consider a suitable one (preferably unsupervised, but anything works), and if you even have some ideas how I could improve performance. I can train on small subsets of data, but the results only get better when using large amounts of data (at least 7 million events).

For now, I've been using NaiveBayes variations (Multinomial and also DMNBText) and those are really the only ones that are able to chew up data with acceptable speed.

Thanks a lot. If you need more information, please let me know.

Cheers.


texts: Would Topic Modelling be classified as NLP or NLU?

texts:
I recently started my journey into the world of NLP, it's been one heck of a ride. I'm currently trying to understand whether topic modelling would be considered as NLP or NLU.

Initially I would assume that topic modelling would be classified as NLP. However, if we use word embeddings for topic modelling wouldn't it then be classified as NLU, as we have deeper understanding of how the words relate to each other in vector space?

Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two?

Your insight regarding this matter would be highly appreciated.


texts: random forest classifier - impact of small n_estimator and repeated training

texts:
trying to have a better understanding of random forest algorithm here. With the same training and holdout datasets, I tried two things here:

Set a small n_estimator (10), train on my training dataset and apply to my holdout dataset. If I repeat this several times, the result (e.g. correctly predicted target class) varies somewhat from run to run. My understanding is that since the # of trees is small in my model, there are variations in my model after training thus leading to different results.
Set a high n_estimator (300) and do the same. Then the results don't vary. My take is that impact of high n_estimator reduces variation in the model and thus i get the same prediction every time.

So if I run my scenario 1 a bunch of times and consolidate the results (i.e. run 1 predicts A B in class 1, run 2 predicts A C in class 1, run 3 predicts D in class 1), my final results would be A B C D are in class 1. My question is:
1. Is this essentially the same as running it once with a large n_estimator?
2. Is this approach problematic because I am relying more on "guessing" (e.g. small n_estimator leads to larger variation in outcomes)?

Thanks!


texts: Creating a Feature to determine popularity

texts:
I am building a recommendation system where I have multiple categories. I would like to  Know how popular a product is in each category. For that, I am considering probability as one factor. For example, I have 3 categories ($C1$,$C2$,$C3$) so, I am calculating:
(Number of times a particular item in C1 is purchased)/total number of items purchased from C1)
This gives me the probability of each item within a category. Apart from that, I am also trying to consider:
(Total number of times an item has been purchased/total number of times it has been viewed)
However, I do not understand how to use the purchase/view ratio with probability. Apart from that, what another factors can I conclude?


texts: Create a weighted graph from a continous text

texts:
My goal: plot a graph that shows how much the words of a large text are connected to each other.
My idea is that the closer the range between words, somehow affected by the occurence, the more connected the words are in the text.
I browsed the web for quite some time and noticed that there is a vast amount of algorithms and techniques in the field of text mining. It seems like i can't find a way to start without a deep study of all these topics and i need someone that can point me in the right direction.
my naive approach would be to calculate the distance of every word to any other word and somehow create a graph with weighted distances out of it but this seems like a pretty uncomputable task. Well i know there must be better solutions out there.
Thank you for any suggestions.


texts: Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input?

texts:
fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to autoregresive models where each forward run predicts the next token from the input and the previous predicted tokens)
However, from the code it appears that the models still expects the previous tokens as input:
def forward(self, src_tokens, src_lengths, prev_output_tokens, tgt_tokens, **kwargs)

https://github.com/pytorch/fairseq/blob/master/fairseq/models/nat/nonautoregressive_transformer.py#L78


texts: How would one separate digits for number recognition?

texts:
Assuming that one has a neural network capable of returning the numerical digit from a given image of size 28x28px.

How would one split an image of unknown size and an unknown amount of digits into a series of 28x28px images to feed to this network? (The order of the digits must be obtainable.)

For example:

How can this be split up to:

Assuming that there are not always 5 digits and the initial image is not always the same size.

Firstly, my thoughts would be to create a secondary neural network. This neural network would output an (x, y) coordinate. This could be used to crop the image to 28x28px about this coordinate. However, this neural network would only be able to locate 1 digit at a time.

Secondly, another idea is that a series of random crops could be performed and then all given to the number recognising neural network. However, this would create a high error rate and the number recognising neural network has no way of telling that there was no valid number given (Unless that output was added). But more importantly the order of the digits would/could be lost.

I am struggling to find any resource explaining a possible solution. Google's house number recognition takes the entire image of digits and returns a value. This can be found in How Google Cracked House Number Identification in Street View and Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks.


texts: Choosing between CPU and GPU for training a neural network

texts:
I've seen discussions about the 'overhead' of a GPU, and that for 'small' networks, it may actually be faster to train on a CPU (or network of CPUs) than a GPU.

What is meant by 'small'?

For example, would a single-layer MLP with 100 hidden units be 'small'?

Does our definition of 'small' change for recurrent architectures?

Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?

EDIT 1:

I just found a blog post (possibly outdated? It's from 2014):

"...Most network card[s] only work with memory that is registered with the CPU and so the GPU to GPU transfer between two nodes would be like this: GPU 1 to CPU 1 to Network Card 1 to Network Card 2 to CPU 2 to GPU 2. What this means is, if one chooses a slow network card then there might be no speedups over a single computer. Even with fast network cards, if the cluster is large, one does not even get speedups from GPUs when compared to CPUs as the GPUs just work too fast for the network cards to keep up with them.

This is the reason why many big companies like Google and Microsoft are using CPU rather than GPU clusters to train their big neural networks. "

So at some point, according to this post, it could have been faster to use CPUs. Is this still the case?

EDIT 2: Yes, that blog post may very well be outdated because:

Now it seems that GPUs within a node are connected via PCIe bus, so communication can happen at about 6GiB/s. (For example: https://www.youtube.com/watch?v=el1iSlP1uOs, about 35 minutes in). The speaker implies that this is faster than going from GPU1 to CPU to GPU2. It would mean the network card is no longer the bottleneck.


texts: Product Prediction to group of customers

texts:
I have multiple groups of customer, say for segment 1 as shown in the pictures, I have a list of products that I can choose the cross-sell to that group.
Consider the product has multiple features, e.g., prices, types, how do I come up with a model that recommend top 3 products for the segment of customers?

The difficuties lie in:
For segment of customers, we have range of values (e.g., for ALL_FEES we have \$13-\$4835), how to model that?


texts: t-SNE on extremely high-dimensional spaces

texts:
I successfully applied t-SNE to the number handwriting dataset. n=3823 data points (i.e. handwritten numbers) in an D=64 dimensional space (i.e. 8x8 pixels). Worked great.

Now I would like to cluster n≈60 data points in an D≈3000 dimensional space. Even after many iterations, t-SNE fairs far worse than say PCA.

Is there an upper bound on the number of dimensions (relative to the number of data points) above which applying t-SNE is not adviced?


texts:
In XGBoost documentation it's said that for ranking applications we can specify query group ID's qid in the training dataset as in the following snippet:

1 qid:1 101:1.2 102:0.03
0 qid:1 1:2.1 10001:300 10002:400
0 qid:2 0:1.3 1:0.3
1 qid:2 0:0.01 1:0.3
0 qid:3 0:0.2 1:0.3
1 qid:3 3:-0.1 10:-0.3
0 qid:3 6:0.2 10:0.15

I have a couple of questions regarding qid's (standard LTR setup set of search queries and documents, they are represented by query, document and query-document features):

1) Let's say we have qid's in our training file. Does it mean that the optimization will be performed only on a per query basis, all other features specified will be considered as document features and cross-query learning won't happen?

2) Let's assume that queries are represented by query features. Should we still have qid's specified in the training file or we should just list query, document and query-document features?

UPDATE:

So far, I have the following explanation, but how correct or incorrect it is I don't know:

Each row in the training set is for a query-document pair, so in each row we have query, document and query-document features. If we specify "qid" as a unique query ID for each query (=query group) then we can assign weight to each of these query groups. If the weight in some query group is large, then XGBoost will try to make the ranking correct for this group first.

From a file in XGBoost repo:

weights = np.array([1.0, 2.0, 3.0, 4.0])
...
dtrain = xgboost.DMatrix(X, label=y, weight=weights)
...
# Since we give weights 1, 2, 3, 4 to the four query groups,
# the ranking predictor will first try to correctly sort the last query group
# before correctly sorting other groups.

and also:

In ranking task, one weight is assigned to each query group
(not each data point). This is because we only care about the
relative ordering of data points within each group, so it
doesn't make sense to assign weights to individual data points.

UPDATE 2:

Found this link.  Given

3 qid:1 1:1 2:1 3:0 4:0.2 5:0 # 1A
2 qid:1 1:0 2:0 3:1 4:0.1 5:1 # 1B
1 qid:1 1:0 2:1 3:0 4:0.4 5:0 # 1C
1 qid:1 1:0 2:0 3:1 4:0.3 5:0 # 1D
1 qid:2 1:0 2:0 3:1 4:0.2 5:0 # 2A
2 qid:2 1:1 2:0 3:1 4:0.4 5:0 # 2B
1 qid:2 1:0 2:0 3:1 4:0.1 5:0 # 2C
1 qid:2 1:0 2:0 3:1 4:0.2 5:0 # 2D
2 qid:3 1:0 2:0 3:1 4:0.1 5:1 # 3A
3 qid:3 1:1 2:1 3:0 4:0.3 5:0 # 3B
4 qid:3 1:1 2:0 3:0 4:0.4 5:1 # 3C
1 qid:3 1:0 2:1 3:1 4:0.5 5:0 # 3D

the following set of pairwise constraints is generated (examples are referred to by the info-string after the # character):

1A>1B, 1A>1C, 1A>1D, 1B>1C, 1B>1D, 2B>2A, 2B>2C, 2B>2D, 3C>3A, 3C>3B, 3C>3D, 3B>3A, 3B>3D, 3A>3D

So qid seems to specify groups such that within each group relevance values can be compared to each other and between groups relevance values can't be directly compared (inc. during the training procedure). So during training we need to have qid's and during inference we don't need them as input.

Thank you!


texts: Is there an R package for Locally Interpretable Model Agnostic Explanations?

texts: One of the researchers, Marco Ribeiro, who developed this method of explaining how black box models make their decisions has developed a Python implementation of the algorithm available through Github, but has anyone developed a R package? If so, can you report on using it?


texts: How to select features which have non zero importance using SKLearn's Decision Trees Classifier?

texts: I need to discard all the features which have zero importance and keep only those features which have non zero importance while implementing DecisionTreesClassifier. The feature importance here is determined using clf.feature_importance_ parameter of DecisionTreesClassifier. How can this be done?


texts: How to get Keras accuracy for each step in an epoch like in Tensorflow?


texts: I have a regression problem which I have to predict 3 numerical values from a provided data. For example let's say I have a data set containing X1,X2,X3,X4,X5,X6...X100,Y1,Y2,Y3 columns. In this case I have to predict Y1,Y2,Y3 values. Is there are any way to construct the model to get all the outputs at the same time using Keras.


texts: Does a variable with different range for rows affect the classification model

texts:
I am trying a binary classification on a Time Series data using XGBOOST and have sales variables with different currencies for different customers. Will different currencies (basically variable with different range) have a significant impact on the model? The two solutions I could think of is:

Conversion for all rows for sales variable in a single currency.
Adding a region of sales/currency as a categorical variable.

Will the second solution work or I should stick with the first one.


texts: why evaluate_generator accuracy too much by using flow_from_directory with shuffle true?

texts:
i am using tensorflow and keras on colab i train with shuffled data but i met this strange accuracy with evaluate_generator with my training set that has 95% acc

if i did not shuffle my data with evaluate_generator

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=32,
    shuffle=False,
    class_mode="sparse",
    subset='training')

i get that

acc = model.evaluate_generator(train_generator, steps=2, verbose=1)
print(acc)
2/2 [==============================] - 1s 293ms/step - loss: 1.7815 - acc: 0.7344
[1.781482219696045, 0.734375]

but if i used shuffle like i did with training epochs

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=32,
    shuffle=True,
    class_mode="sparse",
    subset='training')

i got my acc back

acc = model.evaluate_generator(train_generator, steps=2, verbose=1)
print(acc)
2/2 [==============================] - 19s 9s/step - loss: 0.2059 - acc: 0.9219
[0.20589596964418888, 0.921875]

as i can not say i train model and got high accuracy then doing evaluate to my model by my train set i must get near accuracy to it ...
same with my validation set

so why shuffling my data effect my evaluation of my model

(i trained model with shuffle)

edit
my model : ResNet34

my classes number: 7

complie function :

model.compile(loss='sparse_categorical_crossentropy',
     optimizer='rmsprop',
              metrics=['accuracy'])

i used 10000 images dataset with split 0.2 rate 8000 images for train

and now i retrain my model(with non shuffled data) it is like shuffled data is complete strange than without shuffle , when i start training loss was too high and get smaller but this was model with training data and 26 epochs and acc > 90% and re-continue train with low acc and high loss


texts: Score probability groups with Random Forest in R

texts:
I would like to use RF in order to get a score probability of a binary outcome, in order to know groups with higher probability of buy something. Like this:

fit<-randomForest(formula = buy ~.  , data = pract,importance = TRUE ,ntree =1000)

pract$PREDICTION <- as.vector(predict(fit,type='prob',newdata=pract)[,2])

test$PREDICTION <- as.vector(predict(fit,type='prob',newdata=test)[,2])

First Question, When I perfom auc of the pract$PREDICTION I get 1! It is because overfiting? Should I use out of bag instead of pract, like this?:

pract$PREDICTION <- as.vector(predict(fit,type='prob')[,2])

Second Question:

With test$PREDICTION I want to make groups ordered by the score prob, like this:

test$score[test$PREDICTION >= 0 & test$PREDICTION <=0.3] <-  1
    test$score[test$PREDICTION > 0.3 & test$PREDICTION <=0.6] <- 2
...
test$score[test$PREDICTION > 0.9 & test$PREDICTION <=1.0] <- 10

How can I measure how good my model is? by doing the same segmentation group in the pract$PREDICTION and then comparing how right and how wrong both tests are in each group ? This make any sense?


texts:
I'm relatively new to data science / machine learning (yes, I know) and am experimenting with text analysis. I only want a relatively naive approach and am looking to know whether my approach is valid procedurally even if I may not get perfect results. I want to compare my model to a fulltext tf-idf search of specific skills/keywords against an index of resumes.

The task: Classify job descriptions, rather cluster them in an unsupervised way. Eventually I want to match resumes against them as a basic implementation of my idea. I was thinking that a naive implementation of word embeddings and kmeans would be fine. Basically, extract document_vectors from specific job descriptions, then cluster the job descriptions based on these vectors. This would create a ghetto "labeling" system so that I could then apply a classification algorithm Then I would use resume raw_text and extract a new vector, applying a simple classification it into one of these clusters (regression).

To summarize

1. Use a large database of job descriptions
2. Tokenize the text, and apply doc2vec and generate document vectors
3. Cluster the documents and group similar job descriptions
    NOTE: as to how many clusters to use, that will have to be tuned
4. Tokenize the raw_text of a resume
5. Use regression (or some technique) to find the closest cluster based on the resume vector compared to the cluster vector.

Now, I realize there are most likely some huge holes in the way I'm doing this (I'm using intuition based on no training). For example, my assumption that I can even match resume to job description is based on the idea that these are even comparable text structures - and how good the matching is is also dependent on how good the clustering of documents is.

I would like to know if this approach is good for something naive or whether I'm missing anything critical. I'm using nltk, scikit-learn, and gensim Word2Vec


texts:
I am trying to detect timeline of brands histories. For my specific case, I believe it is easy because data is already clustered. For each Wikipedia article I can spot sentences surrounding dates. Here is an example:

McDonald's Corporation is an American fast food company, founded in
1940 as a restaurant operated by Richard and Maurice McDonald, in San
Bernardino, California, United States. They rechristened their
business as a hamburger stand, and later turned the company into a
franchise, with the Golden Arches logo being introduced in 1953 at a
location in Phoenix, Arizona. In 1955, Ray Kroc, a businessman, joined
the company as a franchise agent

From this, it is easy to narrow results programmatically to

McDonald's is founded in 1940
Golden Arches logo introduced in 1953
Ray Kroc, a businessman, joined the company in 1955

This seems easy if documents are clustered. If not, I am thinking of a basic algorithm to mine timelines 'or natural numbers). So I want to discuss existing studies and my intuition here.
Definitions:

Timeline: a logical succession of events on one single subject.
Dates in a timeline are natural numbers, and can be "unordered relatively".
Timelines are continuous (one range like) and cannot intersect.

Let's ignore the NLP related part, and try to figure out timelines in natural numbers ignoring topics (1st definition).
Distance: Initial timeline length. It represents the minimum.
Example:
Step A
1, 4, 2, 5,  3, 8, 7, 9, 20, 21, 23, 24, 1, 5, 7, 9

dist = 4

Becomes:

1, 4, 2, 5/  3, 8, 7, 9/ 20, 21, 23, 24/ 1, 5, 7, 9

Score each set (of 4 elements): Scoring is critical but lets think of bubble sort score, where score = 1 / number exchange ops.
1,4,2,5 => 1/1  | 3,8,7,9 => 1/1

Step B
The reason to score sets is to identify if a set represents a timeline or the combination of two sets represent a timeline, to decide, we score the combined set and divide by two
1,4,2,5,3,8,7,9 => 5/2

We conclude 1,4,2,5 and 3,8,7,9 are two sets, while 1,4,2,5,3,8,7,9 is not.
We move sequentially to process next sets.
The reason I said Distance is minimum is that before comparing scoring initial sets, we first   identify sets of 4, 5, 6 or more elements and score them ((step A)) and only take separate sets with better score (minimum bubble sort score here).
Any thoughts ?


texts: Is it possible to change test and train data size when using crossvalind function with Kfold param?

texts:
I was looking at MATLAB Help and want to work with "crossvalind" function.
It would two parameters that you can use it.

If you use "HoldOut" you can define partition size of test and train data set and when you use "Kfold" you can define "fold counts".

Now I want to know is there a way to use "Kfold" parameter and define partition size of test and train data sets?

I checked it and it seems when you use "Kfold" as bellow code, it always default partition size like 75% to 25% .

indices = crossvalind('Kfold',species,10);


texts:
I have to think about a model to identify prospects (companies) that have a high chance of being converted into clients, and I'm looking for advice on what kind of model could be of use.

The databases I will have are, as far as I know (I don't have them yet), the list of current clients (in other words, converted prospects) and their features (size, revenue, age, location, stuff like that), and a list of prospects (that I have to score) and their features. However, I don't think I'll have a list of the companies that used to be prospects but for which the conversion to clients failed (if I had, I think I could have opted for a random forest. Of course I could still use a random forest, but I feel it would be a bad idea to run a random forest on the union of my two databases, and treat the clients as converted and the prospects as non-converted...)

So I need to find, in the list of prospects, those who look like the already existing clients. What kind of model can I use to do that ?

(I'm also thinking about things such as "evaluating the value of the clients and apply this to the similar prospects", and "evaluating the chance each prospect has of going out of business" to further refine the value of my scoring, but it's kinda out of the scope of my question).

Thanks


texts: Question on bias-variance tradeoff and means of optimization

texts:
So I was wondering how does one, for example, can best optimize the model they are trying to build when confronted with issues presented by high bias or high variance. Now, of course, you can play with the regularization parameter to get to a satisfying end, but I was wondering whether it is possible to do this without relying on regularization.

If b is the bias estimator of a model and v of its variance, wouldn't it make sense to try to minimize b*v?


texts: What is wrong with a neural network model which is so dependent on the seed of initialization?

texts: I have a fully-connected neural network with one hidden layer with 2 units which its goal is to classify a dataset with 324 samples and 300 features. 50% of the dataset is used for train and 50% of it is used for the test. There are two classes in the dataset which one of them has 75 samples and the other one has 249 samples. When I train the model with different seeds (actually when I re-run the training function), its results on the test set have a high variance, and the accuracy of the classification for test set changes about 20%. What is wrong with the model? How can I make it stable? How can I report the results of it?


texts: Web scrapping with beautifulSoup is done slowly

texts:
I have developed a web scrapping code in Python which takes data from Hattrick.org's matches and returns them in a table so it can be mined, determined likelihood of goals, etc.

I have the difficult that is really slow, returning 12.000 rows in 5 hours or so.

This question is to ask if there is a way to improve the web scrapping technique so it does not take that amount of time.

This is the code in Python.

import requests
from bs4 import BeautifulSoup
import re
import pandas as pd
import numpy as np

ini = 631163587
q = 200000 # Change to q = 10 to try a sample

Cols = {'01. Local MF',
        '02. Away MF',
        '03. Local RD',
        '04. Away RD',
        '05. Local CD',
        '06. Away CD',
        '07. Local LD',
        '08. Away LD',
        '09. Local RA',
        '10. Away RA',
        '11. Local CA',
        '12. Away CA',
        '13. Local LA',
        '14. Away LA',
        '15. Local IndD',
        '16. Away IndD',
        '17. Local IndA',
        '18. Away IndA',
        '19. Local Attitude',
        '20. Away Attitude',
        '21. Local Tactic',
        '22. Away Tactic',
        '23. Local Tactic Level',
        '24. Away Tactic Level',
        '25. Local Score',
        '26. Away Score'}

df_ht = pd.DataFrame(data=np.nan,index=range(ini,ini+q),columns=Cols)
cont=[]

for i in range(ini,ini+q):
    url2 = 'https://www74.hattrick.org/Club/Matches/Match.aspx?matchID='+str(i)
    response = requests.get(url2)
    soup = BeautifulSoup(response.text, 'html.parser')
    s1 = soup.findAll('td')

    m = soup.findAll('meta')[10].attrs['content']
    d = re.findall('[ ,.,A-Z,a-z,0-9]* - [., ,A-Z,a-z,0-9]*',m)
    d2 = re.findall('[0-9]+',d[1])

    partido = d[0]

    try:
        D = {'01. Local MF': float(s1[3].contents[0]),
                          '02. Away MF': float(s1[4].contents[0]),
                          '03. Local RD': float(s1[10].contents[0]),
                          '04. Away RD': float(s1[11].contents[0]),
                          '05. Local CD': float(s1[17].contents[0]),
                          '06. Away CD': float(s1[18].contents[0]),
                          '07. Local LD': float(s1[24].contents[0]),
                          '08. Away LD': float(s1[25].contents[0]),
                          '09. Local RA': float(s1[31].contents[0]),
                          '10. Away RA': float(s1[32].contents[0]),
                          '11. Local CA': float(s1[38].contents[0]),
                          '12. Away CA': float(s1[39].contents[0]),
                          '13. Local LA': float(s1[45].contents[0]),
                          '14. Away LA': float(s1[46].contents[0]),
                          '15. Local IndD': float(s1[54].contents[0]),
                          '16. Away IndD': float(s1[55].contents[0]),
                          '17. Local IndA': float(s1[61].contents[0]),
                          '18. Away IndA': float(s1[62].contents[0]),
                          '19. Local Attitude': (s1[67].contents[0]),
                          '20. Away Attitude': (s1[68].contents[0]),
                          '21. Local Tactic': s1[70].contents[0],
                          '22. Away Tactic': s1[71].contents[0],
                          '23. Local Tactic Level': s1[75].contents[0],
                          '24. Away Tactic Level': s1[76].contents[0],
                          '25. Local Score': float(d2[0]),
                          '26. Away Score': float(d2[1])}

        df_ht.loc[i,:] = D

    except:
        cont.append(i)

    df_ht.to_csv(r"Datos9.csv")


texts: An unbiased simulator for policy simulation in reinforcement learning

texts:
I'm reading the following paper

https://arxiv.org/pdf/1003.0146.pdf on building a fair online simulator for contextual bandits. In particular i'm interested in the proof on page 665 of Theorem 1

I don't believe this has been proved rigourously so i wanted to try and prove it myself. Is the following correct?

For any given tuple $(x_{1},\ldots,x_{k},a,r_{a})$ the probability that the policy evaluator adds this tuple to the history $h_{t-1}$ is the same as the probability that the for loop terminates with a match on $(x_{1},\ldots,x_{k},a,r_{a})$. Since the stream is infinite this success can come at any time from $r=1,\ldots,\infty$. The probability that we have a success at time $r=p$ is equal to the probability of having $p-1$ failures and a sucess on the $p$th trial and thus is equal to

$((k-1)/k)^{p-1}P((x_{1},\ldots,x_{k},a,r_{a})\times 1/k$ the total probability of obtaining event $(x_{1},\ldots,x_{k},a,r_{a})$ in the simulated environment at time $h_{t}$  is then

$\sum_{p=1}^{\infty} ((k-1)/k)^{p-1}P((x_{1},\ldots,x_{k},a,r_{a})\times 1/k=P((x_{1},\ldots,x_{k},a,r_{a})$ and thus the policy evaluator is unbiased

Is this correct? If not could someone provide a more detailed proof? Thanks!


texts: Why do we have multiple neurons in the output layer of a neural network?

texts:
Why do we have two neurons in the output layer?

What does each neuron mean?

If our classifier is a binary classifier, will we have only one neuron in the output layer?

Here is a picture of the neural network:


texts: Building a static local website using Rmarkdown: step by step procedure

texts:
I am trying to understand the procedure of building a static local website using R and Rmarkdown. I am aware of a Rmarkdown website where the procedure is outlined, but unfortunately I do not understand the steps.

Does anybody here have some experience in building a static local website and would be so kind as to describe the procedure in more detail?


texts: PPO, A2C for continuous action spaces, math and code

texts:
Edit:
Question has been edited to better reflect what I learned after asking the original question.

I implemented the clipped objective PPO-clip as explained here: https://spinningup.openai.com/en/latest/algorithms/ppo.html

Basically I used a dummy actor network to find the new action probability without training the local actor network.

"""use temp_actor to get new prob so we don't update the actual actor until
        we do the clip op"""
        curr_weights = self.actor.get_weights()
        self.temp_actor.set_weights(curr_weights)
        self.temp_actor.fit(state, advantages, epochs=1, verbose=0)
        new_policy = self.temp_actor.predict(state, batch_size=1).flatten()
        new_aprob = new_policy[action]

Then I worked out the ratio of action probabilities and implemented the PPO clipping parts of the algorithm:

ratio = new_aprob / old_aprob
        # scale = min(ratio * advantages, K.clip(ratio, 1 - self.epsilon, 1 + self.epsilon) * advantages)
        no_clip = ratio * advantages
        clipped = np.clip(ratio, 1 - self.epsilon, 1 + self.epsilon) * advantages

        self.actor.fit(state, np.minimum(no_clip, clipped), epochs=1, verbose=0)

The full code is here (please excuse some coarse language in the comments):    https://github.com/nyck33/openai_my_implements/blob/master/cartpole/my_ppo_cartpole.py

How can I adapt it for a continuous action space problem such as Pendulum v0.
Update:  I just read that the distribution of actions is a normal or Gaussian distribution here:  Reddit normal distribution

That seems strange since I imagined the curve of actions to be skewed one way or another depending on whether a particular state would get better results with certain action tendencies.

Update:  I found this on the Stable baselines site for PPO:

From this URI:  ppo explanation

I also saw that section 13.7 in Sutton's RL book seems to be a must-read for this type of problem as the section is titled:  Policy parameterization for Continuous Actions.

Furthermore I also learned in the process of searching for a readable implementation that GAE (advantage) is normalized so that probably makes an implementation more robust against wild fluctuations.  I also think my implementation is incomplete, ie. missing other components compared to the Pytorch "solution" I linked to in the answer.


texts: Unsupervised text classification with R/Python

texts:
I am relativity new to machine/deep learning and NLP. As a part of my Phd thesis I have scraped vast number of job vacancies (most of them are in Polish, and about 10% are in English ones) and then extracted required skills/competencies. As an output I got a vector of strings contains single skills/competences. I performed initial preparation (removed extra spaces, special symbols, stop words etc.) and built then a frequency distribution table to figure out the most demanded skills. The most commonly required skills were as follow:
prawo jazdy kat b ## driver license
umiejętność pracy w zespole ## team work
wykształcenie wyższe ## higher education
czynne prawo jazdy kat b ## driver license
znajomość pakietu ms office ## MS Office
prawa jazdy kat b ## driver license
prawo jazdy kategorii b ## driver license
fluent english
doświadczenie na podobnym stanowisku ## work experience
dobra znajomość języka angielskiego  ## English
excellent english skills
doświadczenie ## work experience

As you may see some of these competencies are quite similar. The problem is that job offers were provided by a different employers. European classification of skills/competences qualifications and occupations define about 3500 different competencies for instance.
My aim is to create trivial classifier using logistic regression, SVM or random forest that would be able to classify skills in real time using pre trained model. The problem is that I can not label vector of unique skills because of vast number of skills within its synonyms. I spent a while trying to figure out a solution.
So, the idea was to perform cluster analysis and to put similar skills into separate groups (clusters) and then label this clusters. In this case I would have much less group to label. Moreover, I would have a lot of synonyms within each group, which would increase precision of classification (I guess). Using labelled data set I would be able to trained model, and then to classify future job vacancies (skills). However, as I said I am new to machine learning, so I am not sure whether my solution would be good enough.
So, I am looking for advice/tutorials/links that would help me to solve my problem rather than for complete solutions. I can use both R and Python. Would be appreciated for any help.


texts: How does XGBoost use softmax as an objective function?

texts:
I'm quite used to seeing functions like log-loss, RMSE, cross entropy as objective functions and it's easy to imagine why minimizing these would give us the best model. What's difficult to imagine is how XGBoost uses softmax, a function used to normalize the logits, as a cost function. As mentioned in the docs here.

How can a softmax function be minimized?


texts: where to store embeddings for similarity search?

texts:
I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the application again:

text corpora (few hundred thousand documents, containing a few paragraphs)
embeddings create with BERT (for each paragraph)
Application: similarity search (retrieve similar paragraphs and reference to the document)

I've seen tutorials about creating embeddings with BERT etc. and it all works. The Crux I have is how to manage having a few million embeddings and searching for similar ones. Where to store them, plus the additonal information (raw text related to the embeddings and document which contains the text).
So the question is:
How does one store a few million embeddings (768-Dimensional numpy arrays) in an efficient and searchable way without using cloud-environments (data privacy reasons)?
Is Tensorflow Records the right answer?
Is it in the end a relational database?
Is it something different? It's my first NLP task and I might simply not know the obvious answer. However, searching on stackexchange and google didn't provide a solution.


texts: Methods for Determining Possible Causation Between Two Time Series

texts:
I'll set this question up as a simplified example:

We hypothesize that there is a causal relationship between average gasoline prices and road traffic in a particular city.  The data cover the same time period at the same frequency (e.g.: weekly) and contain no missing values. We'll put aside the idea that there could be confounding variables at work, presume that we've made the time series stationary (via differencing), and determine there is no autocorrelation in the series.

What are reliable methods to determine that if a shock occurs in the price data that a corresponding shock occurs in the traffic data at some time lag?

I've looked a ;little bit into Dynamic Time Warping, but I suspect that this is better suited to determining how to map one series to another rather than for determining causality.


texts: Why Mini batch gradient descent is faster than gradient descent?

texts:
According to me:
Mini Batch Gradient Descent :
1.It takes a specified batch number say 32.
2.Evaluate loss on 32 examples.
3.Update weights.
4.Repeat until every example is complete.
5.Repeat till a specified epoch.
Gradient Descent :
1.Evaluate loss for every example.
2.Update loss accordingly.
3.Repeat till a specified epoch.
My questions are:
1.As Mini batch GD is updating weights more frequently shouldn't it be slower than normal GD.
2.Also I have read somewhere that we estimate loss in SGD (ie. we sacrifice some accuracy in loss calculation for speed). What does it means and does it helps in increasing speed.


texts: What if MNIST dataset had another feature

texts:
MNIST is a famous data set of hand written digits. Suppose we knew who wrote digits for example, female, left handed, 25 yrs old.

How would I use these information in CNN in tensorflow? or any other library.

Digits are images and that what CNN handles well but gender, dominant hand, and age are not images. How would you use that information?


texts: Can ReliefF implicitly predict non-linear correlation between features and targets?

texts:
Let's assume that we have a collection of instances with their features and already labelled and train them for Relief/ReliefF classifier.
If the targets themselves have parameters that defined themselves, e.g. targets are category A, B, and C that determined by the range value of x, and the features of instances naturally non-linear correlate with x, could Relief/ReliefF tell us about it implicitly by determining the weights of each features?


texts: Neural Nets: unordered sets of ordered tuples as features of data

texts:
I'm working on a very small scale pet project in which inputs are essentially sets of (x, y) pairs, and are to be classified into categories, using deep learning, specifically using Keras (I know this may not be the best for this, but it's more of a proof of concept / I want to try it out).

However, I'm not sure how to go about representing the data.

I'm starting with a simple classification problem (i.e. if (a, b) is a feature of a sample, and a and b are both within 5% of a certain (c, d), then they are a positive example, and not otherwise), but I'm not sure how to represent the data such that the network can learn this.

I was thinking of doing one-hot encoding, but then the dimensionality of the data may grow immensely (x and y both take on values in a continuous interval), and I'm worried that it would not generalize well to data augmentation (I would augment the data by adding noise to each x, y).

Any ideas?


texts: How tangent distance works in three-dimentional space

texts: I've read about tangent distance used in machine learning but I don't really know how it works. Concretely, consider two very different manifolds M1 and M2 and two points x1, x2 on that two manifolds respectively. M1 is very far from M2. What if the tangent plane p1 of M1 at x1 isn't parallel to the tangent plane p2 of M2 at x2 ? In fact, p1 will intersect p2 in this case, so distance(x1, x2) is zero. This means that x1 and x2 will be classied into the same class regardless to the fact that M1 is very far from M2. Can anybody explain this paradox for me ? Thanks alot for your helps


texts: Keras binary classification model is not learning anything. How to improve my model?

texts:
I am trying to train a binary image classifier on a an imbalanced dataset of images which are really small in dimensions(the largest dimension is 40*70). I am augmenting images by grayscale, rescaling and resizing to 48*48. My keras model is not learning at all. Can you suggest some technique using which I can improve the model or data?
I figured out the problem. It was my data. Without knowingly I was doing single class classification while doing binary classification. Any advice what approach to be taken?


texts: What is the difference between a data-driven model and an empirical model?

texts:
Are they the same?
Empirical models, per Wikipedia, are

  any kind of (computer) modelling based on empirical observations rather than on mathematically describable relationships of the system modelled.

But since data-driven models are also based on observations, what is the difference?


texts: How to calculate the output shape of conv2d_transpose?

texts:
Currently I code a GAN to generate MNIST numbers but the generator doesnt want to work. First I choose z with shape 100 per Batch, put into a layer to get into the shape (7,7, 256). Then conv2d_transpose layer
to into 28, 28, 1. (which is basically a mnist pic)

I have two questions
1.) This code doesn't work for obvious. Do you have any clue, why?
2.) I am very aware how transpose convolution works but I can't find any resource to calculate the output size given input, strides and kernel size specific to Tensorflow. The useful information I found is https://arxiv.org/pdf/1603.07285v1.pdf but e.g. padding in Tensorflow works very different. Can you help me?

mb_size = 32 #Size of image batch to apply at each iteration.
X_dim = 784
z_dim = 100
h_dim = 7*7*256
dropoutRate = 0.7
alplr = 0.2 #leaky Relu

def generator(z, G_W1, G_b1, keepProb, first_shape):

    G_W1 = tf.Variable(xavier_init([z_dim, h_dim]))
    G_b1 = tf.Variable(tf.zeros(shape=[h_dim]))

    G_h1 = lrelu(tf.matmul(z, G_W1) + G_b1, alplr)
    G_h1Drop = tf.nn.dropout(G_h1, keepProb)  # drop out

    X = tf.reshape(G_h1Drop, shape=first_shape)
    out = create_new_trans_conv_layer(X, 256, INPUT_CHANNEL, [3, 3], [2,2], "transconv1", [-1, 28, 28, 1])
    return out

# new transposed cnn
def create_new_trans_conv_layer(input_data, num_input_channels, num_output_channels, filter_shape, stripe, name, output_shape):
    # setup the filter input shape for tf.nn.conv_2d
    conv_filt_shape = [filter_shape[0], filter_shape[1], num_output_channels, num_input_channels]

    # initialise weights and bias for the filter
    weights = tf.Variable(tf.truncated_normal(conv_filt_shape, stddev=0.03),
                          name=name + '_W')
    bias = tf.Variable(tf.truncated_normal([num_input_channels]), name=name + '_b')

    # setup the convolutional layer operation
    conv1 = tf.nn.conv2d_transpose(input_data, weights, output_shape, [1, stripe[0], stripe[1], 1], padding='SAME')

    # add the bias
    conv1 += bias

    # apply a ReLU non-linear activation

    conv1 = lrelu(conv1, alplr)

    return conv1

...

    _, G_loss_curr = sess.run(
        [G_solver, G_loss],
        feed_dict={z: sample_z(mb_size, z_dim), keepProb: 1.0} #training generator


texts: IOU and dice_coef exceed 100% in multi task learning

texts:
Im doing a multi task learning for road and center line extraction (2 classes) I used IOU and dice_coef as a metrics :
def dice_coef(actual, predicted, eps=1e-3):

    y_true_f = K.flatten(actual)
    y_pred_f = K.flatten(predicted)
    intersection = K.sum(y_true_f * y_pred_f)
    return (2. * intersection + eps) / (K.sum(y_true_f) + K.sum(y_pred_f) + eps)

def iou(actual, predicted):

    actual = K.flatten(actual)
    predicted = K.flatten(predicted)
    intersection = K.sum(actual * predicted)
    union = K.sum(actual) + K.sum(predicted) - intersection
    print(intersection, type(intersection))
    print(union, type(union))
    return 1. * intersection / union

I used Radam as an optimizer and weighted binary cross entropy as loss function, the IOU and Dice_coef for the second class(centerline) was more than 1, but for the first class (road)was within the normal range as shown here ,
the accuracy was within the normal range also for both classes.
I Found this discussion here and they mentioned that, should used an image dice not a batch dice, when i followed the same implementation of the image dice for IOU and dice_coef, nothing has changed still the metrics exceed 100%, my batch size =2, image size(224,224,3), my y_ture is between (0,1).
but even with this strange result Ive got a good predictions for both classes and
I should mentioned that I used the same first implementation for the IOU and dice_coef. for single task and it worked perfectly.
I dont know why Ive got this strange results, even when I used different implementation for the mentioned metrics.
Please any kind of help will be appreciated.


texts: I want to use KFold from mode_selection instead of cross_validation ut it didn't work for the object Kfold

texts:
from sklearn.model_selection import KFold
import xgboost as xgb
# Some useful parameters which will come in handy later on
ntrain = X_train.shape[0]
ntest = X_test.shape[0]
SEED = 123 # for reproducibility
NFOLDS = 10 # set folds for out-of-fold prediction
kf = KFold(shuffle=False, n_splits= NFOLDS, random_state=SEED)
def get_oof(clf, x_train, y_train, x_test):
    oof_train = np.zeros((ntrain,))
    oof_test = np.zeros((ntest,))
    oof_test_skf = np.empty((NFOLDS, ntest))

    for i, (train_index, test_index) in enumerate(kf):
        x_tr = x_train[train_index]
        y_tr = y_train[train_index]
        x_te = x_train[test_index]

        clf.train(x_tr, y_tr)

        oof_train[test_index] = clf.predict(x_te)
        oof_test_skf[i, :] = clf.predict(x_test)

    oof_test[:] = oof_test_skf.mean(axis=0)
    return oof_train.reshape(-1, 1), oof_test.reshape(-1, 1)
xgb_oof_train, xgb_oof_test = get_oof(xgb,x_train, y_train, x_test)

I tried for i, (train_index, test_index) in enumerate(kf.split(x_train)):

But got below error:

  KeyError: "None of [Int64Index([ 6996, 6997, 6998, 6999, 7000, 7001, 7002, 7003, 7004,\n 7005,\n ...\n 13981, 13982, 13983, 13984, 13985, 13986, 13987, 13988, 13989,\n 13990],\n dtype='int64', length=6995)] are in the [columns]"


texts:
I am currently working on an image classification problem. To ease the implementation I used transfer learning in Keras with Resnet50 as base model.
I have 2 classes, however there are many possibilities that the input image may not belong to both the classes. In such cases the CNN should be able to output as Unknown.
Note : I can not create a separate class for unknown as it can come from any distribution.

I read that the Bayesian neural network would help in such cases.

Could you please help me in understanding how can I implement Bayesian CNN with transfer learning. Any material or reference link would be really helpful.

Thank you


texts: How would one use a network trained on CIFAR-10 to find things in a photo?

texts:
There are a million and one examples and tutorials on how to train up a neural network on the sample sets like the MNIST data and the CIFAR-10 data, but how does one go from the toy examples of recognising 200x200 clips each containing a single centred object to a real problem like finding CIFAR-10 category objects (the dog and the cat below) within a picture, like I presume Google does for their photo annotation.

Can someone describe how one might approach this leap from the classroom to the real world?


texts: testing statistical significance when comparing regression models

texts: I have 3 models, a random forest, a XGBoost and a baseline regression model. I've performed a 5-fold cross validation on all models and use MAE as the scoring metric. So this means I basically have 3 MAE values to compare. My question is: How do I (statistically) test if and which model performs better? I've read that one should use a paired non parametric test but when there are enough data points one can assume normal distribution for the sample population (central limit theorem). I'm not sure which test to use, could someone please explain this to me?


texts: Extracting and classifying information from images of semi-structured text

texts:
My problem statement is to identify and label the images of text in a particular type of document (say Deposit Slips). The document can have many different formats but they do not stray too far from the norm (example: the date will usually always be in the upper half of the image). However, it's unlikely that the same field will occur at the same position every time.
So the three steps of my pipeline would be:

Identify the sections of the image with text and generate bounding boxes
OCR the text to recognize the characters and words in those boxes
Use these bounding boxes + text data to correctly classify the text with a particular label
Assume steps 1 and 2 are already done, I just need help with step 3.

My current approach is to OCR the entire image, then use a Named Entity Recognition model to label the correct parts. However, in this approach I am not utilizing the coordinate data (location of text with respect to image) at all and simply discarding it.
Concatenating the entire text also has the disadvantage that if a label and a field are on the same row, they are likely to be placed next to each other in the text (good) but if they are on different rows, there will most likely be extra text between the label and the field (bad), making it less useful
Example:
--------------------                  --------------
| Date: 02/10/2021 |        vs        |    Date:   |
--------------------                  | 02/10/2021 |
                                      --------------

To combat these disadvantages, is there any way to make use of this location information? What type of NN model should I look into at this stage?