texts: How to retrain a K-Modes model based on daily data?

texts:
I have read that retraining a model depends highly on what you are trying to achieve. I am conscious that maybe I need to retrain my model daily and after a certain time I have to train the model again from scratch. That's no problem. The thing is, if I were to retrain a K-Modes model, I would need the previous centroids saved. I think this is also not a problem. Seeing the implementation of K-Modes here, the parameter init can be Huang or Cao. In the K-Means implementation of sklearn (see here), we can pass a ndarray of shape (n_clusters, n_features). I can't find this feature in the K-Modes implementation. So my specific questions are:

Is saving the previous centroids sufficient to retrain the model?
Is there a way to pass previous centroids in the K-Modes implementation I have mentioned? If not, should I implement my own K-Modes?


texts: How to create visualisation on medical data?

texts:
I want to create a data visualization on medical data i.e. patient's medical history, allergies one is having, any chronic patient's name must be highlighted, etc. And, there can be a separate visualisation for the medicines, which shows medicines' availability, their expiry date can be highlighted by using mouse_hover_on functionality.  It will be useful for the doctors and the medical staff as well.

Problems:

bl.ocks.org has variety of base visualization models to work upon, but the schema of database I have (given by my mentor) is a bit confusing.
Also, he won't provide us with the actual data of our college due to its confidential nature, so can we get dummy data on the net?

Experience:

I have done work on Weka earlier to understand about data mining, and many different algorithms and classifiers.
I have built a generic data visualization model (for hierarchical data) using the D3.js
I have good working knowledge of Python. So, if anyone could please tell whether there is any hope for doing it(the data visualisation) in python.


texts:
I asked a data science question regarding how to decide on the best variation of a split test on the Statistics section of StackExchange. I hope I will have better luck here. The question is basically, "Why is mean revenue per user the best metric to make your decision on in a split test?"

The original question is here: https://stats.stackexchange.com/questions/107599/better-estimator-of-expected-sum-than-mean

Since it was not well received/understood I simplified the problem to a discrete set of purchases and phrased it as a classical probability problem. That question is here: https://stats.stackexchange.com/questions/107848/drawing-numbered-balls-from-an-urn

The mean may be the best metric for such a decision but I am not convinced. We often have a lot of prior information so a Bayesian method would likely improve our estimates. I realize that this is a difficult question but Data Scientists are doing such split tests everyday.


texts: Conceptual questions on MLP and Perceptrons

texts:
I am facing some confusion regarding the terminologies assocaiated to classification and regression problems esp. using the MLP and Perceptron models.
These are the following:

1) When the data is linearly inseparable, we use MLP. Here what is meant b "data"--is it the response or the input feature that is linearly inseparable?

2) If it is linearly inseparable then does it mean that the mapping function from input to output will always be non-linear? Hence, we prefer MLP or the latest new models such as deep learning?

3) Linear regression fails in the case of linearly inseparable data or can linear regression work for inseparable data but if the function mapping is nonlinear then it fails?


texts: Which stage should the correlation analysis be done?

texts:
I was thinking about it, but I couldn't find a logical explanation.

Mostly im following below steps after data become ready:

Correlation analysis and elimination
Apply dummy if categorical variables exist
Balance the data if data is unbalanced
Scale data
Feature selection (Backward, Stepwise etc.)
Train model

Where would the correlation analysis be applied for this path I followed would make more sense? After the data is balanced? After scaling? Or at first?


texts:
I  have questions of users and I want to classify them automatically without manually labelling them. What deep learning method would be good for text classification just from text (so unsupervised).

Does those algorithms have to rely on word embedding?


texts: Should reinforcement learning always assume (PO)MDP?

texts:
I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP.
However as I read A3C and recent vision based deep RL papers, it seems some of them aren't assuming MDPs but used RNNs or LSTM to make it seem as if it is MDP.

So my question is: how does reinforcement learning algorithms work without the assumption of (PO)MDPs?


texts: learning rate very low 1e-5 for Adam optimizer good practice?

texts:
I am used to of using learning rates 0.1 to 0.001 or something, now i was working on a siamese net work with sonar images. Was training too fast, overfitting after just 2 epochs. I tried to slow the learning rate lower and lower and I can report that the network still trains with Adam optimizer with learning rate 1e-5 and decay 1e-6. But I am curious if this is a good practice to use the learning rates so low?
I am using keras


texts: Survival regression with major event that won't happen

texts:
I would like to do some survival regression about the duration before the "death" of an individual. The final purpose is to know, given an individual, how long it should take before he'll most likely "die" (probability of the survival function to be less than 0.1 for instance).

My problem here is that I have, in my training set, a variable that influences a lot my target variable, but is not available for the test set (and won't happen in real life).

Let's say my training data is the following:

id   status   poison_time  death_time     sex
 0     1          90           92          f
 1     0          90          150          f
 2     1          90           91          f
 3     1          60          130          m
 4     0          60          150          m
 5     1          60           62          m

With :

status = 1 for a dead person and 0 for a censored data
poison_time : time corresponding to the injection of a poison
death_time : time of the death or last follow-up
sex : sex of the individual (not relevant here, imagine a bunch of useful variables)

I can't just ignore the influence of poison_time: although for some individuals, the poison won't be as effective (individual with id 3, or individuals that ended up right-censored). It has a real impact on death_time.

In my test data the poison is not injected, but I still would like to have a good idea of "how long should it take before an individual most likely die", knowing my other variables (sex, etc.)

Is it possible to still have relevant results with such corrupted data as a training set?


texts: What does this expression from gradient descent mean?

texts:
I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy):

''let's think about what happens when we move the ball a small amount Δv1 in the v1 direction, and a small amount Δv2 in the v2 direction. Calculus tells us that C changes as follows:''

I don't understand where this expression has come from. It looks like the product rule? Could someone either explain what it means and/or state the area of calculus this belongs to, so I can read up on it? Many thanks!


texts: What pandas function is most similar to the transpose procedure in SAS?

texts:
I am trying to transpose Amt by Txn_ID across Item_Desc in Python using pandas. Which pandas function(s) is/are most similar to the transpose procedure in SAS?
Below is an example of the input dataframe and the desired output dataframe.
Input DataFrame:
| Txn_ID | Item_Desc | Amt  |
|--------|-----------|------|
| 1      | Apples    | 0.96 |
| 1      | Milk      | 2.10 |
| 2      | Eggs      | 7.00 |
| 2      | Flour     | 2.20 |
| 2      | Salt      | 4.75 |
| 3      | Eggs      | 3.50 |

Output DataFrame:
| Txn_ID | Apples | Eggs | Flour | Milk | Salt |
|--------|--------|------|-------|------|------|
| 1      | 0.96   | NaN  | NaN   | 2.10 | NaN  |
| 2      | NaN    | 7.00 | 2.20  | NaN  | 4.75 |
| 3      | NaN    | 3.50 | NaN   | NaN  | NaN  |


texts:
I'm working on a CNN model that classifies images. After scraping image files from the Internet, I found that many of them didn't look this way as described by the searching keyword (for example keyword='dog' but image not containing a dog).
So, I cleaned my dataset manually which was quite labor intensive and time consuming.

Was my approach right or is there any tools or methods by which image data can be cleaned? Actually this sounds quite controversial because this cleaning tool should do the job for what the model are being built - classify the images.


texts: In clustering, sequence number such as customer ID and dates such as purchase date should be dropped?

texts:
I am learning K-means clustering and found that in most datasets, there are sequence number such as customer ID and dates such as purchase date.
I don't see any use in them for clustering.
Should I include them for clustering or can simply ignore them?
Let's say other attributes are like purchase amount and number of purchases and etc.


texts: Implication of a dominant Principal Component in PCA analysis

texts:
I need help, are there any practical implications of a dominant principal component. For example, if of three PCs, PC1 explains almost 100% of the variance in this dataset, What does this mean in practical terms? or what does it say about the dataset?

Any help is highly appreciated. Thanks!


texts:
In my university, we have an HPC computing cluster. I use the cluster to train classifiers and so on. So, usually, to send a job to the cluster, (e.g. python scikit-learn script), I need to write a Bash script that contains (among others) a command like qsub script.py.

However, I find this process very very frustrating. Usually what happens is that I write the python script on my laptop and then I login to the server and update the SVN repository, so I get the same python script there. Then I write that Bash script or edit it, so I can run the bash script.

As you see this is really frustrating since, for every little update for the python script, I need to do many steps to have it executed at the computing cluster. Of course the task gets even more complicated when I have to put the data on the server and use the datasets' path on the server.

I'm sure many people here are using computing clusters for their data science tasks. I just want to know how you guys manage sending the jobs to the clusters?


texts: How to split natural language script into segments?

texts:
I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the following categories:
MainConceptDescription-> Explanation of the Main concept(s)
SubConceptDescription-> Explanation of Subconcept related to the main concept
Methodology / Technique-> To achieve something, what should one do
Summary-> Summary of the discussed material or of the whole course
Application-> Practical advise for the concept
Example-> Concept example

Now, for the first 2 i think I should try to apply Latent Dirichlet Allocation to extract the topics. Another idea was to look in the resource name and search for these words in the text. Another idea was to read some of the resources and manually fix some sort of a dictionary for every category and then create regex patterns and search for them in the text.

But the latter seems too lame. So now I am not sure what can I do. I've seen similar works for research papers, however research papers have their own specific expressions etc that are more or less constant and seen in most of the papers, but that is not the case with my video scripts where it's 100% spoken natural language I need to work on. Do you have any ideas how can I approach this? I do have a list of keywords that kind of signifies whether an example follows or a concept is explained, but I am doing this manually which is surely not what I want to do for 563 files that may as well become many more.

Furthermore, I'd like to connect topics found to ontologies in order to enrich the metadata about each file. I have no idea how can I approach this either. Any advice would be VERY appreciated.

Forgive me if my explanations make no sense. I am not too familiar with the terminology. So if you also explain some of the terminology you use, i'd appreciate that too. And please advise on algorithms I can try with.


texts: Deriving new continuous variable out of logistic regression coefficients

texts:
I have a set of independent variables X and set of values of dependent variable Y. The task at hand is binary classification, i.e. predict whether debtor will default on his debt (1) or not (0). After filtering out statistically insignificant variables and variables that bring about multicollinearity I am left with following summary of logistic regression model:

Accuracy ~0.87
Confusion matrix [[1038 254]
                 [72 1182]]
Parameters Coefficients
intercept  -4.210
A          5.119
B          0.873
C          -1.414
D          3.757

Now, I convert these coefficients into new continuous variable "default_probability" via log odds_ratio, i.e.

import math
e = math.e
power = (-4.210*1) + (A*5.119) + (B*0.873) + (C*-1.414) + (D*3.757)
default_probability = (e**power)/(1+(e**power))enter code here

When I divide my original dataset into quartiles according to this new continuos variable "default_probability", then:

1st quartile contains 65% of defaulted debts (577 out of 884 incidents
2nd quartile contains 23% of defaulted debts (206 out of 884 incidents)
3rd quartile contains 9% of defaulted debts (77 out of 884 incidents)
4th quartile contains 3% of defaulted debts (24 out of 884 incidents)

At the same time:

overall quantity of debtors in 1st quartile - 1145
overall quantity of debtors in 1st quartile - 516
overall quantity of debtors in 1st quartile - 255
overall quantity of debtors in 1st quartile - 3043

I wanted to use "default probability" to surgically remove the most problematic credits by imposing the business-rule "no credit to the 1st quartile", but now I wonder whether it is "surgical" at all (by this rule I will lose (1145 - 577 = 568 "good" clients) and overall is it mathematically/logically correct to derive new continuous variables for the dataset out of the coefficients of logistic regression by the line of reasoning described above?


texts:
The text and output to be extracted are similar to the following :
"Check it every two weeks" - two weeks
"Check it on day 1 and day 14" - day 1 and day 14
"day 19 and day fourteen are important" - day 19, day fourteen
"The game is in 6 wks" - 6 weeks
"Check it in 6mo" - 6 months
"days 1 and d14 are important" - day 1, day 14
"Check it on days 11 and 14" - day 11, day 14
"check it on day one and twelve" - day one, day twelve
I have tried using SUTime library to extract the necessary information, but it just works in the case of first example and fails to properly extract information from the most of the other sentences. Considering the many ways in which the same text can be written, it is not very feasible to use ReGex.
I was wondering if there are any feasible NLP/ML solutions to extract this information. If NLP/ML is the best way to solve this, I have two questions: 1. I am not sure if I can use classification models and my first preference was to use a regression model, is it right to do that? 2. In either of the cases, I don't have a lot of labeled data to train. So, it'd be helpful if anyone can let me know if there's any open source labeled data for training+.


texts: How to get the best combinations of features for a sale optimization problem?

texts:
I have a database of shoes items from the same brand with many variables (features) like the size, the color or the shape. I also have the produced and sold quantity for the last years.

This is a sample of the database:

I would like to know what are the best combinations of features to have the perfect shoes, i.e the shoes which will be sold the most for sure. The Ratio (last column) is the variable which gives a "score" to an item (=an combination of features). For example, the best combination can be : construction=vulcanized + colour=brown but without width=welded.

I am using Python to read the database and using the library Pandas. This problem can be an optimization problem. What are the possibilities in data science to solve this kind of problem ?


texts: Why the VC dimension to this linear hypothesis equal to 3?

texts:
I am trying hard to understand this. Here is the scenario:

X = R^2
H = { h(x) = x + 10 }

I need to calculate the VC dimension for the above linear separator. Somehow, the VC dimension for this linear separator is 3. I just cannot understand how.

According to what I understand, VC dimension is the size of the largest finite subset of X which can be shattered by h. So if there exists a subset of size n where all points are shattered by h and there exists a subset of size n+1 where at least one point is not shattered by h, then the VC dimension will be n.

Is the VC dimension >= 1?

  Yes. We just need to pretend there is a single point on the line and by keeping the line (X-axis) steady we can flip which side is positive/negative

Is the VC dimension >= 2?

  Yes, because we could separate all 4 combinations { ++, --, +-, -+ } using a single line

Is the VC dimension >= 3?

  This should be NO, according to what I understand. How could we separate this case + - +?

But I was going through a video, that explains VC dimensions which says it is three and I do not understand how.


texts: SVM is not fitted when tried to fit it into a model

texts:
I was trying to apply SVM into a dataset to find its accuracy. I also applied bagging and boosting on the same dataset and they worked properly. But when I tried to fit SVM into a model its not working. I couldn't understand whats the problem.

I'm providing the sample of my dataset

train_new.head()

# define X and y
feature_cols = ['age','workclass','fnlwgt','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']

# X is a matrix, hence we use [] to access the features we want in feature_cols
X = train_new[feature_cols]

# y is a vector, hence we use dot to access 'label'
y = train_new['label']

from sklearn import svm
model = svm.SVC(kernel='linear', C=1, gamma=1)

and while trying to fit with model it's almost minutes passed and it's just look like,

What's the actual problem and could I solve it ?


texts:
I am trying to build a simple LSTM based model but I am getting "can't set attribute error" on the line to add LSTM layer to the model. I am unable to figure the reason as to why this error is appearing. This is the code I am using.

left = Sequential()
left.add(LSTM(64,activation='sigmoid',batch_input_shape=(10,look_back,dim)))
left.add(Dense(dim, activation='linear'))
left.compile(loss='mean_squared_error', optimizer='rmsprop')

I do not think there is an issue with code. The version of Keras is 2.0.7 with tensorflow backend. It is difficult to trace the cause of error.


texts: Is my model predicting new images wrong?

texts:
I used a CNN network unet for a segmentation task.
This is the architecture I used:

inputs = Input((IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS))
s = Lambda(lambda x: x / 255) (inputs)

c1 = Conv2D(16, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (s)
c1 = Dropout(0.1) (c1)
c1 = Conv2D(16, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c1)
p1 = MaxPooling2D((2, 2)) (c1)

c2 = Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (p1)
c2 = Dropout(0.1) (c2)
c2 = Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c2)
p2 = MaxPooling2D((2, 2)) (c2)

c3 = Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (p2)
c3 = Dropout(0.2) (c3)
c3 = Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c3)
p3 = MaxPooling2D((2, 2)) (c3)

c4 = Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (p3)
c4 = Dropout(0.2) (c4)
c4 = Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c4)
p4 = MaxPooling2D(pool_size=(2, 2)) (c4)

c5 = Conv2D(256, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (p4)
c5 = Dropout(0.3) (c5)
c5 = Conv2D(256, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c5)

u6 = Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same') (c5)
u6 = concatenate([u6, c4])
c6 = Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (u6)
c6 = Dropout(0.2) (c6)
c6 = Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c6)

u7 = Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same') (c6)
u7 = concatenate([u7, c3])
c7 = Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (u7)
c7 = Dropout(0.2) (c7)
c7 = Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c7)

u8 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c7)
u8 = concatenate([u8, c2])
c8 = Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (u8)
c8 = Dropout(0.1) (c8)
c8 = Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c8)

u9 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c8)
u9 = concatenate([u9, c1], axis=3)
c9 = Conv2D(16, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (u9)
c9 = Dropout(0.1) (c9)
c9 = Conv2D(16, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same') (c9)

outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)

model = Model(inputs=[inputs], outputs=[outputs])
model.compile(optimizer='SGD', loss='binary_crossentropy', metrics=['accuracy' ])
model.summary()

earlystopper = EarlyStopping(patience=5, verbose=1)
checkpointer = ModelCheckpoint('./accuracy.h5', verbose=1, save_best_only=True)

results = model.fit(x_train, y_train, shuffle=True, validation_split=0.1 , batch_size=16, epochs=50,callbacks=[earlystopper, checkpointer])

I used SGD optimizer and accuracy as the metric.
These are the learning graphs for loss and accuracy:

And this is the code for the prediction:
model = load_model('......../accuracy.h5')
PRED_PATH = '....../predict/' folder to save the predicted images
TEST_path = '...../testing/' folder for the test images
test_ids=os.listdir(TEST_PATH)
########## read the test images #############

preds_test = model.predict(X, verbose=1)
preds_test_thresholded=255*(preds_test > 0.5).astype(np.uint8)
ctr=0
for i in range(len(test_ids)):
   file_name=test_ids[i]+'_pred.png'
   img=preds_test_thresholded[ctr]
   imsave(PRED_PATH+file_name,img)
   ctr= ctr+1

During the training, accuracy = 0.977.
After prediction and to evaluate my model performance, I calculated the Dice_coefficient for all the images. I got 0. It seems like everything is working! Could anyone explain to me why got but results?


texts: Why isn't dimension sampling used with gradient boosting machines (GBM)?

texts:
GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used with dimension sampling at every split of the tree like is common practice with random forests.

Are there some tests that show that dimensional sampling with GBM would decrease its accuracy because of which this is avoided, either in literature form or in practical experience?


texts: For logistic regression, Predict.glm() outputs $p$ or $ln(p/1-p)$?

texts:
I'm performing a logistic regression in R.
I wanted to know if the function predict.glm outputs $p$ (probability of event occurring) or log odds i.e. $log(p/1-p)$?


texts: How to create these two plots from two arrays with numbers?

texts:
I have two numpy arrays, one with the data for x axis and another one with the data for the y axis. The position i in each array matches the other array.
Some values in the x axis array are repeated. I would like each of them to appear in a separate area, as in the following examples (note that 5 and 10 appear twice in the x axis):

Here is the code I tried:
dataFrame = pd.DataFrame({'xx': x_data, 'yy': y_data})
plt.plot('xx', 'yy', data=dataFrame, marker='o')

But this code is creating a chart where there is more than one y axis value for each x axis value (possible due to the repetitions).
The dataFrame is like this:
     xx   yy
     5    1
     10   2
     15   1
     0    0
     5    3
     20   4
      4   3
     10   4

Thanks in advance.


texts: How can I evaluate the constant parameters of a series of linear functions with varying input parameters?

texts:
I have the following setup of linear functions:

g0(x0, y0) -> (x1,y1)
g1(x1, y1) -> (x2, y2)
... about 30 more cases

Where gi is a function parameterized by 6 values (ai-fi). Which can be described by the following pseudo code:

function g:
  Parameters = [a, b, c, d, e, f]
  Variables = (x, y)

  return (
    x * a + y * b + c,
    x * d + y * e + f
  )

For each of the 100k runs I have a random, unknown starting condition (x0, y0), but similar parameters (ai - fi). As observable I have all the x parameters except the initial condition, eg xi>0.

I'm lost as to how to tackle that problem due to the unknown, variable incoming condition of any run. Without it I would simply try to tackle the problem by linear regression. I would appreciate any help pointing me in the right direction.


texts: How to backup a Caffe (DL framework) project? The Caffe folder (downloaded from GitHub) can't be copied or compressed

texts:
I am using Ubuntu18.04 and Caffe deep learning framework to do my project. However, when I tried to backup my project, I found that the Caffe folder (downloaded from GitHub) can't be copied or compressed.

Here is what I did:

Download Caffe from GitHub, and install it.
Do some modification to the source code for my project.
Backup the whole Caffe folder.
However, if I copy the folder to a USB flash drive, I will have a
folder that misses files (I compared the size and number of files of two folders). If I compress the folder then copy it to the USB flash drive, then I get error says I can't open the compressed file.

I wonder if I did anything wrong, what is the correct way of backup Caffe project?

Thanks.


texts: I need to find Simplest exercises available for learning neural networks

texts:
I'm learning neural networks, and I'd like to give myself simple exercises and try to train neural networks to solve them.

I know that classifying hand-written digits is a classic problem, but I'm hoping for something much simpler than that. No image recognition, no thousand of input neurons.

One exercise I thought of, is to get a number and find its modulo when dividing by 7. But it was explained to me that this is actually a difficult task for a neural network beginner for various reasons.

Does anyone have ideas for a very simple exercise/training idea? Or a link to an existing list of exercises?


texts: Multi-Class classification with CNN using keras - trained model predicts object even in a fully white picture

texts:
I built an multi classification in CNN using keras with Tensorflow in the backend. It nicely predicts cats and dogs. However, when it comes to an image which does not have any object-white background image-, it still finds a dog ( lets say probability for dog class 0.75…, cats 0.24… ). I am a quite newbie learner in learning built with neural network.

Sorry if I am asking a silly question, even though I have searched the internet I could not find any answer.

What is my exception from the case of the white background image as an input to prediction method, is 0 probability for dog and cat classes.

Any suggestion would make me so happy.

The below is how I implemented the training.

classifier = Sequential()

classifier.add(Conv2D(32, 3, 3, input_shape=(64, 64, 3), activation='relu'))
classifier.add(MaxPool2D(pool_size=(2, 2)))

classifier.add(Conv2D(32, 3, 3, activation='relu'))
classifier.add(MaxPool2D(pool_size=(2, 2)))

classifier.add(Flatten())

classifier.add(Dense(units=128, activation='relu'))

classifier.add(Dense(units=2, activation='softmax'))

# Metrics will be categorical_accuracy
classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)
training_set = train_datagen.flow_from_directory(
    '/Users/ozercevikaslan/Desktop/Convolutional_Neural_Networks/dataset/training_set',
    target_size=(64, 64),
    batch_size=32,
    class_mode='categorical')

test_set = test_datagen.flow_from_directory(
    '/Users/ozercevikaslan/Desktop/Convolutional_Neural_Networks/dataset/test_set',
    target_size=(64, 64),
    batch_size=32,
    class_mode='categorical')

classifier.fit_generator(
    training_set,
    steps_per_epoch=8000,
    epochs=25,
    validation_data=test_set,
    validation_steps=2000)


texts: DQN - how is it possible to train separate outputs for each action?

texts:
I'm trying to implement a Deep Q Network, but I'm stuck on how you train a network to predict multiple action-values when you can only collect data on a single action.
In the paper it recommends using a different output for each action

We instead use an
architecture in which there is a separate output unit for each possible action, and
only the state representation is an input to the neural network. The outputs correspond
to the predicted Q-values of the individual actions for the input state.

Since we can only visit one action, we only know the loss for that action (ie. a single output). But as far as I'm aware, we need to have values for all of the outputs in order to train the network. What black magic can you use to get the other output values?
It seems like a bad idea to get the network to predict the other action-values and feed them back, as it would affect the optimizer. And if you tried to ignore the other outputs and train it as if there were only the one you were currently focused on, you would still affecting the others as they would share edges.

DQN Paper


texts: R - Bayesian network for satisfaction survey data

texts:
I'm trying to build a bayesian network for satisfcation survey data.
My data is made of 13 questions about services, products etc... each customer can answer from 1 (Very unsatisfied) to 4 (very satisfied) with no "neutral feeling"). There is an other question about the overall satisfaction where the customer can answer 1 (very unsatisfied), 2 (unsatisfied), 3 (satisfied), 4(very satisfied) and 5 (totally satisfied).
I plan to use bnlearn to build the bayesian network. The goal is to identify important features who lead customers to give 5 on overall satisfaction item.
In your opinion, what is the best way to recode the overall satisfaction : 0 for 1, 2, 3, 4 and 1 for 5 (totally satisfied)? I tried but results let me skeptical.

Thanks for your help.

Best regards,


texts: What are some good techniques to decrease the size of Image Embeddings returned by CNN model?

texts:
I want to extract features from pre trained ResNet model for over 2M data. Problem? Even with the average pooling applied on the last layer's result, it provides a feature vector of length 2048 which looks something like [0.2,0.4,0.5,0.01,0.003,0.09....], if I convert to float32 for  200K images, I need 1.5GB of memory. But I have a bigger problem at hand now with a very huge dataset. I need to have a good algorithm which can decrease the dimensionality so that I can use any of ANNOY, LSH, SCANN, HNSWLIB etc for recommendation.
there are a few techniques that know but not sure about which one do good in case of Image Embeddings. Can someone suggest me which one would be good for my use case?
Some of the algorithm defined are in this and this blog but none them tells which one is better for embeddings.


texts: do autoencoders work well for non images?

texts:
I have a classification problem for which a feedforward, fully connected neural net works reasonably well (two classes, true positive and true negative rate close to 80%).

I want to get these rates to 90%, and more features is one of the catalysts for improvements I can think of.

Do autoencoders to learn additional, interesting features work well for problems that do not involve images?


texts: I am unable to understand what is weighted quantile sketch in xgboost. Can anyone help me give an intuitive understanding of this?


texts: What would cause regression to focus on a few values?

texts:
Hi I have a neural network for a regression problem but the results seem to be heavily focused on a few values. I added an image, y axis is predicted values and x axis is real values. What could be causing my problem?
Edit: Shapes of my train set is (5192, 383) and I used 4 dense layers with 512,256,128,64 units with dropout layers in between. I used %15 of the set for validation. I used the epoch with the best validation score. Trained the network for 300 epochs but it does not seem to get better with more.


texts:
I have a list of 50k sentences  such as : 'bone is making noise', 'nose is leaking' ,'eyelid is down' etc..

I'm trying to use Doc2Vec to find the most similar sentence from the 50k given a new sentence.

tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i)]) for i, _d in enumerate(data)]

max_epochs = 100
vec_size = 20
alpha = 0.025

model = Doc2Vec(size=vec_size,
                alpha=alpha,
                min_alpha=0.025,
                min_count=1,
                dm =0)

model.build_vocab(tagged_data)

for epoch in range(max_epochs):
    print('iteration {0}'.format(epoch))
    model.train(tagged_data,
                total_examples=model.corpus_count,
                epochs=model.iter)
    # decrease the learning rate
    model.alpha -= 0.0002
    # fix the learning rate, no decay
    model.min_alpha = model.alpha

test_data = word_tokenize("The nose is leaking blood after head injury".lower())
v1 = model.infer_vector(test_data)
#print("V1_infer", v1)

similar_doc = model.docvecs.most_similar(positive=[model.infer_vector(test_data)],topn=3)

for i in range(0,len(similar_doc)):
    print(tagged_data[int(similar_doc[i][0])],similar_doc[i][1])

Such that for the sentence "The nose is leaking blood after head injury" i would like to get the sentence with the highest similarity score ( i guess that it will bring sentences with words like leak or even synonyms like dripping?) . But the sentence i get back are unrelated and change each iteration of model.infer_vector(test_data)

Any idea about what is wrong?


texts: Pandas - how can I calculate a field to count the number of occurences that values in the ID field appear in sequential order

texts:
I know how to read in the data frame in Pandas and do the basic manipulation, but how do I populate the order column based on the ID column? For instance, if bike occurs twice such as seen below, how can I populate the order column with 1 for the first occurrence and then 2 for the second occurrence and then do the same for car.

    ID      Color       Order?????
    bike    red         1
    bike    black       2
    car     green       1
    car     orange      2
    car     blue        3


texts: Image clustering by similarity measurement (CW-SSIM)

texts:
I'm trying to use scikit-learn and pyssim for clustering a set of images - less than 100.

The end goal is to place the images into several buckets (clusters) according to the calculated similarity measures - CW-SSIM.

The task seems to be trivial, but I can't figure out the best way to handle "similarity based" clustering in scikit-learn. K-Means clustering looks like a good choice, but it doesn't accept any "comparison functions" or custom distance functions.

So how to handle the comparison based (similarity based) clustering in scikit-learn?

I was thinking about "comparison matrices" with 1 (similar) or 0 (not similar) per cell according to the calculated CW-SSIM similarity values. This matrix will be used for fitting into K-Means clustering. But then we will face the scalability issue, because such matrix will have dimensions equal to the amount of images ... which might grow to 1+ million in the future.

If there is an easier option in R than in Python, then I'm ready to review as well.

Thanks in advance.

UPDATE from Jan 18, 2016

I've created some code on GitHub about this topic: https://github.com/llvll/imgcluster

This project also includes IP[y] Notebook with step-by-step instructions and extra comments: https://github.com/llvll/imgcluster/blob/master/ip%5By%5D/imgcluster.ipynb


texts: Network either overfits or underfits, but never generalizes - what to do?

texts:
I have a simple network with 1st level an LSTM, dropout, fully-connected and softmax layers; loss is cross-entropy (four classes, well balanced). Sequence length to LSTM is 172 samples, data is z-score normalized, LSTM outputs at the end of a sequence.

Sequences are EEG recordings from 17 sensors (channels). Training set has 2146 sequences, validation set has 271 sequences. Minibatch size is 48.

Depending on LSTM layer sizing (number of cells), the netowrk either overfits or underfits, but never generalizes. When it overfits, after initial warm-up, as the training loss starts to decrease, validation loss starts to go up.

I'm using drouput with 0.5 probability, L2 regularization with 0.0005 weight and Adam solver.

What could be the reason and how to solve this?


texts: Automatically find an optimal threshold for multiple datasets

texts:
I am running an unsupervised model on data with no labels to find outliers, and have 100 datasets in total. The way I do this is, first I run it using a default threshold on each dataset, and second, I change the threshold until I get the desired output (desired- find a maximum of 10 outliers). I want to make this process automatic where the threshold/contamination will be tuned automatically in each dataset, and in the end, I will have 100 datasets with their outlier predictions.
An example for one dataset
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import IsolationForest

rng = np.random.RandomState(42)

# Generate train data
X = 0.3 * rng.randn(100, 2)
X_train = np.r_[X + 2, X - 2]
# Generate some regular novel observations
X = 0.3 * rng.randn(20, 2)
X_test = np.r_[X + 2, X - 2]
# Generate some abnormal novel observations
X_outliers = rng.uniform(low=-4, high=4, size=(20, 2))

# fit the model
clf = IsolationForest(max_samples=100, contamination=0.1, random_state=rng)
clf.fit(X_train)
y_pred_train = clf.predict(X_train)
y_pred_test = clf.predict(X_test)
y_pred_outliers = clf.predict(X_outliers)


texts: Which is best Model to implement Question Answering System

texts:
I Would like to know or get a clear picture of Question answering system. How to find which is the best model to implement. I don't need it to be any domain specific.

Moreover, I find all models looks same to me, it may be IR-based question answering system, Knowledge-Based Question answering system, Ontology-based system or few other systems.

Please guide me which one is best and what all approaches to be followed to implement those models.


texts: How to map RGB image segmentation ground truth to classes/one-hot vectors in TensorFlow?

texts:
Is there a TensorFlow function which takes multicategorical ground truth for semantic image segmentations in the form of RGB images and outputs a tensor with a one-hot vector encoding of the corresponding class per pixel? In other words, I'm starting with RGB images for the ground truth, so each class has a distinct color, like this:

This image was taken from the SYNTHIA dataset.Many semantic segmentation datasets supply their ground truth this way. Every pixel has just one class. What I am looking for is a function that first enumerates the number of different colours in an annotation, and then considers each colour to be a different class automatically. I thought that usually when performing semantic segmentation, each ground truth class is encoded using a one hot vector, to which predicted class probabilities can easily be compared.

I'm basically asking the exact same as this question, and am merely wondering whether such a function has been added since that question was posed, or if someone has a more efficient solution. The answer to it seems convoluted, and I can't imagine such functionality does not exist, as it would seem like a common task. Also, while for many datasets the ground truth is additionally given as some sort of text file (like JSON), writing parser for each different dataset you use seems needlessly cumbersome.


texts:
After a half year of working as a data scientist, I found myself tend to clone my previous work (in machine learning, like predictive modeling, EDA, data preprocessing etc.) in the new project and the work becomes a repeated style.

I need new directions to lift my work with some innovative pieces. That is a key incentive to my work and also make me feel fresh and confident.

Could anyone put forth advice for me? Cheers


texts: Active learning with mixture model cluster assignments - am I injecting bias here?

texts:
Suppose I have a dataset of people's phone numbers and heights, and I'm interested in learning the parameters $p_{girl}$, $p_{boy}=1-p_{girl}$, $\mu_{boy}$, $\mu_{girl}$, and overall $\sigma$ governing the distribution of peoples' heights. I don't have labels for boys or girls yet, but if I really want to, I can call the phone number and ask if the person is a boy or girl.
Procedure:

Fit a Gaussian mixture model to heights via EM. Assign the greater of the $\mu$s to be $\mu_{boy}$.
Pick a datapoint according to some scheme (more on this in a second) and call the associated phone number to get a boy/girl label
Re-fit a Gaussian mixture model to heights, but in the E step, restrict the membership probabilities to 1 or 0, according to the labels received. Somehow enforce the constraint that $\mu_{girl} <= \mu_{boy}$ (this may not be trivial).
Repeat step 2 until tired of calling people

This seems to make sense to me, but only at an intuitive level. I'm worried that if my label querying scheme is not random, I'll be biasing my results.

If I pick the point with the greatest cross-entropy, I'll call a bunch of people who have height 5'4.5", and probably get a 50/50 breakdown of girl/boy. After a couple of points, I think this could end up with two clusters sitting on top of each other.
if my label querying scheme is to pick the points furthest from the decision boundary, it should lead to a nice separation, and it would probably help enforce $\mu_{boy} > \mu_{girl}$ if that constraint in step 3 turns out to be hard to enforce. But after a couple of points, I'll be calling the least informative points. Also, perhaps my results will imply greater separation than there actually is.
Picking points randomly seems so inefficient.

I'm not confident whether the degenerate situations in steps 1 and 2 will actually unfold, but my undergrad-level instinct for why a nonrandom sampling scheme could inject bias is that this smells just like missing data imputation, where the data are not missing at random.
Am I in trouble if I query labels by some scheme, or even arbitrarily? Or am I stuck with calling people randomly?


texts:
I am applying a ML model (LGBM binary classifier) to data and would now like to identify the part of data where I have a low ratio of false-negatives (false-postives are not such a problem) and as much as possible true-negatives.

Background

The data I am classifying is from a system, that includes many complex rules that decide whether or not for some cases the company pays goodwill/warranty. In many of the cases this rule based system already finalizes the decision and initiates the payments. However there are cases, where this rule based system isn't able to finalize a decision but instead forwards the case along with a proposed decision to a manual decision process. It seems that this often also happens, because the rules (which are highly configurable) are not sharp enough or because the price-depending parts are not adjusted to inflation etc.

Now in this configuration it turns out, that the manual decision process very often just accepts the proposed decision. I am working on training a ML model (the LGBM classifier) to identify the cases, where the manual decision process leads to just accepting the proposed decision without changeing it.

Motivation

I am only interested in identifying cases where the proposed decision made by the rule system can just be taken over, because I'd like to reduce the amount of decisions that have to be checked manually.
My model currently gets a MAE of around 0.8 at the moment.

I'd now like to find a segment of the data, where the ratio of false negatives (the decisions where the model states, they need not change but in fact the descider changed them) is smallest (or below a certain threshold --> say 0.02).

Question

Can anybody suggest, how I could find such a region with a low ratio of false-negatives (false negatives in region by number of samples in the region)?
I thought about a simple decision tree, but would like to know if there are better options.

E.g. I suspect, that in case a leaf contains a high percentage of say 95% class 1 samples and 5% class 0 and some condition would allow to split it in two regions with 98% class 1 and 92% class 1 it wouldn't perform that split, since both would be classified as class 1 anyways, or am I wrong here?


texts: ROC AUC score is better if test data is imbalanced

texts:
I have an imbalanced dataset and I'm using XGBoost to do binary classification. I used down sampling together with target and one hot encoding for train data. For test data I once used just the encodings and left it unbalanced and once tried with a balanced test dataset.

The ROC AUC score was quite higher for the imbalanced test data than the balanced one. How is this possible? I thought for the ROC AUC score there should not be any difference?


texts: How to train effective domain specific word2vec models?

texts:
I'm training a word2vec model based on our custom data. Due to the size of data and the resources available to me, I'll use pyspark to train spark's built-in word2vec model.
However, I'm curious to hear what are the best practices around training more effective, useful models?
For example, I'm currently not converting sentences to lower case, since in our scenario, capitalized words can have specific meaning.
I'm also not removing punctuation. Is that something practitioners find useful? Should I remove stop words or use TF-IDF to remove common words?
Is there a rule of thumb for the number of dimensions for a given amount of data. Is there a heuristic around the amount of data (in terms of gigabytes) I should feed this model?
I will be feeding this model all scraped text from our public websites, internal chat session and a few other sources. The output will be used for finding trends in chats, topic modeling, various types of classification.
Look forward to hearing about other people's experiences. I'll also be happy to review any other resources: papers, blog posts, youtube videos, etc.
On this site, this question from 2 year ago has some useful info, but I'm hoping for more suggestions: Word2Vec: Using pre-trained models


texts: ML algorithm with fixed number of inputs and variable number of outputs

texts:
I am trying to solve the following problem:

Let's say I have a chess position:

I encode each square as one-hot encoded vector of length 13

index 0 for empty square
index 1 for white pawn
index 2 for black pawn
index 3 for white bishop
index 4 for black bishop
etc..

So, at the end I have a matrix ${M}$ with a shape 64x13, which I intend to use as an input of the model. I want to create a model able to map matrix ${M}$ to FEN representation of the position.

Given the example above, FEN representation of the position is: 1b1N4/7k/4b3/8/8/8/3P4/3K4

The length of a FEN-string is not fixed. I failed to find a ML algorithm which accepts fixed number of inputs and produces variable number of outputs. What machine learning algorithm can I use in that case ? Is RNN appropriate for that task ?

P.S. I do understand, that it is trivial to just write a function to map ${M}$ to FEN, but I do not want it, I want the model to learn it from data.


texts: Is there a way to draw custom tables like in SPSS where you put some attributes to rows and some to column to visualize clustering results?


texts: Time-Series analysis with small data set, but long sequences

texts: I'm working on a time-series classification problem. There are 3 classes. Dataset consists of 6 sequences from each class (total 18 sequences). Each sequence is 19,000 in length. What are some time-series classification approaches I can look into?


texts: Is this the way to obtain the same individuals for x_test and y_test?

texts:
x_train, x_test = train_test_split(x, test_size = 0.3,random_state=250)
y_train, y_test = train_test_split(y, test_size = 0.3,random_state=250)

Is this the way to coincide the same individuals in x_test and y_test as well as x_train and y_train?


texts: How max_features parameter works in DecisionTreeClassifier?

texts:
What is the parameter max_features in DecisionTreeClassifier responsible for?

I thought it defines the number of features the tree uses to generate its nodes. But in spite of the different values of this parameter (n = 1 and 2), my tree employs both features that I have. What changes so?

max_features = 2

max_features = 1

You can see x1 and x2 are used in both cases


texts: How is the cross-product transformation defined for binary features?

texts:
I am reading the paper on Wide & Deep learning and for the wide component, it states that one of the most important transformations is the cross-product transformation. This is defined as follows:

$$\phi_{k}(\mathbf{x})=\prod_{i=1}^{d} x_{i}^{c_{k i}} \quad c_{k i} \in\{0,1\}$$

Then, the authors argue that for binary features, a cross-product transformation (e.g., “AND(gender=female, language=en)”) is 1 if and only if the constituent features (“gender=female” and “language=en”) are all 1, and 0 otherwise.

But how does this relate to the given formula? And how can I think of the cross-product transformation in general?


texts: How to deal with Nominal categorical with label encoding?

texts:
So if my dataset looks like this:

  names life_style instrument  times
0   sid   creative      piano    1.5
1  aadi   artistic     guitar    1.4
2  aman  traveller       drum    1.1
3   sid   artistic     guitar    1.5
4  aadi   creative       drum    1.4

Now i want to deal with those Nominal categorical variables , Easy and go to approach is use Label encoding , But suppose if i am using sklearn label encoder then:

from sklearn.preprocessing import LabelEncoder
big_data = dataset_pd.apply(LabelEncoder().fit_transform)

which will output:

   names  life_style  instrument  times
0      2           1           2      2
1      0           0           1      1
2      1           2           0      0
3      2           0           1      2
4      0           1           0      1

Now it is converting each column but each column have same  numeric values range from 0 to 5. The instrument variable is now similar to 'names' variable since both will have similar data points, which is certainly not a right approach. I have few questions :

How should i treat those values without loosing information ? better
approach for this type of data points ?
I am thinking to use random forest for this type of data , Any
suggestion on model also will be helpful for me.
If some variable are appearing only once in huge dataset should i
remove those variables ?

Thank you in advance !


texts: Check Accuracy of Model Provided by Consultant

texts:
My company has recently engaged a consultant firm to develop a predictive model to detect defective works.

I understand that there are many ways to validate the model, for example, using k-fold cross-validation and I believe that the consultant firm will carry out the validation before submitting the model to us.

However, at the employer's side, how can I check the accuracy of the model developed by the consultant firm ??

Someone suggested that I can give 2000-2015 data to the consultant firm and keep 2016 data for our own checking. However, a model with good accuracy on 2016 data does not imply that it will have good predictive power in the future. In my view, keeping 2016 data for checking seems like adding one more test set for validation, which in my view, is unnecessary since I already hv "k-fold" cross validation.

Could someone advise what the employer can do to check the consultant's model?


texts: What is the meaning of colors in activation maps?

texts:
I am trying to understand how the neural network works. I plotted the intermediate layer activations and I got the following image.

I used the matplotlib library and selected cmap='jet'. I would like to know what is the meaning of these colors? Does the area with color red mean the filter is activated in that region? Then, what about the green, yellow and blue color means?


texts: corpus development for plagiarism detection

texts:
There are many simple plagiarism detection algorithms that work on search engines like google etc. I want to have a index of corpus of the whole internet to serve as a back-end database for my plagiarism detection software.
What should be the approach to build such database? Are there any opensource or collaborated live repositories?

somewhere i read instead of having local database of the entire internet, one can index and use it for faster search.

I know Elastic Search seems to be usable. Anyone has tried before?


texts: Show distribution of users affected by outlier response times

texts:
My dataset is performance metrics (response time) of a web page over the course of a single day.
The data looks roughly like this:
| Date | User Id | Response time in milliseconds |
|------+---------+-------------------------------|
| ...  | U1      |                           390 |
| ...  | U2      |                          1965 |
| ...  | U2      |                          7789 |
| ...  | U1      |                           479 |
| ...  | U1      |                          9876 |

Charting the percentiles 50th, 75th, 90th, 95th and 99th I could see that the response time for the vast majority of users is below 600 ms - which is seen as acceptable.
But there are extreme spikes in response times: I tried to sketch that in the data example above.
What I would like to do is visualize if it's always the same users affected by these spike or if the behaviour is more erratic than that. In the example above, user U2 would experience frequent spikes, whereas U1 has better performance with one spike (I know the sample size is obviously small in the example but I hope it helps to illustrate what I'm after.)
Goal of this effort is to reveal if it's always the same few users that are affected by spikes, then we could narrow down the problem.

An idea would be to use a bubble chart, which visualizes the following triplet for each user:

Count of how often we saw the user id during the day. (This will be the size of the bubble)
99th percentile for this user (x-axis)
75th percentile for this user (y-axis)

What I thought we could see from such kind of chart:

The bigger the bubble, the more frequent we saw the user.
The closer to 0 on both axes, the better the overall performance.
The further away from 0 on the x-axis, the more extreme the performance peaks.
The further away from 0 on the y-axis, the worse the common case performance.

Assumption is, users that are shown as large bubbles near the upper right corner could be the ones to investigate.
I thought a while about this, but I can't really say I'm totally convinced this is what I'm after.
Does this approach make sense to find out if we have mostly the same users experiencing the worst performance?


texts: Why we prefer VIF if we can find multicollinearity from correlation matrix as well?

texts: Why we prefer VIF if we can find multicollinearity from the correlation matrix as well? What is the exact logic behind it?


texts: Is there an overview over recommender system architectures?

texts:
I want to learn more about the recommender system topic.
I am very interested in the usage of different database systems for this use case.
My problem is that I cannot find a good overview of different architectures of recommender systems, especially with the focus on the database part.
Can someone help me out with a good reference or some own thoughts?
Thanks a lot. As interesting as this topic is for me as hard it seems to get some concrete information about it.


texts: Neural net only works for small datasets

texts:
I have a neural network that attempts to solve some regression problem.

The network works fine when the dataset has a small number of training examples, lets say 20. It overfits of course, but the training loss decreases to 0 which is what I want right now.

However, when I take my entire dataset, which has 5000 training examples in it, the loss plateaus at around 375. It won't decrease to zero, no matter how many hidden layers and neurons and activations I add. I tried everything.

What does this mean? Why does this happen and how can I fix it? Is it plausible that it's simply not possible to train a model that fits the training data?


texts: Is it conscientious to use a threshold for a model output in order to play on the recall and precision?

texts:
I have just finished reading an article about the F1 score, recall and precision. Everything was clear except the fact that the author, in his example (see https://towardsdatascience.com/beyond-accuracy-precision-and-recall-3da06bea9f6c#a9f0), is using a threshold for his model output to play on the F1 score, recall and precision values.

So my question is: is it conscientious to use a threshold as the author did in his article?

In my opinion, this is like a kind of manual over-fitting, but maybe there is something I misunderstood...


texts: Syntax error but nothing appears to be wrong?

texts:
I'm trying to create a decision tree, and everything appeared to be going smoothly until I encountered this syntax error. I don't see what the issue is?

print ("start fitting the data")
import warnings
warnings.filterwarnings("ignore")

for cv in tqdm(range(3, 6)):
    create_grid = GridSearchCV(pipeline, param_grid=check_params, cv=cv)
    create_grid.fit(X_train.fillna(X_train.mean(), y_train.fillna(y_train.mean()))
    print("score for %d fold CV := %3.2f" %(cv, create_grid.score(X_test, y_test)))
    print ("!!!! best fit parameters from GridSearchCV !!!!")
    print (create_grid.best_params_)

print ("loop has exited")

jupyter throws the error:

  File "<ipython-input-38-6b9dca710e77>", line 8
    print("score for %d fold CV := %3.2f" %(cv, create_grid.score(X_test, y_test)))
        ^
SyntaxError: invalid syntax


texts: Can I classify set of documents using classifying method using limited number of concepts ?

texts:
I have  set of documents and I want classify them to true and false

My question is I have to take the whole words in the documents then I classify them depend on the similarity words in these documents or I can take only some words that I interested in then I compare it with the documents. Which one is more efficient in classify document and can work with SVM.


texts: predict the probability of buying a product

texts:
I have a huge dataset with 3 variables Company_ID, Area_code, Product_ID each one of them is a categorical variable of levels 1500,50,15 respectively,where Product_ID is the product the Company_ID is using.Each company can be located in more than one area_code and can be using more than one product.Lets call this train data.

Given a combination of Company_ID, Area_code, Product_ID ,
example- comp_025, area_012, p_10 that is not present in train data,i have to predict the probability that a company Company_ID located in area Area_Code uses the product Product_ID. So that i can recommend that product to company.

I am a beginner and having difficulty in coming up with a model. Any input is highly appreciated.

Thank you.


texts: why transpose convolution is called "transpose"

texts:
https://d2l.ai/chapter_computer-vision/transposed-conv.html
https://en.wikipedia.org/wiki/Transpose
I understand what transpose convolution does, but I am confused about the name of 'transpose'.
In linear algebra, transpose is the action which flips a matrix over its diagonal;, which is absolutely what transpose convolution does in this case.
So why named as transpose convolution？


texts: How do I interpret the given classification report?

texts:
The given classification report was obtained from running a Random Forest binary classifier on the test data. There is huge class imbalance in the training data.
How do I interpret the given classification report showing very high values for one particular label?

            precision    recall  f1-score   support

      0       0.98      1.00      0.99     35050
      1       0.98      0.72      0.83      1982
  total       0.98      0.98      0.98     37032


texts: What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF?

texts: So I can generate X from arbitrary CDF F(x) by the procedure above. Can it be generalized to two variables? How, exactly? If not, what's the best way to generate (X,Y) from arbitrary CDF F(x,y) or PDF f(x,y)?


texts: macro average and weighted average meaning in classification_report

texts:
I use the "classification_report" from from sklearn.metrics import classification_report in order to evaluate the imbalanced binary classification

Classification Report :
              precision    recall  f1-score   support

           0       1.00      1.00      1.00     28432
           1       0.02      0.02      0.02        49

    accuracy                           1.00     28481
   macro avg       0.51      0.51      0.51     28481
weighted avg       1.00      1.00      1.00     28481

I do not understand clearly what is the meaning of macro avg and weighted average? and how we can clarify the best solution based on how close their amount to one!

I have read it:
macro average (averaging the unweighted mean per label), weighted average (averaging the support-weighted mean per label)

but I still have a problem in understanding how good is result based on how close these amount to 1? How I can explain it?


texts:
Recently in a data analytic job interview for an e-commerce site, they asked me, do i have some knowledge of buyer classification problem. Unfortunately i heard this term for the first time.
After interview i tried to search a lot about it over google but didn't find something meaningful. Please any one let me know if you have heard this term before and paste some links explaining this concept. Thanks


texts:
I have try to drop a nan value like so

import pandas
data = pandas.read_json("data.json")
lat = data["Latitude"]
lng = data["Longitude"]
shop = data["Store - Business Name"]
df2 = lat.dropna(axis=0, how='any')
df1 = pandas.to_numeric(df2, errors='coerce')

print(df1)

what am i missing, because i still get nan value in my panda series.


texts: Mirror column plot with different y-axis in ggplot

texts:
I am trying to convert this plot

Into a mirror column plot

I have two datasets:

school_purchases
recent_school_shootings

I am not very experienced with visualisation but this is my approach

ggplot(school_purchases_month, aes(x = date, y = total)) +
  geom_col(fill = "black", color = "white") #this plots one column chart of spending

I also want recent_school_shootings$killed on the other side

recent_school_shootings$killed=mutate(recent_school_shootings, killed= -killed)

ggplot(school_purchases_month, aes(x = date, y = total)) +
  geom_col(fill = "black", color = "white") +
  geom_col(data = subset(recent_school_shootings, killed > 0),
             aes(x = date, y = killed))

but it's throwing all kinds of error. Can you point me in right direction?


texts: How can you automate collecting curriculum vitae data?

texts:
I'm doing a machine learning project, for which I need data from thousands of curriculum vitae. For this, I need to collect data from the employees of some 50 specific companies. From each company, I require data from thousands of employees. This data simply consists of what positions they have previously held, and with which company; what qualifications they have (e.g. Computer Science BSc from University of Oxford); and what skills they have.

Initially I thought about using a webscraper on LinkedIn, but this is apparently very difficult for technical and legal reasons.

So what other viable solutions are there?


texts: How to interpret a regression model performances (Loss, accuracy) under keras

texts:
I built a regression model using Keras. The following parms were used:
model.compile(loss=custom_mse, optimizer=RMSprop(0.0001), metrics=mae)
history = model.fit(x=x_train, y=y_train, validation_data=(x_valid, y_valid), epochs=100, batch_size=16, verbose=2)

And here the funcs for the loss and accuracy:
import tensorflow.keras.backend as K

def custom_mse(y_true, y_pred):
    loss = K.square(y_pred - y_true)
    return K.sum(loss, axis=1)

def mae(y_true, y_pred):
    eval = K.abs(y_pred - y_true)
    return K.mean(eval, axis=-1)

I'm quite new with CNN and regression models. I couldn't interpret the loss and accuracy.
The performances' history is depicted in the following figures.

From the above plot, is the model overfitting or its doing alright? And what is all those spades mean?


texts: Are there databases specializing in scientific data

texts:
I am currently comparing file formats (HDF5 , etc.) to DBMS systems for a scientific data repository.

I know of proprietary solutions such as Oracle extensions, but are there open source/free systems for scientific data? I would define that as a system that would have the MKSA unit system integrated, and an extensive library of scientific conversions/operations.


texts: Would it make technical sense to model its two known subelements separately (CLV example)?

texts:
Let's assume Customer Lifetime Value (CLV) is defined as average basket x frequency.

Option A is to build model predicting CLV for each customer.

Option B is to model customer's basket size and frequency, each via separate models and multiply the results at the end.

Is there any mathematical reason why one would select B over A? E.g. would option B  reduce variance of the CLV because it is an ensemble of two models?


texts: Random Forest Classifier gives very high accuracy on test set - overfitting?

texts:
I have a financial dataset, where I'm trying to predict company types, based on the amount dollars, what time of day, and whether they buy or sell (currency pairs). It looks like this:

The features I use to predict:

X.head():

Dollars   | Hours      | Buy | Sell
-0.761916   0.364838     1     0
-0.924413   0.377558     1     0
-0.573336   0.397836     0     1
-0.561639   0.399144     0     1
-1.164036   0.423715     1     0

The features I want to predict could look like this:

y.head()
Bank  Tech  Fund  Holding  Defence  Financial Services  Pharma
1     0     0     0        0        0                   0
1     0     0     0        0        0                   0
1     0     0     0        0        0                   0
1     0     0     0        0        0                   0
1     0     0     0        0        0                   0

Agriculture  Commodities  Energi  Pension
0            0            0       0
0            0            0       0
0            0            0       0
0            0            0       0
0            0            0       0

In this snippet, the first five companies are banks.

Using a training/test ratio of 0.25, I get an accuracy of 0.99, which seems too good to be true:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

rand_forest = RandomForestClassifier(max_depth = None, random_state = 0)

rand_forest.fit(X,y)

predictions = rand_forest.predict(X_test)

The result of the classification_report:

         precision    recall  f1-score   support

      0       0.98      0.95      0.97      5074
      1       0.98      0.91      0.94      2292
      2       0.98      0.82      0.89       572
      3       0.99      0.83      0.90       235
      4       0.98      0.91      0.94       261
      5       0.99      0.81      0.89       411
      6       0.98      0.83      0.90       239
      7       1.00      0.70      0.82       144
      8       1.00      0.81      0.89       384
      9       0.99      0.81      0.89       200
     10       1.00      0.81      0.90       232

avg / total   0.98      0.90      0.94     10044

Adjusting the max_depth parameter of the classifier changes this number significantly though, but I'm still reading up on what the consequences of that parameter actually are.

It is worth mentioning that there is only 50,000 entries in this dataset, across 11 different companytypes, which might be too little?

Using a simpler DecisionTreeClassifier yiels an accuracy of about 50%.

UPDATE:

I used the entire dataset for training, not the actual training set. Switching these two outs gives an accuracy of 54%, which sounds much better (or more realistic anyways).


texts: Dealing with multiple distinct-value categorical variables

texts:
So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.

For instance, one column have more than one million unique value, it's an IP address column in case anyone is interested. Someone suggested to split it into multiple other columns using domain knowledge, so split it to Network Class type, Host type and so on. However wouldn't that make my dataset lose some information? What if I wanted to deal with IP addresses as is?

Nevertheless, the domain knowledge solution might work on the IP column, however, I've got other columns that have more than 100K distinct values, each value is a constant-length random string.

I did work with Embedding Layers before, I was dealing with max thousands of features, never worked with 10K++ features, so I'm not sure if that would work with millions.

Much Regards


texts: Q&A answer comparison multiple sentences using

texts:
I have been working on a Q&A app that has a template of questions and answers. The hope is to take answer text from the user, and compare it to the correct answer. I’d like to weight it on the keywords/buzzwords and general accuracy of the response.

I’m a mobile developer, not quite a data scientist or well versed in machine learning, so I’m a bit lost in the woods here and not sure if I’m going down the right path.

I’ve been looking into natural language process and some related python libraries/models that evaluate sentences.

Am I headed in the right direction? or should I go about this another way?


texts: SARIMAX with seasonality greater than one year

texts:
I would use Statsmodels SARIMAX with data having seasonality greater than one year.
In my case, I have seasonality equal 4 years, therefore I have tried:

smt.SARIMAX(endog, order=(p, d, q), seasonal_order=(P,D,Q,4))

Is it correct set seasonality = 4 when the seasonality is equal 4 years?


texts: Lasso regression not getting better without random features

texts:
First of all, I'm new to lasso regression, so sorry if this feels stupid.
I'm trying to build a regression model and wanted to use lasso regression for feature selection as I have quite a few features to start with.
I started by standardising all features and plotting the weights of each feature as I changed my regularisation strength to see which ones are most important. I also plotted the RMSE on holdout set to find a U-shaped plot, i.e. as I increased the regularisation strength, my RMSE would decrease, then after a certain point it start to increase. I couldn't. My RMSE plot was non-decreasing.
Then, I decided to throw in a random feature to see how the model would perform and compare my features vs. the random one. My expectation was again to see a U shape. However, that wasn't the case as I got another non-decreasing RMSE plot again which I pasted below. It also looked like two of my features, feat1 and feat3 were just as good as my random feature.

So here are my questions.

Why did could my RMSE have kept increasing as I increased my regularisation strength?
Does the fact that the coefficients of feat1 and feat3 were pushed to 0 at the same time as the random feature mean that they're not good and I should remove them from the model? (It felt strange that they hit 0 at the same time. I was expecting some difference.)


texts: What is the state of the art method for synonym detection?

texts:
In Natural Language Processing and Computational Linguistic what methods are deemed as SOA for similar word extraction .

Can anyone direct me to those resources?


texts: Choosing correct Period for SARIMA on a 4 month (120 day observation) dataset

texts:
I'm testing SARIMA on the first 4 months (observations are in days, total 120 days) of this dataset:
https://api.covidtracking.com/v1/states/ny/daily.cs
I'm not sure what value I should assign to the period since my range is not even a year.
I set a lag in my data for 11 days since COVID-19 is asymptomatic around that time period and impact seemed clearer using those amount of days.


texts: Software for automated database processing

texts:
I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this.

I have a dataset, for example: (brand-id, brand-name, product-class-name;)

0, Audi, economy business premium;
1, Rolls Royce, luxury;
2, Seat, economy;
3, Tesla, business premium;

And I'd like to automatically process this dataset, resulting in creating an additional table to classify parameters in column 3, like: (product-class-id, product-class-name, brand-id;)

0, economy, 0 2;
1, business, 0 3;
2, premium, 0 3;
3, luxury, 1;

Is there any database managing tool with features like this one? (preferably Mac)


texts: How to build a database of image data for machine learning?

texts: I want to build a database of image data for machine learning. But how should this be done? I'm assuming people don't just dump all of their image data into a folder? Do they use a relational database management system, like MySQL? Or do they use a NoSQL database, like MongoDB? Is there a textbook that explores this part of machine learning in particular? Is this what "data warehouse" refers to?


texts: How can I have the same initialization for all my networks?

texts:
I want to have the same weights for layer initializations in all my networks, so that when I'm comparing their first epoch loss they all start from the same value. Is there a way in keras to do this?

I have set the random seed for the numpy and tensorflow, but still I get different results in initializations.


texts: SAS Nested Likelihood Ratio Test for a Logistic Model

texts:
Using SAS Studio (online, student version)...
Need to do a "nested likelihood ratio test" for a logistic regression.
Entirety of instructions are: "Perform a nested likelihood ratio test comparing your full model (all predictors included)to a reduced model of interest."
The two models I have are:

Proc Logistic Data=Project_C;
Model Dem (event='1') = VEP TIF Income NonCit Unemployed Swing;
Run;

and

Proc Logistic Data=Project_C;
Model Dem (Event='1') = VEP TIF Income / clodds=Wald clparm=Wald expb rsquare;
Run;

I honestly have no idea where to even start.
Any suggestions would be appreciated.
Thanks!


texts: Probability Distribution of the Process Duration Time based on the Process Status Measurements

texts:
I have data of execution of several processes. Each execution is identified by the process_id.
The duration of the execution is measured by external agents, testing with some polling logic if the process is still executing or if it is finished.

Sample data for one process execution looks like follows:

   process_id  time status
        <dbl> <dbl> <chr>
 1       1001     1 execute
 2       1001     2 execute
 3       1001     3 finish

There are typically 0..n observation of the execute state, followed by only one observation of the finish state.
The time is measured relative in seconds from the execution start.

It is important to note, that there are several observation agents using different polling strategies. Some measure each second as in the example above, other only once per 10 seconds.
Some agent use such a high interval that in most case the first measurement delivers the status finished.
There are also agents with  aperiodical behaviour (i.e. with increased intervals).

The question is, is it possible with such a data to define the distribution of the probability (or possible the confidence interval of the probability distribution) of the process execution time?

I.e. I want to know the probability that the process will be completed within N seconds.


texts: Clustering evaluation metrics with subquadratic time complexity

texts:
It exists many evaluation metrics but often they are quadratic or more on number of data points preventing any application on massive data sets as RAND or Silhouette indexes.

For the moment i used :

External metrics

Normalized Mutual Information, NMI

Internal metrics

Davies Bouldin, which is in $O(n.c^2)$ with $c$ the number of clusters

I implemented both under Spark framework in Scala here for those who are interested but i would be happy to know if others exists, especially concerning internal metrics because even if Davies Bouldin could be a good evaluation metric, it is in my opinion, only with elliptic clusters.


texts: Deciding on sequence prediction approach

texts:
I have a space of 128 states expressed with binary bitstrings. The transition probabilities among these states are known to me, as follows:
(('1111111', '1111111'), 0.255),
(('0000000', '0000000'), 0.027),
(('1111111', '1111101'), 0.016),
(('1111111', '1111110'), 0.016),
(('1111011', '1111111'), 0.014),
...
(('0000000', '0111010'), 0.0),
(('0111010', '1000101'), 0.0),
(('1000101', '0000000'), 0.0)

The values at different bit positions can be interdependent, i.e., the value at position x of the bitstring expressing a state can influence the value at another position y in the bitstring for that state. However, I do not know how the values at different bit positions depend upon each other.
I want to take a sequence of states up to t steps as input, e.g., [0111111, 1111110, 1111110, 1111011, 1101110, 1111111, 1111101], and want to predict the state at (t+1)th step.
I have considered Viterbi algorithm on each bit position separately and Seq2Seq model on the integer values of the state bitstrings so far. However, I think they do not address the possibility of the different bit positions being inter-dependent and do not utilize the transition matrix that I have calculated.
The problem seems to be usual, but since being in social studies field I might not very familiar with any existing algorithm that addresses problems of this kind. Could you please suggest me a good algorithm or model to apply in the given scenario? Also if you could direct me to any available Python libraries that can help me use that would be highly appreciated.


texts: What preprocessing steps to be followed before image comparison?

texts:
1
down vote
favorite

For example I am trying to find the similarity between two images using skimage - SSIM. The code block will be as follows

from skimage.measure import compare_ssim as ssim
from skimage import io
from skimage.transform import resize

a = io.imread("http://ecx.images-amazon.com/images/I/51PV4Dd8wAL._AC_UL246_SR190,246_.jpg",as_grey=False,dtype="float64")
b = io.imread("http://ecx.images-amazon.com/images/I/914ZHE6JolL._UY500_.jpg",as_grey=False,dtype="float64")

a = resize(a,b.shape)
ssim(a, b,multichannel=True)

I got the similarity score as follows

0.21009350738786017

Although both the images are same where one images in slightly oriented , I am getting very really low scores for this comparisons. Am I missing any preprocessing steps here before I compute compare images. If so what are the things I should consider before comparing images.

Thanks In Advance !!


texts:
To use pre-trained models it is a preferred practice to normalize the input images with imagenet standards.
mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225].
How are these parameters derived?


texts: ValueError: Input 0 of layer conv1d is incompatible with the layer

texts:
tensorflow-gpu=1.15 and also tensorflow-gpu=2.2
Ubuntu 20.04.1 LTS
import tensorflow as tf

input_shape=(4,7,8,16)
x = tf.random.normal(input_shape)
y = tf.keras.layers.Conv1D(32, 3, activation='relu', input_shape=input_shape[2:])(x)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lucifer/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 894, in __call__
    self._maybe_build(inputs)
  File "/home/lucifer/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2127, in _maybe_build
    self.input_spec, inputs, self.name)
  File "/home/lucifer/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/input_spec.py", line 177, in assert_input_compatibility
    str(x.shape.as_list()))
ValueError: Input 0 of layer conv1d_4 is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [4, 7, 8, 32]


texts: How different should discriminator be from generator in GAN

texts:
When training a GAN, the generator $G$ strives to fool the discriminator $D$, while $D$ attempts to catch any output generated $G$ and isolate it from a real data point. They grow together training in turns for each epoche.

Assuming $D$ is already an expert classifier (for example, classifying birds and nonbirds images). What will happen if I freeze the weights of $D$ and only train $G$ (to generate high-resolution bird images from low-resolution ones for example)? Is there a mathematical problem here? Is $D$ so good that the generator will not be able to learn due to a very high initial error? I have obviously simulated it and failed.


texts: How to distinguish between normal fluctuation and outliers in ARIMA model?

texts:
I have a dataset about sales per day of certain products at the ITEM/DAY/STORE level , I've plotted the series and visually examined it for any outliers, volatility, or irregularities.

And this is what i got :

So there is huge spikes and drops in sales , so i used the  tsclean() function - tsclean() identifies and replaces outliers using series smoothing and decomposition-

And this is what i got after plotting :

But now , i'm wondering if what i did was right or not , in the sense maybe those fluctuations were more natural fluctuations than outliers , even though it seems quit weird to me to have a spike in sale of 4000 units in a day and than drops to 2000 the next day.

Any insight would be much appreciated , thank you .


texts: SVM SVC: Metric for parameter optimization on imbalanced data

texts:
I trained a multiclass SVC with RBF kernel on a down-sampled (and therefore balanced) dataset.
Now I want to perform grid search to find best cost and gamma.

What performance metric should I optimize for?

I have a highly imbalanced test set. There might be a factor of over 100 between the number of instances of different classes.
I am classifying 3D points (car, facade, human) - so I think one could assign equal weight to all classes.


texts: How can I get an objective measure of song similarity?

texts:
I was browsing through ML project ideas and found an interesting one (just the problem statement ) which was: detecting if two songs are similar using lyrics. I found it to be an interesting idea but Im not quite sure how I'd go about getting a score of similarity for the songs. For my dataset, I have features of genre, artist and lyrics. What is a potential method of scoring the similarity considering there is no such 'training data' to begin with.

I have come across word embeddings and stuff but their working isnt completely clear to me. Moreover, I think they dont take the song-like features that are available into account: things like type-token ratio, sentiment rating, word density(average number of words per sentence)etc.

Could an approach where first the songs were first clustered based on "high level features" such as type-token ratio, sentiment etc followed by a semantic similarity measure i.e. something like a cosine similarity metric on the word embeddings of songs in the same cluster make sense? How would I validate the usefulness of such an approach?


texts:
So I'm a recent transfer 2nd year student from Computer Science major to Mathematics major. Though I do have a bit of an issue here. I can choose between the applied mathematics, pure mathematics and statistics concentrations.

Along with this major, I'm doing a minor in Data Science with courses focused on Economics and Statistics.

In the future, I'm interested in doing a master's degree in Data Science and a career in data analysis for businesses.

Although I know any degree can be used to get into graduate school, I still want to which would be most beneficial for education in the future as well as opportunities for internships, research, and jobs.

Thank you!


texts: Does EDA helps only in case of linear regression?

texts:
I know what Explanatory data analysis is and how it helps us investigate and understand the data. What I dont understand is how does this help in case of nonlinear relationships? I mean if I'm using neural nets for nonlinear regression for example, how will a heatmap help me understand my data since it gives only linear correlation and that doesn't matter because the data can have some nonlinear relationship that can't be seen with heatmaps or some other plots.

I think if we are using linear regression, EDA ist very usefull to understand and know which features are better than others etc.. but how does it help for nonlinear models?


texts: why an advanced LSTM model produce the same results as a simpler one?

texts:
I have implemented the model proposed in this article which is a text classification model that uses sentence representation rather than only word representation to classify texts.

model=tf.keras.Sequential()
embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[word_embeding_matrix],trainable=False,mask_zero=False)
model.add(TimeDistributed(embeding_layer))
model.add(TimeDistributed(tf.keras.layers.LSTM(50)))
model.add(tf.keras.layers.Bidirectional(costumized_lstm.Costumized_LSTM(50)))
model.add(layers.Dense(6,activation='softmax'))
opt=tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=opt,loss='categorical_crossentropy',metrics=['accuracy',self.f1_m,self.precision_m, self.recall_m])
self.model=model

and I use a dataset with 40000 documents with 6 different labels to train it. (30000 for train and 10000 for the test). I uses a pretrained word embeding and the input for this model is (sample,sentences,words). it achieves 84% accuracy. the problem is that I can achieve this accuracy very easily with this simple model:

        model=tf.keras.Sequential()
    embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[word_embeding_matrix],trainable=False,mask_zero=False)
    model.add(embeding_layer)
    model.add(tf.keras.layers.Bidirectional(layers.LSTM(50)))
    model.add(layers.Dense(6,activation='softmax'))
    opt=tf.keras.optimizers.RMSprop(learning_rate=0.001)
    model.compile(optimizer=opt,loss='categorical_crossentropy',metrics=['accuracy',self.f1_m,self.precision_m, self.recall_m])
    self.model=model

this one is not based on sentence representation and the input for this model is (sample, words).
what is the first model ? is my implementation wrong? what should I do?

the training process for both models is as below picture. I also have used every trick to overcome overfitting but I haven't got any results. any suggestions please?


texts: SHAP Kernel explainer for my pipeline model

texts:
I am trying to use SHAP kernel explainer to understand my XGBOOST model. My data is the lending club data and I am trying to predict the Grade of each customer.

The data contains different types of data (Objects, float) so I use one hot encoding for object features and scaling for numerical features.

When I try to use the kernel explainer with the below code I get the following error.

    #creating training and validation sets
    X_train, X_test, y_train, y_test = train_test_split(features, eh, test_size=0.2,random_state=777)
    #%%
    preprocess = make_column_transformer(((make_pipeline(IterativeImputer(initial_strategy='median',add_indicator=True,verbose=2,max_iter=100),StandardScaler())),numerical_features),((make_pipeline(SimpleImputer(strategy='constant',fill_value="Not Available",add_indicator=True),OneHotEncoder(handle_unknown='ignore'))),categorical_features))
grid_clf = make_pipeline(preprocess,RandomOverSampler(random_state=777),XGBClassifier(n_jobs=-1,verbosity=3,
 objective= 'multi:softprob',
 nthread=-1,
 scale_pos_weight=1,
 seed=27,tree_method='hist',n_estimators = 125,
                            max_depth = 15,
                            learning_rate = 0.2,
                            gamma = 0.24,
                            min_child_weight =6.0,
                            subsample = 0.8200000000000001,
                            colsample_bytree = 0.96))

clf=grid_clf.fit(X_train, y_train)

import shap
shap.initjs()

def model_predict(data_asarray):
    data_asframe =  pd.DataFrame(data_asarray, columns=X_test.columns)
    return grid_clf.predict_proba(data_asframe)
#%%

explainer = shap.KernelExplainer(model_predict,X_test2).shap_values(X_test2.sample(1))

The Error I get:

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

How I can Solve this problem


texts: How do I separate Interleaved sentences using LSTM networks?

texts:
I have a problem statement wherein the following input

The James world Bond shall can't end save us tomorrow.

must be converted into this -

The world shall end tomorrow

&

James Bond can't save us

Essentially my inputs are a set of sentences interleaved with each other, and I need to separate them.

My training data contains slight variations of these interleaving sentences, I have been trying to work with a 1024 cell 2 Layered LSTM network, currently achieving about 65% accuracy when trying to predict the next word given the current set of words. I use the prediction from this model to chain words together into distinct sentences.

The questions that I have are

Is LSTM RNN the best way to achieve something like this?
What should be a good LSTM Cell size? (I have a vocab of about 22K words.)
Are 2 layers sufficient for modeling this problem statement?


texts:
I want to know the effect of Add and Multiply in keras by functionality. The dumb way of thinking is that they are meant to add and multiply keras tensors. I want to know when are they to be used. For example, look at the code below from here. Why use Multiply in spatial_squeeze_excite_block and why use Add in channel_spatial_squeeze_excite? Can we switch Add and Multiply in these functions? Why not?
def spatial_squeeze_excite_block(input):
    ''' Create a spatial squeeze-excite block
    Args:
        input: input tensor
    Returns: a keras tensor
    References
    -   [Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks](https://arxiv.org/abs/1803.02579)
    '''

    se = Conv2D(1, (1, 1), activation='sigmoid', use_bias=False,
                kernel_initializer='he_normal')(input)

    x = Multiply([input, se])
    return x

def channel_spatial_squeeze_excite(input, ratio=16):
    ''' Create a spatial squeeze-excite block
    Args:
        input: input tensor
        filters: number of output filters
    Returns: a keras tensor
    References
    -   [Squeeze and Excitation Networks](https://arxiv.org/abs/1709.01507)
    -   [Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks](https://arxiv.org/abs/1803.02579)
    '''

    cse = squeeze_excite_block(input, ratio)
    sse = spatial_squeeze_excite_block(input)

    x = Add([cse, sse])
    return x


texts:
I am trying to implement the A* algorithm into a graph I have, but I'm not sure if I am doing it properly.
So basically I have some nodes and some edges. In my case the edges are a cost, e.g. dollars defined by the edge type and the actual distance of the edge. So depending on the edge type and the distance of the edge between two nodes, a weight of an edge might look something like this:
node1 to node2 -> 100km * 10$_per_km + edge_insertion_cost -> 1500$

So the price pr. km might vary, as well as the insertion cost of using a particular edge. So far so good.
My "problem" is which heuristics is the best in this case ? I mean, if I just use Euclidian distance I will just get some distance in km, which has nothing to do with cost, node insertion and so on. And since it's a Euclidian distance, I can't put a price pr. km on - unless I should just use some kind of average of all different km prices.
So yeah, basically, what would be the best option here ? In some way I feel that it should at least be the same unit as the edge weight.


texts: How to label images for CNN use as classifier

texts: I have theorical question that I couldnt decide how to approach. I have tons of grayscaled shape pictures and my goal is seperate these images to good printed and bad printed. For this, I look at roughness of images because some of those is corrupted while printing (e.g I expected triangle shape but I saw some corruption in its edges. So its bad printed). Should I train my model to give each of them spesific label names (like good_printed_triangle, bad_printed_triangle, good_printed_square etc.) or just good and bad is ok? I was thinking about second option but how can CNN find correlation between different shapes, it makes me confuse. I am open for all ideas and thank you in advance.


texts: 3D projection - help understanding formula

texts:
I'm currently in the starting-phase of making a 3D projection in python. The problem I'm having is understanding the variables presented in the following article;
https://en.wikipedia.org/wiki/3D_projection

Can someone please explain what the variable "e" is? I can't figure out how you would represent the display surface, a plane, as a point...?
Thanks in advance.


texts: Which is the best algorithm for entity extraction for unstructured document

texts:
I have unstructured documents from which I have to extract the information like
let buyer name, seller name, expiry date, buying date etc. I had planned to use spacy(Custom entity recolonization(Followed this blog https://medium.com/@manivannan_data/how-to-train-ner-with-custom-training-data-using-spacy-188e0e508c6)). But it seems sometimes buyer name predict as seller name and vice-versa and also sometimes got multiple predicted data wrongly in single entity when I passed whole document content. FYI.. This documents have approx 2-20 pages. so it has large content.

Can someone share if we can use any other packages for higher accuracy? if not how I need to train the model so that accuracy will be higher? Thanks in advance


texts: NLP: What are some popular packages for phrase tokenization?

texts:
I'm trying to tokenize some sentences into phrases. For instance, given

  I think you're cute and I want to know more about you

The tokens can be something like

  I think you're cute

and

  I want to know more about you

Similarly, given input

  Today was great, but the weather could have been better.

Tokens:

  Today was great

and

  the weather could have been better

Can NLTK or similar packages achieve this?

Any advice appreciated.


texts: Understanding Object Detection Chart Showing Category Percentages "As Total Detections Increase"

texts:
I'm trying to understand the following picture, which comes from the "description of updates" pdf on this site for "Diagnosing Error in Object Detectors" by Derek Hoiem, Yodsawalai Chodpathumwan, and Qieyun Dai .
The chart shows the results detected by an object-detector for the category "dog". It displays the percentage of correct (Cor/white) object detection results along with the percentages of various false positives: localization error (Loc/blue), confusion of similar images (Sim/red), confusion with background (BG/purple), and other (Oth/green). The graph also shows a recall curve (the red lines) which differ in their Intersection over Union thresholds.

What I don't understand is the X axis. It shows how both the true/false positive percentages and the recall values change as the total detections increase, but what does that mean, for them to increase? In what context are they increasing? If it is merely iterating over the data, wouldn't the graph be different depending on the order in which you encountered the data? I have the vague idea that maybe this is an information retrieval concept?
For reference, here is the whole chart and caption. The chart compares the results from 3 object detectors (the 3 columns) across 3 object categories (the 3 rows):