texts: Compare ratings of players over different leagues

texts:
I want to compare ratings of players from different leagues and predict rating of player in a league he/she didnt participate in. Rating of a player is estimated within a league where he was playing.
There are some cross-observations which are players that have ratings estimated for more then one league.

For example there is a player P1 who has a rating 40 in league L1 and rating 55 in league L2. As you can see at the picture. There are distributions of rating in L1 and L2. Yellow line indicates the mean rating of a league and the green lines are cross-observations .

My question is if I know that player Px has a rating R in L1, what is his rating in L2?

Any ideas are appreciated!

Thank you


texts: Is autocorrelation of residuals a problem in machine learning?

texts:
Let's assume I have a random forest model and the residuals of the model are autocorrelated. Is this a problem?
As an example, let's assume I have two different random forest models, A and B, with a similar predictive performance. The residuals of model A are less autocorrelated than the residuals of model B. Should I prefer model A?


texts: Keras: X and Y are the same, yet validation accuracy is 50%, what is wrong?

texts:
I am trying to understand what is going on so I built a simpler version of my project. I set the X and the Y to be identical and I'm trying to predict Y using X, this should be very simple, but my setup isn't working. Here is my code :

import numpy
import keras
import pandas

# I want to evaluate the model when X and Y are the same
# This should be very easy for the model to evaluate
X = numpy.random.randint(2, size=100000)
Y = X

# Setup the model
model = keras.models.Sequential()

model.add(keras.layers.Dense(1, input_dim=1, init='uniform', activation='relu'   ))
model.add(keras.layers.Dense(1,              init='uniform', activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

hist = model.fit(X, Y, nb_epoch=10, validation_split=.3, batch_size=10, verbose=1)

df             = pandas.DataFrame()
df['loss']     = hist.history['loss']
df['acc']      = hist.history['acc']
df['val_loss'] = hist.history['val_loss']
df['val_acc']  = hist.history['val_acc']
df.index       = df.index + 1
#
print(df)

And this is my output:

        loss       acc  val_loss   val_acc
1   0.693162  0.504357  0.693300  0.496233
2   0.693150  0.503100  0.693250  0.496233
3   0.693157  0.502357  0.693132  0.503767
4   0.693171  0.502214  0.693119  0.503767
5   0.693167  0.502043  0.693121  0.503767
6   0.693129  0.504014  0.693133  0.503767
7   0.693167  0.503243  0.693129  0.503767
8   0.693157  0.502357  0.693181  0.496233
9   0.693180  0.502614  0.693141  0.503767
10  0.693170  0.502300  0.693119  0.503767

I expected that accuracy to go to 100%, but that is not the case. What am I doing wrong?

This is the example that I was following.


texts: Suppose I have 2 datawarehouses of two different companies that are being merged. Some of the tables are common but includes different records, while others are different. If I use big data environments, can I put all the tables together as data swamp and to create new data lake with archive of the previous datawarehouses data or should I build a new datawarehouse based on the common tables and adding the different ones? In addition, is there a good source for best practice steps for merging datawarehouses?


texts: Informed sender in Lazaridou et al. (2017) 's guessing game

texts:
I'm trying to implement the informed sender from Multi-Agent Cooperation and the Emergence of (Natural) Language (Lazaridou et al., 2017). However I'm confused about the tensor shape in the forward pass.

In the paper the network is described as following:

  The informed sender also first embeds the images into a “game-specific” space. It then applies
  1-D convolutions (“filters”) on the image embeddings by treating them as different channels. The
  informed sender uses convolutions with kernel size 2×1 applied dimension-by-dimension to the
  two image embeddings (in Figure 1, there are 4 such filters). This is followed by the sigmoid
  nonlinearity. The resulting feature maps are combined through another filter (kernel size f×1, where
  f is the number of filters on the image embeddings), to produce scores for the vocabulary symbols.

Figure 1:

So I assume the data should pass through the net as following:

The input is 2 feature vectors of shape (None, n_features)
They each get embedded via a Dense layer, output shape (None, n_game_features)
Here I'm not sure but I assume that in Keras, before passing them to the convolution layer, I need to stack those two vectors, forming (None, 2, n_game_features)
The vector passes through Conv1D(n_feature_filters, kernel_size=2, activation="sigmoid", data_format="channels_first"), outputting (None, n_feature_filters, n_game_features).
This passes through the vocabulary filter Conv1D(1, kernel_size=n_feature_filters, data_format="channels_first"), outputting (None, 1, n_game_features).

However, according to the paper, the output vector should be of length vocabulary_size which is independent on all the other parameters. Should I assume an additional Dense layer on the output, even though it was not mentioned in the paper? Or was there a mistake in my interpretation of the model specification?


texts: How to treat patients without events in time-to-event analysis?

texts:
I'm working with longitudinal data for a series of patients. Duration of followup on a patient-level is non-uniform.
Patients can either experience a discrete event (e.g., a heart attack) or never experience the event. This feature is of course binary. Additionally, patients that have experienced an event (e.g., the first heart attack) can also continue to experience more events (e.g., subsequent heart attacks). Each event is anchored to an event date which will be compared to when the patient was diagnosed with their primary, chronic condition (e.g., multiple myeloma).
I'd like to append to my dataset a derived column - TimeSinceLastEvent. Derivation of this value for first events would be calculated by (FirstEventDate - DiagnosisDate)/365 and subsequent events would be calculated by (SecondEventDate - FirstEventDate)/365, (ThirdEventDate - SecondEventDate)/365, etc.
How should I code this derived column for patients that never experience the event? I can't insert NA/NaN for these patients because downstream analyses require non-NA and finite data; so they would be imputed incorrectly anyway. One thought I had was setting these values to something drastically different, but standardized, such as -1 or 9999. Is this a valid and reasonable approach? If not, what have you used?


texts:
I am using xgboost for timeseries forecasting of a certain attribute while including seasonal features.Trained on nearly 4 years of data and tested on the last month. My rmse is as below :
Hyper params used are depth num rounds and learning parameter. Bringing the max depth down from 6 to say 2 keeps the test rmse constant but the train rmse instead of the below is increased to a number as close to the test RMSE ( eg maybe 26000 ).
Trying regulariziation and sub sampling didn't work.
Is this overfitting and what can I do to reduce the test rmse and generalize better ?
*[1611] validation_0-rmse:15886 validation_1-rmse:28546.4
[1612]  validation_0-rmse:15882.1   validation_1-rmse:28539.1
[1613]  validation_0-rmse:15879.9   validation_1-rmse:28531.4
[1614]  validation_0-rmse:15876 validation_1-rmse:28526.3
[1615]  validation_0-rmse:15872.9   validation_1-rmse:28522.9
[1616]  validation_0-rmse:15870.6   validation_1-rmse:28518.8
[1617]  validation_0-rmse:15866.9   validation_1-rmse:28511.8
[1618]  validation_0-rmse:15864.6   validation_1-rmse:28507.9
[1619]  validation_0-rmse:15862.2   validation_1-rmse:28505.7
[1620]  validation_0-rmse:15858.4   validation_1-rmse:28500.7
[1621]  validation_0-rmse:15855.3   validation_1-rmse:28497.4
[1622]  validation_0-rmse:15853.1   validation_1-rmse:28489.8
[1623]  validation_0-rmse:15850.8   validation_1-rmse:28485.9
[1624]  validation_0-rmse:15847.1   validation_1-rmse:28478.9
[1625]  validation_0-rmse:15844.9   validation_1-rmse:28474.5
[1626]  validation_0-rmse:15841.2   validation_1-rmse:28469.8
[1627]  validation_0-rmse:15839 validation_1-rmse:28477.7
[1628]  validation_0-rmse:15835.5   validation_1-rmse:28475.4
[1629]  validation_0-rmse:15833.3   validation_1-rmse:28472.2
[1630]  validation_0-rmse:15830.7   validation_1-rmse:28471.8
[1631]  validation_0-rmse:15828.4   validation_1-rmse:28469.6
[1632]  validation_0-rmse:15825.9   validation_1-rmse:28474.4
[1633]  validation_0-rmse:15822.2   validation_1-rmse:28469.7
[1634]  validation_0-rmse:15819.2   validation_1-rmse:28470.4
[1635]  validation_0-rmse:15816.3   validation_1-rmse:28471
[1636]  validation_0-rmse:15814.1   validation_1-rmse:28467.9
[1637]  validation_0-rmse:15810.5   validation_1-rmse:28463.7
[1638]  validation_0-rmse:15808.3   validation_1-rmse:28461.8
[1639]  validation_0-rmse:15805.4   validation_1-rmse:28468.5
[1640]  validation_0-rmse:15802.4   validation_1-rmse:28469.1
[1641]  validation_0-rmse:15799 validation_1-rmse:28466.2
[1642]  validation_0-rmse:15797 validation_1-rmse:28462.9
[1643]  validation_0-rmse:15794.9   validation_1-rmse:28459.8
[1644]  validation_0-rmse:15791.4   validation_1-rmse:28455.9
[1645]  validation_0-rmse:15788.5   validation_1-rmse:28456.7
[1646]  validation_0-rmse:15785.6   validation_1-rmse:28463.5
[1647]  validation_0-rmse:15782.3   validation_1-rmse:28464
[1648]  validation_0-rmse:15779 validation_1-rmse:28464.5
[1649]  validation_0-rmse:15775.5   validation_1-rmse:28461.7
[1650]  validation_0-rmse:15772.2   validation_1-rmse:28462.2
[1651]  validation_0-rmse:15768.9   validation_1-rmse:28462.7
[1652]  validation_0-rmse:15765.6   validation_1-rmse:28463.2
[1653]  validation_0-rmse:15762.4   validation_1-rmse:28463.7
[1654]  validation_0-rmse:15759.2   validation_1-rmse:28464.3
Stopping. Best iteration:
[1644]  validation_0-rmse:15791.4   validation_1-rmse:28455.9*


texts:
I have a panel data set and I have created a model and finally I have obtained some density forecasts.

That is, I run my model for the $y_{it}$ and i obtain predictions for $\hat{y_{i,t+1}}$ , $\hat{y_{i,t+2}}$ ,..., $\hat{y_{i,t+h}}$  etc.

The $\hat{y_{i,t+1}}$ , $\hat{y_{i,t+2}}$ etc, all of them follow a normal distribution, but with different mean and variance. so $\hat{y_{i,t+1}} \sim N(\mu_{1},\sigma_{1}^{2})$ , $\hat{y_{i,t+2}} \sim N(\mu_{2},\sigma_{2}^{2})$, ..., $\hat{y_{i,t+h}} \sim N(\mu_{h},\sigma_{h}^{2})$.

My question is, what is their joint distribution ? And how can I obtain it ?

Edit: The $y_{it}$'s are the daily consumption of a product of individual $i$ at time $t$, so yes, they are highly correlated.

Thank you for your help :)


texts: Combine features in Machine Learning KNN

texts:
I'm trying to build a simple book recommendation system, where I don't have any kind of ratings (no comments, no likes, no 1-5 stars, ...).

The information I can use is the following:

Book metadata  :  title, author, published date, model (printed or
ebook), genres (one or more).
Username.
Type : category of interaction made by a user, it can be 'order' (if the user ordered the book) or 'access' (if the user just checked the book).
Number : if 'type' = 'access' then this parameter indicates the number of times the user accessed the book.

So I have a DataFrame that looks something like this:

    number       type   username  product  price        model publishing_dt          genres
0        6     access   kerrigan     2345  12.99  printedbook    2020-02-01    fantasy,kids
1        4     access   kerrigan      897  14.95  printedbook    2019-03-05         fantasy
2        1  orderline  45michael    86833   2.65        ebook    2020-02-04      action,war
3        1  orderline   kerrigan    86833   2.65        ebook    2020-02-04      action,war

My goal is to develop a KNN system to make book recommendations and I can have two types of classifications:

Item-based : similarity between books (by genre and author).
User-based : similarity between users. Take each user purchases and compare them to other users to find similarities so if userA bought book1, book2 and book3 and userB bought book1, book2 and book4 you can assume that book3 is a good recommendation for userB.

But my doubt here is if I can develop a KNN system combining both technics, so depending on the data I input to it, it decides which solution is the better one.

Is this possible? Can someone point me in a good direction or send me some good tutorials?

EDIT - There are three scenarios:

The user is logged in and there's enough purchase history information.
The user is logged in but he/she is either new or there's not enough
purchase history information.
The user is not logged in.

Desired output:

A combination of purchase history and book features (this is the case I don't know how to approach), it should take into account both the book features and the past purchases, maybe it would be a good choice to perform two searches instead of trying to make one.
As there's not enough purchase history information, here the best
approach would be to just use book features to make a
recommendation.
Same case as before.


texts: What is the allowable limit of oversampling?

texts: Suppose I have 2 classes. One class has 16 samples and the other class has 435 samples. Is it justified to oversample the class with 16 sample to have a 435 number of samples? Or is it better to undersample the class with 435 samples? If so, what should be the number of samples after the undersampling is carried out?


texts: Where can I find an algorithm for human activity classification using thigh and shank sensors?

texts:
I am working on a project where I initially need to classify the type of activity a subject is performing based on raw accelerometer and gyroscope data from the thigh and shank of both legs. The type of activities would be day-to-day tasks such as standing, sitting, running, sit-to-stand, stair climbing, if playing sports, cutting, etc.

I have tried reading papers and looking on git to find an algorithm that will do this for me. I know that there are a lot of algorithms available, but none seem to utilize sensors on the leg. Most are sensors on the chest or through a person's phone.

Where would I be able to find an algorithm that will take the data I have an classify the activities being performed? Can I do this with algorithms that aren't initially made for the sensor placement I am using?

Any help would be greatly appreciated.


texts:
I am trying to cluster the sample of Imagenet Dataset using K-Means clustering.

In this approach, I have used the below 2 approaches to get the optimal number of clusters.

Elbow method   From the Graph it seems like the best number of clusters is from 6 to 10.
Silhouette score

Cluster : 2 | Silhouette score : 0.036273542791604996
Cluster : 3 | Silhouette score : -0.00300691369920969
Cluster : 4 | Silhouette score : 0.0025101888459175825
Cluster : 5 | Silhouette score : -0.005924953147768974
Cluster : 6 | Silhouette score : -0.00808520708233118
Cluster : 7 | Silhouette score : -0.006091121584177017
Cluster : 8 | Silhouette score : -0.00549863139167428
Cluster : 9 | Silhouette score : -0.014739749021828175
Cluster : 10 | Silhouette score : -0.021131910383701324
Cluster : 11 | Silhouette score : -0.04057755321264267
Cluster : 12 | Silhouette score : -0.012825582176446915
Cluster : 13 | Silhouette score : -0.012340431101620197
Cluster : 14 | Silhouette score : -0.032936643809080124
Cluster : 15 | Silhouette score : -0.04154697805643082
Cluster : 16 | Silhouette score : -0.04323640838265419

Where as in Silhouette analysis it looks like only for cluster 4 it is showing better value. Rest of the clusters are seems like samples are wrongly assigned to wrong clusters.

In such cases, which metric needs to be considered?

Updates:
I have reduced the dimension of the features using PCA. Below are the updated graphs for Elbow and Silhouette analysis. However I do not see any improvement in the clustering. As per Silhouette the samples are not being assigned to the closest cluster.

Elbow

Silhouette

Updating the question with latest graphs, VGGNet has been replaced by Resnet50

Code snippet:

        for i in range(lower_range, upper_range):
            # For Elbow curve
            kmeans_cluster = KMeans(n_clusters = i)
            kmeans_cluster_fit = kmeans_cluster.fit(features_list_np)
            loss = kmeans_cluster_fit.inertia_
            error.append(kmeans_cluster_fit.inertia_)

            # For Silhoutte analysis
            preds = kmeans_cluster.fit_predict(features_list_np)
            score = silhouette_score(features_list_np, preds)
            score_list.append(score)

            sample_score = silhouette_samples(features_list_np, preds)
            sample_score_list.append(sample_score)

            print("Cluster : {} | Loss : {} | Silhoutte Score : {}".format(i, loss, score))

            zero_samples = 0
            positive_samples = 0
            negative_samples = 0

            for each_sample in sample_score:
                if each_sample == 0:
                    zero_samples += 1
                if each_sample > 0:
                    positive_samples += 1
                if each_sample < 0:
                    negative_samples += 1

            print("Cluster : {} | Silhouette sample distribution - Zero : {} | Positive : {} | Negative : {}".format(i, zero_samples, positive_samples, negative_samples))

            zero_list.append(zero_samples)
            pos_list.append(positive_samples)
            neg_list.append(negative_samples)

Also should the importance should be given to Silhouette Score or the Silhouette Samples. Because in some of the cases, Silhouette score is less but the number of samples having positive values are more.

Thank you,
KK


texts: Intuitive explanation of Adversarial machine learning

texts: How would you explain Adversarial machine learning in simple layman terms for a non-STEM person? What are the main ideas behind Adversarial machine learning?


texts:
I understand that these are the gradients of the weights/biases in an RNN (correct me if I am wrong):

This is a lot to compute and I’m aware that these equations can be simplified for ease of use. My question is what are these simpler versions, and how are they derived?
I am using 6 timesteps in my network, hence the numbers used above.
Here are the formulas for forward prop, just for context:

One other thing. I just want to make sure I’m computing the loss correctly. I’m computing y-yhat at each time step, and appending it to an array, which at the end will have the same dimensions as x ([6, 4] - a very tiny dataset, I know.)


texts:
I have cloudera CDH5 running inside a virtual box.

when I try to run :

mahout spark-itemsimilarity ....

I get the error:

Unknown program 'spark-itemsimilarity' chosen.

Do i have to install any additional package to run the spark-similarity?

Any help would be appreciated !


texts: Are weights of a neural network reset between epochs?

texts: If an epoch is defined as the neural network training process after seeing the whole training data once. How is it that when starting the next epoch, the loss is almost always smaller than the first one? Does this mean that after an epoch the weights of the neural network are not reset? and each epoch is not a standalone training process?


texts: Neural network does not converge with negative symbols

texts:
I've created a simple 2-2-1 feedforward ANN to predict an XOR using Keras.

The activation function I'm using on all layers is a tanh, so in order to make use of the entire range of the function, i.e. [-1, 1], I've decided to use -1 instead of 0 as the symbol.

My input data is, thus, [[-1, -1], [-1, 1], [1, -1], [1, 1]], for an output of [[-1], [1], [1], [-1]].

I thought this would give better results since I'm using the entire range of the function and it supposedly would converge better because of that. Also, since I'm just using a different symbol, it should be the same as with using 0 and 1.

However, my network can not converge (giving a 0.5 accuracy), and what baffles me the most is that using 0 and 1 as symbols converges, and at a much faster rate.

Is there a reason for such a counter-intuitive (at least in my conception) thing to be happening?


texts: Classification Based Collaborative Filtering Model

texts:
I was going through algorithms for collaborative filtering-based prediction. Most of the places, I read about using matrix factorization based on ratings of the likeness of the user. But for my use case, I have to predict on the bases of classified data. For example, I have a list of user who orders from place1,place2, etc and they mostly get their items delivered to address1,address2 etc. Initially, I thought to give binary ratings to the places, but I am having multiple criteria i.e addresses and places. How can I use collaborative filtering in this case?

User
Place
Delivery Address

One
X
A

One
X
A

One
Y
C

Two
Y
C

Two
Z
B

Three
Y
??


texts: What is the contraction map constraint in the context of Graph Neural Nets?

texts:
In this paper https://arxiv.org/pdf/1511.05493.pdf (GATED GRAPH SEQUENCE NEURAL NETWORKS,2016), it is stated that in a Graph Neural network initialising hidden states is not required, as
'In GNNs, there is no point in initializing node representations because the contraction map constraint
ensures that the fixed point is independent of the initializations'
'This is no longer the case with
GG-NNs, which lets us incorporate node labels as additional inputs'
How exactly does the contraction map theory, which I am understanding to state that
given a metric space, and a mapping, that mapping is contractive if there exists a constant such
that the distance between points A and B after applying the mapping is always less than
the original distance between A and B times that constant.
Is this some kind of guarantee towards convergence? Is it valid to say that the constant c refers to the
'fixed point', and because c doesn't depend on the hidden state, the start point doesn't really matter.
Why does it no longer apply after we incorporate node labels?


texts: Custom regularisation for logistics regression

texts:
My understanding of l2 regularisation:
Weights of the model are assumed to have a prior guassian distribution centered around 0. Then MAP estimate over data adds an extra penalty in cost function.
My problem statement:
I am making a reasonable assumption(based on domain knowledge) that my features are independent which means I can use the weights of the features to infer the importance of features in influencing Y.
From domain knowledge, I want to assume priors about the ratio of weights.
Eg:
W1/W2 is a guassian distribution with mean 0.7
W2/W5 is a guassian distribution with mean 2.4
And then MAP estimate over data will give me a cost function, with extra penalty added.
Is my thought process correct? My intent is weights should be close to priors computed from some heuristic. This will also help me handle sparse data.
Implementation Details:
Is there any library(in any language) where it is easy to do this? Or do I have to compute the cost function and it's gradient myself and implement gradient descent over it?
Help with Math:
Also I have started learning stats(and maths in general) more rigorously, but I have lots of ground yet to cover. Assuming all Wi/Wj are given, can someone please give me the new cost function. I will write a program to solve for weights based on that cost function.
EDIT:
As I know Wi/Wj, can I assume  sum(Wi) = 100 and fit a bayesian logistics regression?
Also bayesian logistic regression fit method api requires a hessian matrix.
http://bayes-logistic.readthedocs.io/en/latest/usage.html?highlight=fit_bayes_logistic calls hessian matrix covariance matrix of fitted MAP parameters. Assuming that features are independent and variance of a feature to be 1, can I assume hessian matrix to be an identity matrix?


texts: Merging three different customer segmentation systems into one

texts:
I have been given a task where I have three existing customer segmentation systems (rule based e.g. if customer spends X in Y amount of time AND whatever then put in top spender segment is one segment, another is measured on metrics such as time spent on the site, etc and the last one is the lifetime loyalty segmentation system).
I have been asked to merge into one customer segmentation system.
I would like to ask your opinion or sources of information to decide whether I keep this rule based system (maybe a point based system) or use another method such as clustering. They want it to be flexible, thus be able to add additional layers as time goes by. They must be able to cater for different markets, which have different 'rules'

DO you believe I should keep the rule based system? Use clustering? What are the disadvantages of merging all three current customer segmentation systems?


texts: R Programming rearranging rows and colums from timeline data

texts:
I am new to R programming and data analytics. This is probably a trivial question but I could not find an answer. I'm also limited with my ontology of data science and data manipulation. So, I'm not sure if I conveyed my question right.

I have huge irregular time series data close to 1 Million records. They are all in the form of time stamps with different discrete attributes. I want to rearrange the data as shown below.

An Example of what I am trying to achieve FROM log entry of every car sold and the nature of the sale.

Date         Vehicle        Sale Type       Registration    Profit
1-1-2017    Toyota  Camry   Cash Sale          CA           2500
1-1-2017    Toyota Rav4     Lease              CA           8500
1-1-2017    Toyota Camry    Installment        AZ           250
2-1-2017    Toyota Corolla  Installment        NV           250
4-1-2017    Toyota Etios    Cash Sale          AZ           2500
4-1-2017    Toyota Camry    Lease              CA           8500
4-1-2017    Toyota Prius    Installment        CA           250
13-1-2017   Toyota Prius    Installment        VA           250

TO with rows having car names and columns containing cash sale, lease, installment

Vehicle Type    Cash Sale   Lease   Installment
Toyota Camry       1           1           1
Toyota Rav4                    1
Toyota Corolla                             1
Toyota Etios       1
Toyota Prius                               2


texts: extract document topic vectors from lda model

texts: how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation


texts:
I want to do some clustering for a dataset where I am looking at 10,000 peoples usage of certain electronic devices. I have 11 columns; the first column is simply a URN representing each person in the study. Then the other ten columns are for the weekly usage, in minutes, of each device (e.g. phone, laptop, t.v. etc).

No individual actually has all ten devices. They normally tend to have 2-4. For this reason they will have a value of n for each of the devices they use, and then a "0" if they do not have said device. Therefore, I obviously can't omit rows which have a 0 in them, and the 0 does not necessarily mean missing data, as opposed to actually meaning they do not own the device.

I am wondering how a typical basic cluster analysis would be carried out in a situation like this? / what would be an appropriate way to approach such a study? (I am using R for the analysis, for reference)

Thanks for the help.


texts:
I have trained a simple multi-input NN for classification. I have 4 inputs ( one text field & other 3 categorical variables : cat1, cat2 , cat3).
For the text field, I use Glove embeddings in the Embedding layer, followed by LSTM layer.
For the other 3 categorical fields, I just encode them in a dense layer. Finally, I concatenate these 2 layers followed by softmax for classification. Below is the main block of code :
embeddings_dictionary = dict()
glove_file = open('glove.6B.100d.txt', encoding="utf8")
for line in glove_file:
    records = line.split()
    word = records[0]
    vector_dimensions = asarray(records[1:], dtype='float32')
    embeddings_dictionary[word] = vector_dimensions
glove_file.close()
embedding_matrix = zeros((vocab_size, 100))
for word, index in tokenizer.word_index.items():
    embedding_vector = embeddings_dictionary.get(word)
    if embedding_vector is not None:
        embedding_matrix[index] = embedding_vector

# Text input handling
text_input_1 = Input(shape=(maxlen,))
embedding_layer = Embedding(vocab_size, 100, weights=[embedding_matrix], trainable=False)(text_input_1)
LSTM_Layer_1 = LSTM(128)(embedding_layer)

# categorical variables handling
layout_input_2 = Input(shape=(3,))
dense_layer_1 = Dense(10, activation='relu')(layout_input_2)
dense_layer_2 = Dense(10, activation='relu')(dense_layer_1)

# Concatenating the above 2
concat_layer = Concatenate()([LSTM_Layer_1, dense_layer_2])
dense_layer_3 = Dense(10, activation='relu')(concat_layer)
output = Dense(3, activation='softmax')(dense_layer_3)
model = Model(inputs=[text_input_1, layout_input_2], outputs=output)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])
print(model.summary())

The problem is , when I infer/predict using the above model weights, the respose is like all the weightage is being given to the categorical variables only , irrespective of the text input.
e.g. predict (text1, cat1, cat2, cat3) & predict (text2, cat1, cat2, cat3) is exactly same.
Even if I provide blank text as input, the output remains same if I don't change the categorical variable values.
Can anybody help troubleshooting this ?


texts: Cross validation for highly imbalanced data with undersampling

texts:
In my problem, I am dealing with a highly imbalanced data set, say for every positive class there are 10000 negative one. A normal starting method to train a model is to undersample the data. In this procedure, it is very important to train our model on the undersampled data and check the model evaluation on the holdout (from the original data -- without undersampling).

Now the quesion is here. KFold-cross validation actually splits the undersampled train set into K segments and take one of the folds as test set (which is now undersampled test set). I believe for model evaluation we actually need to calculate the metric of interest on non-undersampled test set (right? or I am misunderstaning sth here?). If yes, is it possible to perfrom cross validation as follows?

Split data into K segments.
Take the first Segment as test set, and undersample the rest of Folds (for example K=1 as test and K=2,3,4,5 as train sets)
Fit model on undersampled train data and calcualte the metric of interest on the test set.
Consider the other Fold as test set (this time e.g., K=2) and the rest as train set (K=1,3,4,5). Undersample train set and proceed to step 3.
Continue this procedure for the rest of the folds.

Is this a correct way of cross validation, when we undersample the data?
If yes, is possible to do it with standard libraries?

Edited on 21 of Feb:
Thanks to @Wes, I want to know if the following piece of code is the correct implementation of KFold cross validation on highly imbalanced dataset.

import numpy as np
from statistics import mean, stdev
from collections import Counter

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import KFold
from sklearn.metrics import roc_auc_score

from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import make_pipeline

# initial parameters
RANDOM_STATE = 42
RATIO = 0.033
N_SAMPLES = 1000000
K_FOLD = 5

# Generate the dataset
X, y = make_classification(n_classes=2, class_sep=1,
                           n_features=10, n_redundant=2,
                           weights=[0.9999, 0.0001], n_informative=5,
                           flip_y=0.0, n_samples=N_SAMPLES,
                           random_state=RANDOM_STATE)

print('Number of samples in each class %s' % Counter(y))

rus = RandomUnderSampler(random_state=RANDOM_STATE, ratio = RATIO)
rfc = RandomForestClassifier(random_state=RANDOM_STATE, n_estimators=100)
pipeline = make_pipeline(rus, rfc)

auc_roc = []
kf = KFold(n_splits=K_FOLD)
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    y_pred = pipeline.fit(X_train, y_train).predict_proba(X_test)[:,1]

    auc_roc.append(roc_auc_score(y_test, y_pred))

print('ROC_AUC = {} +/- {}'.format(np.round(mean(auc_roc),4),
                                   np.round(stdev(auc_roc),4)))

So that I have obtained the following result ROC_AUC = 0.9374 +/- 0.037.


texts: Flattening output before calculating metrics

texts: I use scikit-learn to calculate precision, recall and f1 scores which only accept 1D arrays, but my model's outputs are 2D (binary segmentation maps). My question is, is it ok to simply flatten the outputs, or is there some other function I should use to calculate the metrics in my case?


texts: What are the activation functions in Convolutional Layers for?

texts:
I read a lot about CNNs but I didn't quite understand some things:

What are the activation function in CLayers for? If I understood it right, the only weights in these layers are the ones in Filters, and for the activation function a weighted sum is needed?

The computational effort should increase, shouldn't it? When there are many Filters many Feature Maps (Map with the dot-products) are produced. All of them are given to the next Layer, so if they are as big as the input Image and there are 10 Filters in the first CLayer the second CLayer would have to apply 10x the computational effort(per Filter in the second Layer) that the first layer had or, since they all take all the last layers outputs (the feature maps).

If there are more Layers than one, how does the backpropagation know what they have to change on each one of them, especially since the backpropagation occurs after all layers are applied.

(CLayer = Convolutional Layer)


texts: Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?

texts:
I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it is possible to fine-tune BERT, ELMO or XLnet for Seq2seq encoder/decoder machine translation.

English: He sells food.

American sign language: Food he sells


texts: Training Validation Testing set split for facial expression dataset

texts:
I am using Convolutional Neural Networks (CNN) and I just want to ask if the way I split my training/validation/testing set is correct.

I have a total of 55 subjects. I plan to split them into 80–10–10 for training (45 subjects), validation (unseen 5 subjects), testing (unseen 6 subjects).

Should the validation set consists of unseen subjects as well? Or can I shuffle the whole training set and use a part of it (10-20%) as validation set?

I have read that using N-Fold cross-validation, the whole training set (instances) are shuffled then split into N-folds and the model is trained and averaged N times. However, in the case for Neural Networks or CNN, we don't use cross-validation since it is very computationally expensive.

I'm just wondering which is correct since using a validation set of unseen subjects, my model starts to overfit after 3-5 epochs and doesn't learn at all. On the other hand, if I use 10-20% of the training set as my validation set, my model learns with reasonable accuracy (45-50%) using a 3-layer CNN but when tested with the unseen testing set, my top-1 accuracy is around 15-16% only.

Thank you very much.


texts: BUG: Tensorflow accuracy on training data different when training than when evaluating

texts:
I think I have made some sort of error because my model has a substantially different accuracy while training:
Epoch 25/2000
19/19 [==============================] - 26s 1s/step - loss: 2.0174 - categorical_accuracy: 0.4253

than when it is evaluated:
>>> model3.evaluate(train_dataset)

19/19 [==============================] - 4s 188ms/step - loss: 2.3323 - categorical_accuracy: 0.2987

even though these two accuracy scores are being evaluated on the same data.
I think there might be something wrong with the batch normalisation layer? Like maybe it's not being applied properly outside of train time? If I look at a confusion matrix for the categorization problem on this data, it is predicting all examples as being from the most probable class.
I'm trying to create the FCN model from Z. Wang, W. Yan and T. Oates, "Time series classification from scratch with deep neural networks: A strong baseline," 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA, 2017, pp. 1578-1585. The model is here:
model3 = tf.keras.Sequential([
    tf.keras.layers.LayerNormalization(axis=1),
    tf.keras.layers.Reshape((-1, 1)),
    tf.keras.layers.Conv1D(filters=128, kernel_size=8, strides=1, padding='same', input_shape=(None, 1)),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.ReLU(),
    tf.keras.layers.Conv1D(filters=256, kernel_size=5, strides=1, padding='same'),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.ReLU(),
    tf.keras.layers.Conv1D(filters=128, kernel_size=3, strides=1, padding='same'),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.ReLU(),
    tf.keras.layers.GlobalAvgPool1D(),
    tf.keras.layers.Dense(11, activation='sigmoid'),
    tf.keras.layers.Softmax()
])

model3.compile(optimizer=tf.keras.optimizers.Adam(),
              loss=tf.keras.losses.CategoricalCrossentropy(),
              metrics=['categorical_accuracy'])


texts: How to interpret statsmodel output - logit?

texts:
I ran a logit model using statsmodel api available in Python. I have few questions on how to make sense of these

1) What's the difference between summary and summary2 output?

2) Why is the AIC and BIC score in the range of 2k-3k? I read online that lower values of AIC and BIC indicates good model. Is my model doing good? Is there any optimal range for AIC and BIC?

3) As you can see covariance Type is non-robust. What is it and should I be concerned about it?

4) Is there any other field/item in the output that I should pay attention to?

5) You can see below how certain significant variables like X2,X8,X45 have very low coefficients. How can they be significant and still have very low or near to zero coefficient? Is it normal?

This is the output that I got

Summary output

Dep. Variable:  vae_flag        No. Observations:   3298
Model:  Logit                   Df Residuals:   3241
Method: MLE                     Df Model:   56
Date:   Mon, 30 Dec 2019        Pseudo R-squ.:  0.3347
Time:   21:18:36                Log-Likelihood: -1392.2
converged:  True                LL-Null:    -2092.7
Covariance Type: nonrobust      LLR p-value:    3.894e-256

Summary2 output

Model:  Logit                   Pseudo R-squared:   0.335
Dependent Variable: op_flag     AIC:    2898.4259
Date:   2019-12-30 21:18        BIC:    3246.1870
No. Observations:   3298        Log-Likelihood: -1392.2
Df Model:   56                  LL-Null:    -2092.7
Df Residuals:   3241            LLR p-value:    3.8937e-256
Converged:  1.0000              Scale:  1.0000
No. Iterations: 7.0000

Significant variables

      coef  std err   z     P>|z|   [0.025  0.975]
x2   0.0321 0.060   11.227  0.000   0.558    0.794
x6   2.2996 0.095   24.332  0.000   2.114    2.485
x7  -1.8795 0.082   -22.835 0.000   -2.041  -1.718
x8   0.0002 0.058   2.116   0.034   0.009    0.237
x16  0.2693 0.059   4.564   0.000   0.154    0.385
x33 -0.3138 0.139   -2.254  0.024   -0.587  -0.041
x34  0.4644 0.137   3.392   0.001   0.196    0.733
x45  0.0088 0.052   2.306   0.021   0.018    0.221
x52 -0.1755 0.087   -2.007  0.045   -0.347  -0.004
x55 -0.0982 0.050   -1.965  0.049   -0.196  -0.000


texts: Are Word2Vec and Doc2Vec both distributional representation or distributed representation?

texts:
I have read that distributional representation is based on distributional hypothesis that words occurring in similar context tends to have similar meanings.

Word2Vec and Doc2Vec both are modeled according to this hypothesis. But, in the original paper, even they are titled as Distributed representation of words and phrases and Distributed representation of sentences and documents. So, are these algorithms based on distributional representation or distributed representation.

How about other models such as LDA and LSA.


texts: K-means clustering on big data stored on multiple nodes on HDFS

texts:
I have a very basic question about how the k-means clustering algorithm works when the data is stored in HDFS.
Does the training take place on the subset of data on each node for a fixed value of K( say 3)? But then we will have 3*number of nodes centroids?

How would it be different if we applied linear regression? Will it only train on data stored on each node locally?


texts:
I am trying to predict the user location for a mobile app for a certain timestamp. So far I've gathered training data throughout different days. The data contains the latitude, longitude and the timestamp.

What would be the best algorithm/best way in order to train a model that takes a timestamp as an input and gives as output a latitude and longitude pair?


texts: Extracting data and saving data from hdf5

texts:
I have a hdf5 file, and I want to extract a part of the data and save it as the same format.

The data type is

DATATYPE  H5T_IEEE_F32BE

So far I have successfully extracted the data using h5dump but it is in ascii format. I couldn't save it as the same hdf5 format. I saw there are several flags for saving as xml formats, but it doesn't work for me. I have tried using flags like -x or -d such as

h5dump -d variable -s 1 10 -X output.file  input.file

It creates the output file, but it is an empty file.

edit: I need to do it using h5dump.


texts: Approximating density of test set in train

texts: I am looking for a method to approximate how similar a test set (i.e., test set  features) to a train set. For example, something like, for each row in test: is there a similar enough data point in train? I've been thinking about using a mixture model approach, but I haven't been able to find a good reference on this. Can anyone suggest a good approach, or provide good references for how to use mixture models for this application?


texts: What does xgb's scale_pos_weight parameter do for regression?

texts: From other posts (see Unbalanced multiclass data with XGBoost) and the documentation, scale_pos_weight in XGBoost appears to balance positive and negative cases, which seems to apply only to classification. However, it also appears to be an option in XGBRegressor (see  documentation). Before I dive into the source code, can someone explain what this does for regression?


texts: Is there any way to generate ROC curves from KEEL?

texts: i am using the KEEL gui for my experiments . I want to plot a ROC curve of the results produced by KEEL rather than a just a roc score value  .


texts: Parameztrization of a model with data-sheet's sensor information or with empirical data

texts:
When working with a 3D laser sensor (LiDAR), the volumetric point density versus distance $\rho_r$ can be theoretically worked out taking into account the physical properties of the laser (number of layers, TOF, etc.). On the other hand, it can also be computed $\hat \rho_r$  from the available training data, yielding into an estimation of this quantity.

When this density vs. distance is wanted to be included in a classification problem (in the preprocessing stage as parameter) what value should be regarded? I share my concerns about the two different approaches:

Theoretical values $\rho_r$ : It will give an upper bound, as it is only taking into account the nature of the sensor. Furthermore, if only physical properties from the data-sheet are used and no statistical information about the uncertainty of the measurements are regarded, this value will be deterministic.
Empirical values $\hat \rho$ : they collect both the nature of the sensor and the scenario. A clear disadvantage is that, it will work well for one scenario and not for others. If the estimator is consistent and asymptoticly unbiased, the more data the better the parametrization.

The classifcation task is approached with CNN. Should the empirical value be used only for sepecific types of scenarios/applications? If a general classification model is itended, should the theoretical value be used, or having enough variability on the training data will make the empirical value work better?


texts: Recommendations for Data Visualization solution for large CSV file on a non-profit website

texts:
I would like your advice and recommendations for a non-profit that is trying to turn a large CSV file into meaningful information.
I am part of a non-profit that would like to turn a 23,000 row x 23 column CSV file into charts and graphs.  The CSV file is published by the Georgia Bureau of Investigation several times per year and is publicly available here:
http://state.sor.gbi.ga.gov/SORT_PUBLIC/sor.csv
As you will see, the CSV contains over 23,000 individual records so grouping of those records by county would be necessary.  Keeping archive data for trends would be nice but isn't required.
The goal of the solution would be to visualize data for each of Georgia's 159 counties as well as an overall visualization.  I envision pie or donut charts representing demographic information like Race, Gender, and other data that the CSV collects.  I'd like for each of the 159 counties to have their own report and then an overall report for the state of Georgia as a whole.  The solution would need to be embedded on our website.  It would not need to track data in real time as I will probably upload a new CSV on a quarterly basis.
My website is running the Wordpress CMS with CiviCRM (PHP 7.3, MySQL 5.7).  I have full root access to the site so I'm able to install web apps if needed.
Does anyone have suggestions for accomplishing this?  I'm currently looking into GoodData.  We are a non-profit so the more affordable the solution, the better.  I don't mind installing an open source solution to save money over a SaaS hosted solution.
Thanks for your suggestions.  I'd be glad to provide additional information if you need it.


texts: Which machine learning algorithms result in trained models with deterministic execution times?

texts: Which machine learning algorithms result in trained models which have a deterministic execution time?


texts:
I am a CS intern at an industrial company that has 30 years of excel files that need to be analyzed. Looking at the data, only a fraction of the files need to be looked at and used. After those files are identified, I need to pull out values from specific columns. The real issue is that there is no standard excel format for the tests and each column name can be different (ex. 'Front Axial Temperature' vs 'axial temp front') but contains the same type of data.

So, the data is semi-structured, are there any python/java libraries which are good at pulling such data out? I could create some regex patterns but as you can see the above names can be drastically different (while still meaning the same thing).

I would also be interested in looking at services that do this already.


texts: Look up a number inside a list within a pandas cell, and return corresponding string value from a second DF

texts:
(I've edited the first column name in the labels_df for clarity)

I have two DataFrames, train_df and labels_df.  train_df has integers that map to attribute names in the labels_df.  I would like to look up each number within a given train_df cell and return in the adjacent cell, the corresponding attribute name from the labels_df.

I've tried variations of the function below but fear I am wayyy off:

def my_mapping(df1, df2):
    tags = df1['attribute_ids']
    for i in tags.iteritems():
        df1['new_col'] = df2.iloc[i]
    return df1

The data are originally from two csv files:

train.csv

labels.csv

I tried this from @Danny :

sample_train_df['attribute_ids'].apply(lambda x: [sample_labels_df[sample_labels_df['attribute_name'] == i]
                                              ['attribute_id_num'] for i in x])

*please note - I am running the above code on samples of each DF due to run times on the original DFs.
which returned:


texts:
I have a dozen of databases that stores different data, and each of them are 100TBs in size. All of the data is stored in AWS services such as RDS, Aurora and Dynamo.

Many times I find myself need to perform "joins" across databases, for example a student ID that appears in multiple databases with data that I want to gather. The joins are usually done after data is streamed out of the database, since the data is not located in the same database, and this sometimes requires hours just for thousands of records.

Can services such as AWS redshift or Google BigQuery allow you to somehow "import" data from many data sources and then you can perform SQL queries to join them?

How about Hadoop and Hive? Where we steam data out from the database and place it as files in Hadoop, and let Hive Query the data?


texts:
Is time series model suitable for network syslogs considering the fact the messages are sequential and the messages are outputted as a result of dependency between themselves which can range from normal notifications to alarms if there are.

Can the state of the art models like ARIMA be applied ?

What kind of questions can be answered with applying time series models on syslogs ?


texts:
I'm working on a dataset which isn't normally distributed. The dataset contains three dimensions like cost, discount and profit.

I'm trying to find possible outliers in all these dimensions. I used Z-score to detect outliers in single dimension to find which high cost is causing outliers.

As a next step I tried to find outliers with high cost and high profit and low discount.

I came up with a formula of :

Zscore(cost) + Zscore(profit) - Zscore(discount)

negative sign because I want to find outliers with low discount.

Is this approach meaningful to do? or is there any further proven way to achieve this?


texts: Deep Learning - Find most similar images - Triplets vs Pairs

texts:
I am working with Python, scikit-learn, keras and with 450x540 rgb images of front-faced watches (e.g. Watch_1, Watch_2).

My aim to run an autoencoder or a Siemese Neural Network to find the most similar watches among them. However, I am not sure if I will get better results by comparing pairs of images or triplets of images. As it is defined in this research paper, triplets of images consist of one target image, one image which is (more) similar to the target image and one image which is not (so) similar to it.

Can someone explain me in simple terms why using triplets of images will (necessarily) yield better results than using pair images, as research papers like the previous one claim?


texts: Averaging two Word2vec vectors to obtain a unified representation for single word

texts:
I have been working on a trained data for Word2vec algorithm. Since we need words to stay as original we don't make them lowercase at the preprocessing phase. Thus there are words with different variations (e.g "Earth" and "earth").

The only way I can think of is to take average of vectors for "Earth" and "earth" to create a single vector to represent the word. (Since the feature vector's dimensions are similar)

Is this an "okay" method? If it is not, what might be a good way to handle this issue?

Note: Lowering all words in preprocessing is not an option for now.

Edit: The info about whether or not feature dimensions are truly linear would also be helpful.

Edit 2: Combining both answers from patapouf_ai and yazhi gave the best results. How are these combined? Weighted average improved the results but putting word frequencies through a scaled sigmoid function gave the best results, because using word frequencies in a linear manner gives them more importance than they bear.


texts: Appropriate Machine Learning algorithm for modeling clustered time-varying binary outcome

texts:
I'll just dive right in. I have a decent-size (100K observations) dataset of time-varying continuous and categorical predictors. Categorical predictors, actually, usually do not change, however, continuous one change every day. Another level of complexity - the one that I am struggling with - is the fact that the data are clustered at several levels (measurements coming from the same individual over time, with multiple individuals in the data set).
So, I have something like:
id | day | cont_predictor | cat_predictor | daily_outcome
1  | 1   | 1.4            | 1             | 0
1  | 2   | 1.7            | 1             | 0
1  | 3   | 1.9            | 1             | 0
1  | 4   | 2.9            | 1             | 1
2  | 1   | 4.0            | 0             | 0
2  | 2   | 4.1            | 0             | 0
3  | 1   | 5.7            | 0             | 0
3  | 2   | 4.2            | 0             | 0
3  | 3   | 3.5            | 0             | 1

I am looking for advice on what algorithm is best suited for modeling daily_outcome. This variable is highly imbalanced (30:1) but also the observations are time-varying and are clustered on an individual level (id). I could work with these data using a mixed effect model for longitudinal logistic regression. However, I am most interested in optimizing prediction, therefore I'd really like to use a machine learning approach.
So far, I have tried logistic regression and random forest, to start with, however, they are not performing well when predicting daily_outcome = 1, even after balancing and oversampling techniques including SMOTE. I am assuming it's because these models do not account for the structure of the data, where observations coming from one individual are highly correlated (I've examined those - very high intraclass correlations).
What are some machine learning algorithms that worked for you in the past? Looking for advice from people who worked with this kind of data (seems like this would be a fairly common problem to work on, but it's my first time working on it).
Thank you so much in advance!


texts: Subtracting grand mean from train and test images

texts: I am building an image classifier based off the VGG_face keras implementation. It is easiest for me to extract a csv file full of the representations and then try classifiers on those representations. When I got the representations, I first subtracted the mean of the entire dataset from each image. Then I realized... am I cheating, so to speak? In other words, since I included the test images when calculating the grand mean to be subtracted, does this now then overestimate my accuracy measurements?


texts:
In the MNIST dataset, you have 10 defined classes, one for each digit. But you don't have a "not a digit" class. It seems that most image classification datasets are the same. But in a business setting, for a production model, you could certainly get invalid images that don't correspond to any of the defined classes.

So let's say you were to create a handwritten digit image classifying model to be used in a real-world project. If you don't have a "not a digit" class, then if somebody submits a picture of the letter "M", then it will incorrectly be classified as one of the 10 digit classes.

So, in this example, should you define a "not a digit" class and train the model on a set of images that the model may be expected to receive that have nothing to do with handwritten digits such that invalid images are correctly classified as "not a digit"?


texts: Why are the regions/decision boundaries overlapping with multi-class classification using SVM in sci-kit?

texts:
I am using the SVM in scikit-learn library for doing multiclass classification. I am wondering why these regions (decision boundaries) are overlapping (as seen in the picture below)?

Could someone please explain the difference between whether I do one-vs-one or one-vs-all in terms of the regions overlapping? I assumed one-vs-one would have clearly delineated regions with no overlap since it's maximizing the margin against each other class and that one-vs-all could have regions overlapping, but perhaps this is inaccurate because 3 of the 4 models I am training are one-vs-one, and they show overlapping regions.

I've considered maybe it's a plotting issue as well, but could not determine any issues. If the alpha is 1, then the regions no longer overlap, but I assume this is expected since it's just covering up the other regions it overlays (which is to be expected and doesn't solve the problem).

Here is the function which creates, trains, and plots 4 different SVM models #(3 different kernels using SVC and 1 with LinearSVC).

def createSVMandPlot(X,y,x_name,y_name):
    h = .02  # step size in the mesh

    # we create an instance of SVM and fit out data. We do not scale our
    # data since we want to plot the support vectors
    C = 1.0  # SVM regularization parameter
    svc = svm.SVC(kernel='linear', C=C).fit(X, y) #1 vs 1
    rbf_svc = svm.SVC(kernel='rbf', gamma='scale', C=C).fit(X, y) #1v1
    poly_svc = svm.SVC(kernel='poly', degree=3, gamma='scale',C=C).fit(X, y) #1v1
    lin_svc = svm.LinearSVC(C=C).fit(X, y) #1 vs rest

    print(str(x_name)+' vs. '+str(y_name))
    for i, clf in enumerate((svc, lin_svc, rbf_svc, poly_svc)):

        X_pred=clf.predict(X)
        X_pred1=np.asarray(X_pred).reshape(len(X_pred),1)
        A=confusion_matrix(X_pred1, y)
        print(A)
        c=0
        for r in range(len(X_pred)):
            if X_pred[r]==y[r]:
                c+=1

        print(str(c)+' out of 34 predicted correctly (true positives)')

    =============================================================================
    with warnings.catch_warnings():

        warnings.filterwarnings("ignore")
        =============================================================================

        x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
        y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
        xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                             np.arange(y_min, y_max, h))

        # title for the plots
        titles = ['SVC w/ linear kernel',
                  'LinearSVC (w/ linear kernel)',
                  'SVM w/ RBF kernel',
                  'SVM w/ poly(degree 3) kernel']

        plt.pause(7)
        for i, clf in enumerate((svc, lin_svc, rbf_svc, poly_svc)):
            # point in the mesh [x_min, x_max]x[y_min, y_max].
            plt.subplot(2, 2, i + 1)
            plt.subplots_adjust(wspace=0.4, hspace=0.4)

            Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

            # Put the result into a color plot
            Z = Z.reshape(xx.shape)
            plt.contourf(xx, yy, Z, alpha=.5)

            # Plot also the training points
            plt.scatter(X[:, 0], X[:, 1], s=13,c=y)
            plt.xlabel(x_name)
            plt.ylabel(y_name)
            plt.xlim(xx.min(), xx.max())
            plt.ylim(yy.min(), yy.max())
            plt.xticks(())
            plt.yticks(())
            plt.title(titles[i])

            plt.show()

The result from this is an image with decision bounaries/regions overlapping. It implies that if a point is at a specific 2D coordinate (x1,y1), then it could be classified as two or more classes instead of just one which is not what is desired or expected. Could someone explain what might be going on? Thanks


texts:
I'm looking for a Python library to process large files without performance problems. For example I want to transfer large files with hash calculating. Python faces memory problems with such processes. Is there any library that helps? (such pandas in data science)

Any help or documentation would be appreciated.

Thanks


texts: Suggestion for a better way to organize data to generate frequent item-sets?

texts:
I have a data of a bag of words in a document. The data has 3 columns: {document number, word number, count of the word in the number}. I am supposed to generate frequent item-sets of a particular size.
I thought that I would make list of all words that appear in a document, create a table of this list, and then generate frequent item-sets using Mlxtend or Orange . However, this approach does not seem to be efficient.


texts: LSTM -RNN : How to get continuous range output instead of categorical?

texts:
I am trying to solve a problem predicting a value between a range for a sentence:

The dataset looks like this:

Index_no               text_sentence                             value

01                     yes I like riding a bike.
                       I was 4 when I learned                   4.2311
                       to ride a bike. the colors of my
                       bike is yellow and black.

02                     i like riding my bike, i learnt
                       riding a bike when i was 8 or            -2.11
                       9 years old ,my bike is sparkling
                       pink with white marks

Range of values is -7 to 7. Now, I am thinking about using a LSTM for text, but I am confused about the continuous output.

I was thinking about two methods:

Converting (normalizing the data between 0 and 1) and then after getting the output from network, denormalize the data, will this work?

Second approach, using a custom activation function?

Or how can I get output between a range?


texts: Understanding of number of cells in layers of sequential models

texts:
I am trying to teach myself RNN, but I have a question.
And so, imagine 2 layers: an input layer with three neurons $(x1, x2, x3)$ and a classic recurrent layer with 2 neurons and an activation function f. I will write out the outputs of each neuron of the recurrent layer. $ht1 = f (W * [x1, [0, 0, N]] + b) ht2 = f (W * [x2, ht1] + b)$. It turns out that $x3$ is not used, what to do in this case?
And also, let's imagine a slightly different RNN architecture.
An input layer with two neurons $(x1, x2)$ and a classic recurrent layer with 3 neurons and an activation function f. I will write out the outputs of each neuron of the recurrent layer. $ht1 = f (W * [x1, [0, 0, N] + bias]) ht2 = f (W * [x2, ht1] + bias)$. It turns out that the 3rd neuron of the RNN layer is not used, what to do in this case?
Please help me figure out how the neural network works in these cases. Thanks!
UPD:
I realized that i don't know how recurrent neural  networks work if  number of neurons in recurrent layer doesn't equal(!=) number of inputs
I have one thought:number of inputs has to always be equal number of neurons in RNN layer.
But code below contradicts with my guess.
model = Sequential()
model.add(Embedding(maxWordsCount, 256, input_length = inp_words))
model.add(SimpleRNN(128, activation='tanh'))
model.add(Dense(maxWordsCount, activation='softmax'))
model.summary()

That's model for predicting next word.


texts: Is Inferential and Descriptive Statistics for Machine Learning?

texts: I am starting with Machine Learning and many people suggested that I should first start with Descriptive and Inferential Statistics on Udacity. I started with it. But I have a doubt that is it absolutely necessary to do the courses to learn ML or I can directly start ML?


texts: Find matches between columns and manipulate data based on match

texts:
I have a pandas dataframe that contains two columns, components, and sub-components (first table below). I would like to add a product column as shown below (second table). The problem is,  some (not all) components are actually sub-components (B is also contained in the sub-component column in this simplified example). I would like some help/hints to write code in python to create this product column.

I envisage something like this but am finding it difficult to get started:

Iterate through Component column look for matches in the sub-component column, if no match is found write Component name to row in Product column - where a match is found in the sub-component column, look for the name of Component corresponding to sub-component and write it to the next row in the column Product. The line containing the "component in the sub-component column" can then be deleted.

Component           Sub-component
A                   a
A                   **B**
A                   c
B                   d
B                   e
B                   n
B                   a
B                   v
B                   c
B                   i
C                   g
C                   s
C                   g
C                   a
C                   c
C                   i
C                   q
C                   g
C                   v
C                   b
C                   l

Product Component           Sub-component
A       A                   a
*A      A                   **B***
A       A                   c
A       B                   d
A       B                   e
A       B                   n
A       B                   a
A       B                   v
A       B                   c
A       B                   i
C       C                   g
C       C                   s
C       C                   g
C       C                   a
C       C                   c
C       C                   i
C       C                   q
C       C                   g
C       C                   v
C       C                   b
C       C                   l


texts: Algorithm for detecting parity of integers

texts:
I'm still a beginner in machine learning and I wanted to do a simple test today with scikit-learn and I couldn't achieved what I expected. I created a dataset in which one column are natural numbers and the other column is whether the number is even or odd, I mean, only 0 or 1. Then I tried several different algorithms and I couldn't predict correctly the output for numbers different than those in the dataset. My question would be this: which algorithms could be useful for this purpose??? I thought for instance Logistic Regression could do the trick or KNN classifier, I'm starting to learn Machine Learning and felt very disappointed that this simple thing didn't work...

Best regards


texts: Unsupervised binary classification on small dataset

texts:
I have a small unlabeled textual dataset and I would like to classify all document in 2 categories.

I understand and have used supervised classification. I have searched a lot but still cannot understand how unsupervised binary classification works. Can you give me an example or a simple explanation ? Or some paper, document or code that explains it well ?

Thank you.


texts: Should the column be kept from which range column was extracted?

texts:
I'm creating a Neural Network model to predict the price range of laptop. Input would be the configurations of that laptop. But the price range column was created from the price column. If I keep the price column in the input, will my network act bias towards the price column, as it is be totally correlated and price range can be predicted with only the price column without all other configurations.
Here is the data-set. I already removed the price column from which the price_range was extracted. But my colleague suggest, it would be better if I keep price column in the independent variable.


texts: How to use the $\chi^2$ test to select the features, that can be String or categorical?

texts:
I want statistics to select the characteristics that have the greatest relationship to the output variable.

Thanks to this article, I learned that the scikit-learn library proposes the SelectKBest class that can be used with a set of different statistical tests to select a specific number of characteristics.

Here is my dataframe:

    Do you agree    Gender  Age     City     Urban/Rural  Output
0   Yes             Female  25-34   Madrid   Urban        Will buy
1   No              Male    18-25   Valencia Rural        Won't
2   ...             ...     ...     ...      ...          Undecided
....

The output is 'Will buy', 'won't' and 'undecided'.

I then tried the chi-square statistical test for non-negative characteristics to select 10 of the best characteristics:

import pandas as pd
import numpy as np
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

data = pd.read_csv("D://Blogs//train.csv")
X = data.iloc[:,0:20]  #independent columns
y = data.iloc[:,-1]    #target column i.e price range#apply SelectKBest class to extract top 10 best features
bestfeatures = SelectKBest(score_func=chi2, k=10)
fit = bestfeatures.fit(X,y)
dfscores = pd.DataFrame(fit.scores_)
dfcolumns = pd.DataFrame(X.columns)
#concat two dataframes for better visualization
featureScores = pd.concat([dfcolumns,dfscores],axis=1)
featureScores.columns = ['Specs','Score']  #naming the dataframe columns
print(featureScores.nlargest(10,'Score'))  #print 10 best features

But certain columns are 'String'. So, I get the terminal back:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-59-e64d61febefd> in <module>
      1 bestfeatures = SelectKBest(score_func=chi2,k=10)
----> 2 fit = bestfeatures.fit(X,y)
      3 dfscores = pd.Dataframes(X.columns)
      4 #concat two dataframes for better visualization
      5 featuresScores = pd.concat([dfcolumns,dfscores], axis = 1)

C:\ProgramData\Anaconda3\lib\site-packages\sklearn\feature_selection\univariate_selection.py in fit(self, X, y)
    339         self : object
    340         """
--> 341         X, y = check_X_y(X, y, ['csr', 'csc'], multi_output=True)
    342
    343         if not callable(self.score_func):

C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    754                     ensure_min_features=ensure_min_features,
    755                     warn_on_dtype=warn_on_dtype,
--> 756                     estimator=estimator)
    757     if multi_output:
    758         y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False,

C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    565         # make sure we actually converted to numeric:
    566         if dtype_numeric and array.dtype.kind == "O":
--> 567             array = array.astype(np.float64)
    568         if not allow_nd and array.ndim >= 3:
    569             raise ValueError("Found array with dim %d. %s expected <= 2."

ValueError: could not convert string to float: 'Yes'


texts:
I am working on a Generative Adversarial Network, implementing in Keras. I have my generator model, G, and discriminator D, both are being created by two functions, and then the GAN model is created using these two models, like this light sample of the code:

gopt=Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-08)
dopt=Adam(lr=0.00005, beta_1=0.9, beta_2=0.999, epsilon=1e-08)
opt_gan = Adam(lr=0.00006, beta_1=0.9, beta_2=0.999, epsilon=1e-08)

G= gmodel(......)
G.compile(loss=...., optimizer=gopt)

D=dmodel(..)
D.trainable = False

GAN=ganmodel(generator_model=G,discriminator_model=D,...)

GAN_model.compile(loss=["mae", "binary_crossentropy"], loss_weights=[0.5, 0.5], optimizer=opt_gan)

D.trainable = True
D.compile(loss='binary_crossentropy', optimizer=dopt)

now my question or better the confusion is, how the optimization will work when we train the GAN model? More precisely I am interested in the learning rate. When I train the GAN, what learning rate will be applied on the generator?

Since I have already compiled G before sending it to the GAN model, its optimizer should not change, and so the learning rate should be 0.0001? or the learning rate of GAN will be applied, ie 0.00006? How about the discriminator?


texts: How do I initialize a Hidden Markov Model when using MFCC features for speech recognition?

texts:
I have a personal dataset of 10000 audio files, each consisting a single spoken sentence. These files each have the transcribed text labels with them that I can use for supervised HMM training.
Now that I have have extracted MFCC features, how do I input these vectors of MFCC sequences for each format into the HMM? After reviewing multiple source materials, my head tells me to initialize the $N \times N$ input transition matrix with all of the vectors, but how does this segment out the starts and ends of each spoken sentence sequence? I'm also unsure how to assign number of states when the files each vary in word length.
This is my personal understanding:
$N \times N$ transition probability matrix – which I believe are for words (language model).
$1 \times N$ start & end probability matrices – which I also believe are $0$th order Markovian per word.
The type of underlying distribution(s) used for each state.
Please correct me if my perspective is wrong.


texts: Usage of Precision Recall on an unbalanced dataset

texts:
I am new to data science and am working on a dataset having roughly 213,000 rows and 31 columns. The 31st column being my Response variable having values 0's and 1's. Its a classification problem and the data set is unbalanced  as after I applied logistic regression to it, I got a model accuracy of 99.79%, however, by just counting the total number of 0's and 1's it would still show an accuracy of 99+% as it correctly classifies max no of 0's.

Also the confusion matrix is of no help too.

I did some digging up and learned to use precision recall in such scenarios. My question is should I even use logistic regression on this dataset? and then use precision recall?

If anyone can shed any light on what approach should I take, that would really help me move forward.


texts: How to calculate Average Precision for Image Segmentation?

texts:
If I've understood things correctly, when calculating AP for Object Detection (e.g. VOC, COCO etc) the procedure is:

collect up all the detected objects in your dataset
sort the detections by their confidence score
categorise each detection as True Positive or False Positive, by comparing the Intersection over Union with a Ground Truth object to a pre-set threshold
plot Precision $\frac{TP}{n}$ against Recall $\frac{n}{N}$, where n is the number of objects in the list that have been considered so far, and N is the total number of objects
integrate Precision with respect to Recall. (There are various different ways to perform the integration.)

When I attempt to replicate these steps for Segmentation, I found that my segmentation CNN didn't provide the confidence as an output. Even if it did, it would presumably be for each individual pixel. So I am stuck at step 2.

Calculating AP without sorting by confidence will obviously change the result. But is it still "valid" in some sense? If not, is there a roughly equivalent metric I could use to compare segmentation results? (Or perhaps more generally, a metric for detection where ranking is not possible?)

Edit: looking at VOCdevkit, it seems that they use accuracy $\frac{TP}{TP+FP+FN}$ rather than AP as the metric to evaluate segmentation. Is that what I should be doing? It seems to me that AP is the "better" metric, so I would prefer to use something as close to that as possible.

Looking at the Berkeley Simultaneous Detection and Segmentation code, and the accompanying paper, they calculate a pixel-wise AP (called $AP^r$), but it seems like they have a confidence score for each object.


texts: What are some possible reasons that your multiclass classifier is classifying alll the classes in a single class?

texts:
I have unbalanced classes.
Group1 N = 140
Group2 N = 35
Group3 N = 30

I ran the code on this data and all the Groups got classified as Group1.
I thought that since group1 is the majority group this is not a surprise.
Then i ran the same code but this time with SMOTE, now all groups are 140, and i still got the same results, where all the groups were classified in Group1. Then i balanced the class weights (W/O SMOTE), but still got the same results. This was confusing to me. What am i doing wrong? Can someone help me understand this please? or what can i do to improve the model?
I tried 5 different classifiers (KNN, AdaBoost, SVC, RF, DT) and in 4 out of 6 i got the same result!

Here's the code:

#Splitting data to training and testing
X_train, X_test, y_train, y_test = train_test_split (X, y, test_size = 0.1, random_state=42)

#Apply StandardScaler for feature scaling
sc = StandardScaler()
X_train_std = sc.fit_transform(X_train)
X_test_std = sc.transform (X_test)

#SMOTE
sm = SMOTE(random_state=42)
X_balanced, y_balanced = sm.fit_sample(X_train_std, y_train)

#PCA
pca = PCA(random_state=42)

#Classifier regularization (SVC).

svc = SVC(random_state=42, class_weight= 'balanced')
pipe_svc = Pipeline(steps=[('pca', pca), ('svc', svc)])

# Parameters of pipelines can be set using ‘__’ separated parameter names:
parameters_svc = [{'pca__n_components': [2, 5, 20, 30, 40, 50, 60, 70, 80, 90, 100, 140, 150]},
                   {'svc__C':[1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
                    'svc__kernel':['rbf', 'linear','poly'],
                    'svc__gamma': [0.05, 0.06, 0.07, 0.08, 0.09, 0.001, 0.002, 0.003, 0.004, 0.005, 0.006, 0.007,
                                   0.008,0.009, 0.0001, 0.0002, 0.0003, 0.0004, 0.0005],
                    'svc__degree': [1, 2, 3, 4, 5, 6],
                    'svc__gamma': ['auto', 'scale']}]

clfsvc = GridSearchCV(pipe_svc, param_grid =parameters_svc, iid=False, cv=10,
                      return_train_score=False)
clfsvc.fit(X_balanced, y_balanced)

# Plot the PCA spectrum (SVC)
pca.fit(X_balanced)

fig1, (ax0, ax1) = plt.subplots(nrows=2, sharex=True, figsize=(6, 6)) #(I added 1 to fig)
ax0.plot(pca.explained_variance_ratio_, linewidth=2)
ax0.set_ylabel('PCA explained variance')

ax0.axvline(clfsvc.best_estimator_.named_steps['pca'].n_components,
            linestyle=':', label='n_components chosen')
ax0.legend(prop=dict(size=12))

# For each number of components, find the best classifier results
results_svc = pd.DataFrame(clfsvc.cv_results_) #(Added _svc to all variable def)
components_col_svc = 'param_pca__n_components'
best_clfs_svc = results_svc.groupby(components_col_svc).apply(
    lambda g: g.nlargest(1, 'mean_test_score'))

best_clfs_svc.plot(x=components_col_svc, y='mean_test_score', yerr='std_test_score',
               legend=False, ax=ax1)
ax1.set_ylabel('Classification accuracy (val)')
ax1.set_xlabel('n_components')

plt.tight_layout()
plt.show()

#Predicting the test set results (SVC)
y_pred1 = clfsvc.predict(X_test)

# Model Accuracy, how often is the classifier correct?
Accuracyscore_svc = accuracy_score(y_test, y_pred1)

print("Accuracy for SVC on CV data: ", Accuracyscore_svc)

# Making the confusion matrix to describe the performance of a classifier
from sklearn.metrics import confusion_matrix
cm1 = confusion_matrix (y_test, y_pred1)

#accuracy
# Get accuracy score
accuracy1 = accuracy_score(y_test, y_pred1)
print('Accuracy1: %.2f%%' % (accuracy1 * 100.0))

#Checking shape after confusion matrix
print (X_test)
print (y_pred1)

print (cm1)


texts: Next year forecasting with monthly data from many, correlated, non-monotonic trends

texts:
I have trend data from many health departments in a local territory (eg. cardiology, orthopedics, etc...). These trends represent health service (visits, diagnostic, admissions) production, service value (proxy of type of service), and ratio of such service consumed by patients in our territory versus those consumed in other health providers than us, per department (eg. how much our patients have hip surgery in our hospitals versus other hospitals).

I would like to predict for each department or group of departments (all diagnostic or all surgery departments) those values for the next year. I have monthly data from the previous ten years.

Since there are many unmeasured sources of variability I can't expect to make long term predictions, by just one year is enough. Also, these trends have many change points (change of organization, personnel, technology), so I can mostly hope to predict the next year value once a new trend has started, predict for how long it will grow, reach a tipping point or going down.

I guess the model for each department could learn from previous growth and fall dynamics from his own past trend data and from the trends of other departments with similar dynamics and adjust for month related activity level, plus from information on where the service was provided.

Which model should I use for such a problem? I thought about a multilevel model for time series, but I guess there are more raffinate solutions.


texts: Best way to handle padding in time series data such as text

texts:
I have a bunch of documents containing sequential data that I want to
use to train a neural network with. It is as a collection of letters each about a 2-3000
characters long. My task is, given an excerpt of such a letter I want
my network to output plausible text in the same style as the letter.

What would be the best way of presenting my data to the neural
network?

One method which I've seen in tutorials is to concatenate all letters
into one long text sequence and letting the network train on that. But
then some training sequences will contain text from multiple
letters. Suppose a letter is 500 characters long and the training
sequence length is 150 then the last training sequence in that letter
will contain 50 characters from that letter and 100 characters from
the subsequent one. I suspect that would confuse the network.

Another idea would be to insert an end of letter token between each
letter and then concatenate them. But again some training sequences
would contain the end of letter token and training on them would
perhaps confuse the network too.

A third idea would be to insert a very long sequence of padding tokens
between each letter so that no real training sequence contains text
from two different letters. But that seems very wasteful. For example,
if my training sequences are 256 characters long I would have to
insert 256 padding tokens between each letter.


texts: How to Determine Alpha values in Random Forest Classifier

texts:
I am working on a classification problem where I am applying various machine learning models.
I have used DecisionTreeClassifier from Sklearn on my dataset using the following steps:
Calculated alpha values for the decision tree using the cost_complexity_pruning_path method
Used GridSearchCV to identify best ccp_alpha value and other parameters. I specified the alpha value by using the output from the step above.
When I review the documentation for RandomForestClassifer, I see there is an input parameter for ccp_alpha. However I am confused on how the alpha value for pruning can be determined in Random Forest.
My understanding of Random Forest is that the algorithm will create n number of decision trees (without pruning) and reuse the same data points when bootstrap is True (which is the default value). The model will predict the classification class based on the most common class value from all decision trees (mode value).
The decision trees in random forest will not be same (generally speaking as that is how the algorithm is designed) and therefore the alpha values for the corresponding decision trees will also differ.
I have 2 questions:

How can you determine the ccp_alphas value in RandomForestClassifier so that it can be entered as an input?
Can you conduct hyperparameter tuning for ccp_alpha value using GridSearchCV for RandomForestClassifier? If yes how can you set this up?


texts: Comparison between addition and multiplication function in deep neural network?

texts:
I designed a specific Convolution Neural Network to study in the area of image processing. The network has a part that there are two tensors which have to be transformed into a tensor in order to be fed to the next layer. This situation happen in several points of the network. In fact, there are several operations such as addition, multiplication, etc. The results of the network are a bit better when I  use the addition pyramid pooling module (the second image between two convolution) and multiply function (in last step of the network). I used tf.math.add and tf.math.multiply which perform the operation element-wisely. The whole network is shown in the first image.

The second image represents the pyramid pooling module which includes several scale images.

I am looking forward to the addition and multiplication function's attribute in a deep neural network.

The question is:

Why does the addition function (between conv1 and conv2) indicate better final performance in Accuracy (precision) and mean Intersection of Union(mIoU) compared to multiplication and concatenation when I unify two tensors into one tensor?


texts: Statistical inference on a very small datasets

texts:
I have been working with machine learning for about a year now, but mostly with large datasets. However, I am currently working on a problem with a very small dataset. Here is my problem: I am creating a rocket fuel with 4 ingredients, x1, x2, x3, x4, and I want to maximize reaction strength, y. I have already mixed them in the arrangements below to get the corresponding values.

(0.9)x1 + (0.0)x2 + (0.1)x3 + (0)x4 = 16.5
(0.0)x1 + (0.9)x2 + (0.1)x3 + (0)x4 = 8.6
(.45)x1 + (.45)x2 + (0.0)x3 + (0.1)x4 = 12.6
(0.6)x1 + (0.3)x2 + (0.05)x3 + (.05)x4 = 18.9
(0.3)x1 + (0.9)x2 + (0.05)x3 + (.05)x4 = 9.8

My next question is, how should I design my next few mixtures to maximize the reaction strength?  Can you suggest any algorithms or statistical frameworks to get me started? Much appreciated.


texts: Is it possible to do Robust PCA on Images

texts:
I have a task to perform outlier detection on a medical image data set (CheXpert), but with more „classical“ techniques (i.e. not use a DL approach like GANs or AEs) and I was wondering whether robust pca could be a possible method? I’ve seen it used on time series but not on images so I was wondering if this was a good approach or what the best practice was and why? If you know of any papers or code on this topic I’d be very grateful. So far I’ve read this 1
Thank you!


texts: Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?

texts:
I am using custom C++ code, and coded a simple "Mean Squared Error" layer.
Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues?

I don't have anything else before this layer - not even a simple Dense layer. It's just MSE on its own.
Its input is a collection of rows of input features. For example, here are 8 rows of input features that will be passed to MSE all at once:

{ a0, a1, a2, a3, a4, a5, a6, a7 }
{ b0, b1, b2, b3, b4, b5, b6, b7 }
{ c0, c1, c2, c3, c4, c5, c6, c7 }
{ d0, d1, d2, d3, d4, d5, d6, d7 }
{ e0, e1, e2, e3, e4, e5, e6, e7 }
{ f0, f1, f2, f3, f4, f5, f6, f7 }
{ g0, g1, g2, g3, g4, g5, g6, g7 }
{ h0, h1, h2, g3, h4, h5, h6, h7 }    //8x8 matrix (contains 64 different values)

Every row of this matrix gets passed into my "Mean Square Error" layer, returning a single scalar for such a row: "Cost".

I then compute a "final error" scalar, which is the average of such Costs.

When doing Gradient Checking, I am looking at how this "final error" quantity changes as I perturb each of the 64 input values, seen above. The idea is that the changes in finalError must correspond to the gradient computed by formula, with respect to my 64 input values. If they match, then I've coded backprop correctly.

Here is the forward prop:

$$finalError =   \frac{1}{r}\sum^r{  \left(  \frac{1}{2n}\sum^n{(input_i-target)^2} \right) }   $$

where $n$ is the number of features per row, and $r$ is the number of rows.

Here is the gradient wrt one of input values, that my backprop is using:

$$\frac{\partial finalError}{\partial input_i} = \frac{1}{rn}(input_i - target)$$

Question:

I peturb each input value 'up', then 'down', running forward prop 64*2 = 128 times. This gives me numerical estimate of gradient for my 64 input values.

However this numerical estimate and the actual analytical gradient become less similar when smaller epsilon is used. This is counterintuitive to me. On the contrary, my vectors match almost exactly, when I use a ridiculously large value for epsilon, such as $1$

Is this expected, or do I have an error in C++ code?

Here is the pseudocode

for every input value i:
   i -= EPSILON
   finalCost_down =  fwdprop( inputMatrix )//very simple - just computes final cost via MSE layer.  finalCost_down is a scalar.
   i += EPSILON
   finalCost_up   =  fwdprop( inputMatrix )
   gradientEstimate[i] = (finalCost_up - finalCost_down) / (2*EPSILON)

//after the loop, some time later, just one invocation of backprop:
trueGradientVec = backprop( vec )

//some time later:

discrepancyScalar =  (gradientEstimate - trueGradientVec).magnitude / gradientEstimate.magnitude + trueGradientVec.magnitude)

//somehow discrepancyScalar decreases the larger the EPSILON was used:
// discrepancy is 0.00275, if EPSILON is 0.0001
// discrepancy is 0.00025, if EPSILON is 0.001
// discrepancy is 2.198e-05, if EPSILON is 0.01
// discrepancy is 3.149e-06, if EPSILON is 0.1
// discrepancy is 2.751e-07, if EPSILON is 1

I would expect discrepancy to decrease when epsilon is decreased, because finer perturbations should give more precise slope estimate...

Andrew NG explanation of GradientChecking


texts: Why does this paper say that 0-1 loss is insensitive to scaling of weights in a neural network?

texts:
When discussing capacity control using norms of weights in a neural network,this paper says the following(see P4):

  Capacity control in terms of norm, when using a zero/one loss (i.e. counting errors) requires us in
  addition to account for scaling of the output of the neural networks, as the loss is insensitive to this
  scaling but the norm only makes sense in the context of such scaling. For example, dividing all the
  weights by the same number will scale down the output of the network but does not change the $0 / 1$ loss, and hence it is possible to get a network with arbitrary small norm and the same $0 / 1$ loss. Using
  a scale sensitive losses, such as the cross entropy loss, does address this issue (if the outputs are scaled
  down toward zero, the loss becomes trivially bad), and one can obtain generalization guarantees in
  terms of norm and the cross entropy loss.

I'm a little bit confused about this paragraph, since it seems that the 0-1 loss is not insensitive to weight scaling. For example, if we scale down the weights, the output would also be scaled down, so the 0-1 loss would definitely be changed(since the true labels are not changed).Can anyone explain to me which part of my understanding goes wrong? Also, I would like to know how the cross-entropy loss settles this problem.


texts: Transforming data points to match other data points

texts:
I have two sets of data points, set A and set B.
Each point of both sets has two dimensions $(x, y)$.
Consider the set $A$ to be the orange set on the left of the image below and $B$ to be the blue on the right (some points are more transparent, because the points represent a position over time; the darker the point, the more recent is the position it represents).

I can observe that there is a similarity between both sets. If I perform some rotation and scaling to set $B$, I end up with a set $B'$ more similar to set $A$.
My question is, what method can I used to find the transformation that makes set $B$ as similar to $A$ as possible? And how can I measure the similarity?
I do not need a programming solution, just some tips on possibly existing algorithms.
If there is no easy solution, I'd like to know if there is some designation for this type of problem or some keywords that could point me in the right direction.
Thanks!


texts: Text generation - Input text (one sentence or many sentences)

texts:
I am currently working on a project: I want to generate text with a LSTM using Pytorch. My model is working but I have a question about the methodology:
I'm using the BPTTIterator and something seems weird to me: you have to give one big example with all your text in it then it will provide your neural network with the current word and the target word for each step. So, it looks like that my LSTM won't be able to "understand" if it's the beginning or the end of the sentence. I saw that usually you feed your LSTM with one sentence at the time with SOS and EOS tokens.
My question is: what does it changes to use one or another method and which one is the best practice?


texts: Key pixels, key "features" detection in CNNs

texts:
I am working on a dataset but I don't know what the labels mean.
I was wondering if using CNNs there was a way to understand which pixels where most significant for the network.
A little bit in the way that running an SVM would show which pixels are the most usefull for the classification.
Or, if there is any way to understand what my  Deep Learning algorithm is training on?

To give a little more detail I'm using VGG on the celebA dataset getting 85% accuracy on test-set. Would you know how to get back to the label's meaning and therefore maybe crop part of the image that is useless.


texts: Contrastive loss problem in a character-level, siamese NN model

texts:
Summary: my NN with contrastive loss does not work, need help debugging

Background:
I am trying to replicate this paper

At first, I used binary crossentropy for loss, and the results were very good, but there was a caveat. The model correctly predicted which job titles were similar, but only if it had seen both titles in the training data. If it saw a given title for the first time, it would ALWAYS predict 1 (i.e. that they are similar). This is very undesirable for me product-wise.

Then, I tried the following implementation of the contrastive loss:

def contrastive_loss(y_true, y_pred):
    """Contrastive loss from Hadsell-et-al.'06
    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
    """
    margin = 1
    sqaure_pred = K.square(y_pred)
    margin_square = K.square(K.maximum(margin - y_pred, 0))
    return K.mean(y_true * sqaure_pred + (1 - y_true) * margin_square)

However, what happens now is that the model always predicts all zeros.
I run this code with print_tensor...

    sqaure_pred =  K.print_tensor(K.square(y_pred))
    margin_square =  K.print_tensor(K.square(K.maximum(margin - y_pred, 0)))
    return  K.print_tensor(K.mean(y_true * sqaure_pred + (1 - y_true) * margin_square))

...and this is what I saw

Any ideas what I should try next?

This is the model:

def build_blstm_encoder(params):
    lstm = params['lstm']
    nb_tokens = params['nb_tokens']
    maxlen = params['max_seq_len']
    offer_rep_dim = params['offer_rep_dim']
    emb_len = params['emb_len']

    input_1 = Input(shape=(maxlen,), dtype='int32')
    input_2 = Input(shape=(maxlen,), dtype='int32')
    emb_layer = Embedding(nb_tokens, output_dim=emb_len, input_length=maxlen, mask_zero=False)
    blstm_layer = Bidirectional(LSTM(output_dim=lstm, return_sequences=True), merge_mode='concat', weights=None)
    dense = Dense(offer_rep_dim, activation='relu')

    blstm_encoders = []
    for char_array in [input_1, input_2]:
        embs = emb_layer(char_array)
        blstm = blstm_layer(embs)
        dropout = Dropout(0.15)(blstm)
        dense_ = dense(dropout)
        flatten = Flatten()(dense_)
        blstm_encoders.append(flatten)

    distance = Dot([1, 1], normalize=True)(blstm_encoders)
    return Model([input_1, input_2], [distance])

The optimizer is just Adam() (default parameters)

Questions:
Any idea what went wrong?
What would you try next?


texts:
I was following andrew-ng coursera course on deep learning and there's a question that has been asked there which I couldn't figure out the answer for?

Suppose your input is a 300 by 300 color (RGB) image, and you use a convolutional layer with 100 filters that are each 5x5. How many parameters does this hidden layer have (including the bias parameters)?

2501
2600
7500
7600


texts:
What is the appropriate method to find n-grams/sub-phrases/parts-of-sequences that are referring to a specific topic or belong to a certain category?

For instance:

Imagine a topic of "transfer of ownership" and some example sentences:

A >change of ownership< occurs when a title is transferred from one person or entity to another
Making a >change in business ownership< is a lengthy and complex process
In this respect, the >transfer of legal ownership< is not the relevant feature for determining the treatment of repo-like operations

What I am looking for is a method to identify sentences and ideally parts of sentences which refer to a specific theme / topic.

What NLP methods are appropriate?


texts: Creative Analysis - What are the possible approaches?

texts:
We're analyzing some creatives from past advertising campaigns and are looking for the best way to approach. Currently we've broken down each creative into 20+ objective characteristics, for example:

CTA button (Y/N)
Exclusive discount mentioned (Y/N)
etc

For each campaign we have two main data points: CTR and conversions.
What are some possible ways to approach analyzing this data? We're looking to find out how each feature affects CTR/conversions.


texts: Is there any activation function that can make batch normalization obsolete?

texts:
While learning batch normalization, I was thinking why can't we solve the "gradient scale problem" by using an appropriate activation function ?

Like can't we delay and scale the activation function instead of scaling the whole dataset and ensure that the variance is preserve through it ?