texts:
I have thousands of headlines and I would like to build a semantic network using word2vec, specifically google news file.
My sentences look like
    Titles
    Dogs are humans’ best friends
    A dog died because of an accident
    You can clean dogs’ paws using natural products.
    A cat was found in the kitchen

And so on.
What I would like to do is finding some specific pattern within this data, e.g. similarity in topics on dogs and cats, using semantic networks.
Could you give me some advice on how I can do it?
Code:
import pandas as pd
import gensim
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from sklearn.manifold import TSNE

main_data.Titles = np.where(main_data.Titles.isnull(),'NA', main_data.Titles)

article_titles = main_data['Titles']

titles_list = [title for title in article_titles]

big_title_string = ' '.join(titles_list)

tokens = word_tokenize(big_title_string)

words = [word.lower() for word in tokens if word.isalpha()]

stop_words = set(stopwords.words('english'))

words = [word for word in words if not word in stop_words]

model = gensim.models.KeyedVectors.load_word2vec_format('path/GoogleNews-vectors-negative300.bin', binary = True)

model.vector_size

vector_list = [model[word] for word in words if word in model.vocab]

words_filtered = [word for word in words if word in model.vocab]

word_vec_zip = zip(words_filtered, vector_list)

word_vec_dict = dict(word_vec_zip)
df = pd.DataFrame.from_dict(word_vec_dict, orient='index')

tsne = TSNE(n_components = 2, init = 'random', random_state = 10, perplexity = 100)

tsne_df = tsne.fit_transform(df[:400])

sns.set()
fig, ax = plt.subplots(figsize = (11.7, 8.27))
sns.scatterplot(tsne_df[:, 0], tsne_df[:, 1], alpha = 0.5)

from adjustText import adjust_text
texts = []
words_to_plot = list(np.arange(0, 400, 10))

for word in words_to_plot:
    texts.append(plt.text(tsne_df[word, 0], tsne_df[word, 1], df.index[word], fontsize = 14))

adjust_text(texts, force_points = 0.4, force_text = 0.4,
            expand_points = (2,1), expand_text = (1,2),
            arrowprops = dict(arrowstyle = "-", color = 'black', lw = 0.5))

plt.show()

However I cannot understand how to interpret the results. I think they are wrong and probably this is not a good approach for building a semantic network. maybe I have been missing something...For instance, this code is still keeping stopwords after the part of
words = [word for word in words if not word in stop_words]

This is an example of output difficult to read and explain (at least, for me):

I would greatly appreciate if you could give me some tips and advice on how to perform a semantic network which can show semantic similarity within titles.


texts: Predicting time of event from multiple timeseries

texts:
I apologize for the newbie question, but I'm hoping someone here can point me in the right direction.
I want to predict the time at which an event (x) occurs given several timeseries which I suspect are correlated with the timing of X. My model would take in the full timeseries as input, and output a single value for the time at which event X is predicted to have occurred. Note that I'm not trying to predict the timing of a future event, but the timing of a past event. When using the model, I will have access to the entire timeseries. Here's a quick sketch of what the data looks like:

My dataset consists of many examples like the above, that is, several timeseries within a set window along with the time of the event.
Can something like an LSTM model be trained for this task? After doing a bit of digging it looks like LSTM and RNN models are good for predicting the next point in a timeseries, not so much outputting the time of an event like I've described.
I am able to fairly consistently fit curves to some of the timeseries, so I tried training some tree-based ML algorithms on the coefficients of these fitted curves with little success. I'm hoping to try a more sophisticated timeseries-based approach next.
I would appreciate being pointed in the right direction here - ie. what type of model or approach should I be looking into.


texts: Softmax classifier never allows for 100% probability in LSTM?

texts:
When working with LSTM I am using a softmax classifier and a one-hot encoded vector approach. The softmax looks like this:

$$S(h_i) = \frac{e^{h_i}}{\sum e^{h_{total}}}$$

notice, LSTM's result is a $h=tanh(c) \circ \sigma(p)$

Where c is the cell state passed through a tanh as well, and $\circ$ is the component-wise product of two vectors.

Recall that tanh never goes beyond -1 and never goes above 1; The $\sigma$ never goes below 0 and never above 1

Does this mean that if we have a 4-neurons on the output - the best guess a network can make will be 1, -1, -1, -1?

When softmaxed, this will produce $\frac{2.72}{3.83}, \frac{0.37}{3.83}, \frac{0.37}{3.83}, \frac{0.37}{3.83}$ which are these probabilities:

0.7,

0.1,

0.1,

0.1

and we can never get ~100% certainty, (Edit: or in this example even above 70%) no matter the learning rate?
Is there a way to combat it without destroying its derivative working nicely with Cross-entropy?

Edit

As a solution, I've tried to determine how close to $e^{−1}$ and $e^1$ the $e^x$ actually lies. Basically an 'inverse lerp', and it indeed puts values to 0 and 1 range. However, I am not sure if it affects derivatives & if it no longer will work out of the box when using Cross Entropy.

Here is the usual way to compute softmaxed-vector:

float totalSum = 0;
Vec softmaxedVec;
for(int i=0; i< tanhVec.Length; ++i){
   softmaxedVec[i] =  exp( tanhVec[i]);
   totalSum += softmaxedVec[i];
}

for(int i=0; i<softmaxedVec.Length; ++i){
   softmaxedVec[i] = softmaxedVec[i] / totalSum;
}

and here is what I am now trying to do:

float totalSum = 0;
Vec softmaxedVec;
for(int i=0; i< tanhVec.Length; ++i){
   softmaxedVec[i] =  inverse_lerp(0.3678,  2.71828,  exp( tanhVec[i]));
   totalSum += softmaxedVec[i];
}

for(int i=0; i<softmaxedVec.Length; ++i){
   softmaxedVec[i] = softmaxedVec[i] / totalSum;
}

//returns percentage: 0 the value sits on min, 1.0 the value sits on max
float inverse_lerp(float min, float max, float currValue){
    return (currValue - min) / (max - min);
}

However, with this I am afraid that I might have lost that gorgeous $\frac{\partial E}{\partial W} = expected - target $  (curtesy of softmax & cross entropy working together)

..or if I basically ruined the benefit of softmax


texts:
I have programmed an Mixture Density Network model to a market price. As input I have many numerical and categorical properties. The output of the network is a probability distribution (shape, location and scale parameter) over the market price.
The great thing about the network is that each datapoint has its own distribution. Very similar data points also have very similar distributions, but not exactly the same. Its a continious change over the input dimensions!
Currently I am facing the problem of evaluation.
How can I evaluate the results?
Here are some ideas:

Take a test set and predict the distribution for each point. Check for each point if the true price is within the 10% interval of the predicted distribution (20%, ..., 90%).
In the end 10% of all prices should also be in the 10% interval of the predicted distribution. (20% of the prices in the 20% interval of their distribution, ...)
I have checked this and it is indeed the case.

Take a cluster of data points that are very similar and predict the distributions. sample a price from the corresponding distribution and save it. Do this for each individual datapoint. In the end, the distribution of the sampled prices should roughly match the distribution of the true prices (perhaps similarity score of the two distributions)

...

Currently I have no further ideas, but I should be able to prove that the predicted distributions really correspond to reality.
Do you have some good ideas for metrics/scores etc to show that the predicted distributions really follow the real pattern of market prices?


texts: difference between feature interactions and confounding variables

texts:
Let me define the problem space.

I am working a binary classification problem. I am trying to build a causal model as well as predictive model.

My aim is to find list of significant features (based on causal model) and use that to build a predictive model. I did refer the suggestions provided in this post and it was very much useful but I have few more questions due to my limitations with ML field.

I understood from literature that there are two ways to adjust/control for confounders. one is through study design phase and other is during modelling/analysis phase.

As I am working on retrospective data analysis, I can only adjust confounders during analysis phase.

We know that certain features like Age in a typical example like "gender causes heart disease" is a confounder.

1) So during analysis phase, we include age as a variable in our model. Similarly all the potential confounders that we could think of are put in the model as features. ex: X_train will have all the columns/features that I think of as potential confounders and then it is fed to the model (logistic regression). Am I right till here?

2) Does this mean our LR model is adjusted for confounders? How would you do confounding adjustment during logistic regression modelling phase? If we include all potential confounders in our model and if the coeff of already existing variables (gender) change by 10% or so, I understand that age is a confounder but does this also mean that LR is adjusted for confounders?

3) Then, why it is said that logistic regression doesn't consider feature interaction? Is feature interaction different from confounding? I understand that feature interaction is usually denoted as gender*age but does this mean both variables work together to influence the outcome? doesn't confounder mean the same?

4) What's the usefulness of having interaction variables? I mean if gender*age impacts the outcome, can I understand that gender (individually) and age(individually) impact the outcome?

5) I see that people usually create 2x2 tables called as strata for stratified analysis and compute risk ratio and compare it with crude risk ratio. But how can we do this for all variables which I think as confounders in my dataset? I know we can use tools like SPSS, STATA etc but is it the only way to do? But then can't we do using multivariate regression?

6) Is it mandatory that all our continuous variables be converted into some categorical variable for analysis/confounder adjustment?

7) Any simple examples/explanation would be helpful as I couldn't find any tutorial for adjusting confounding during logistic regression and finding significant variables. I have been referring this though it's useful, some links are broken. lot of questions arise because I am neither a stats or biostats person. I usually build models using classic ML algorithms and now trying to learn all this.


texts: How to use fminunc in Matlab on only some specified parameters of a function

texts: So I want to use fminunc(fun, X0) in Matlab on my function fun. The problem is that fun is a function of three parameters:(X, y, X0), all of which are vectors. What I want to do is to find the minimum, given the parameters X, y. I.e. the vector X0 which minimises the function for some given values X, y. How would I go about doing this?


texts:
I'm new in Machine Learning and of the first concept I would like to learn is linear regression. I read that to apply linear regression I need to use a linear model. Starting from this assumption I know that this is a simple model for linear regression :

y = w0 + w1x

The definition of linear regression says that the dependent variable y should be a linear combination of the parameters w (but it is not necessary the same for the independent variable x )

So we can say that also this is a linear regression model :

y = w0 + w1x1 + w2(x2)^2

Also in this case, I should say that this is a linear regression model because for the definition, w0 , w1 and w2 are still linear in the expression. Even if there is a quadratic term for the independent variable x2.

Now , I have this question.
A model like the following :

y = w1 x1 + w2 x2 + w3 x3 + w4 x1x2 + w5 (x2^3)

Is it still a linear model ? My first answer is yes , because for the definition the parameter terms are linear , but I'm not sure of it.
Does anyone got any hint ?


texts: So I'm working on a document processing AI and I already have a character recognition model which performs decently well. Now the problem is, how do I feed each character to the model in order to make predictions. Sliding window is one technique to segment arbitrary sequences of the image but it might not work in many cases as the words may get cut off. Can anyone suggest a robust way to detect text regions in an image? I've been researching about MSER but they claim to work well on grayscale images and not RGB. I know that we can convert RGB to grayscale but that is not performant enough.


texts: Split a list of values into columns of a dataframe?

texts:
I am new to python and stuck at a particular problem involving dataframes.

The image has a sample column, however the data is not consistent. There are also some floats and NAN. I need these to be split across columns. That is each unique value becomes a column in the df.

Any insights?


texts: What are Laplacian scores and how do they effect feature selection?

texts: I was going through a paper related to feature selection wherein I constantly came across the term Laplacian Scores which I was not able to understand. Can anyone explain their importance in feature selection?


texts:
Aurelion Geron's example shows how to stored selected operations in a collection, so they can then be easily accessed later:

Adding:

for op in (X, y, accuracy, training_op):
    tf.add_to_collection("my_important_ops", op)

Then retrieving:

X, y, accuracy, training_op = tf.get_collection("my_important_ops")

However when I try to do this I get this error:

X, y, accuracy, training_op = tf.get_collection("my_important_ops")
ValueError: too many values to unpack (expected 4)

What am I doing wrong?


texts: How to improve loss and avoid overfitting

texts:
I'm trying to build a 2 class image classifier using the architecture suggested in first part of this blog https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

My dataset has 1500 images of class1 and 500 images of class2. I created 4 copies of class2 to make the number of images same in both classes.

I also use ImageDataGenerator to augment images

datagen = ImageDataGenerator(
        featurewise_center=True,
        featurewise_std_normalization=True,
        shear_range=0.2,
        zoom_range=0.5,
        rotation_range=45,
        horizontal_flip=True,
        vertical_flip=True)
datagen.fit(train_imgs)

I'm using AdaDelta,RMSProp, SGD, Adam, AdaGrad optimizers and tried adding/removing Conv2D and Dense layers. Tried BatchNormalizationa and Dropout. The results are coming out almost same:

For first few epochs(about 20) training and validation errors keep reducing until log loss reaches about 0.4 (best I have got till now) after that the model starts to overfit and validation loss keeps increasing.

I know I can prevent overfitting by reducing the network complexity and adding dropouts but that reduces the training accuracy too.

Please suggest some tips to improve the accuracy and avoid overfitting.


texts: Handling missing data - secondary driver characteristics in insurance data

texts: I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary drivers, so when preprocessing the data how should I handle the secondary driver's age/sex - deleting those observations is not an option because about 30% of policies have secondary drivers. Should I encode the secondary driver's age as zero?


texts: How do multiple linear neurons together allow for nonlinearity in a neural network?

texts:
As I understand it, the point of architecting multiple layers in a neural network is so that you can have non-linearity represented in your deep network.

For example, this answer says:  "To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

When I watch online tutorials and whatnot, I see networks described as in the screenshot below. In cases like this, I see a series of linear classifiers:

We have a multiply, add, ReLu, multiply and add, all in series.

From studying math, I know that a composite function made out of linear functions is itself linear.

So how do you coax non-linearity out of multiple linear functions?


texts: Is there any possibility to apply deep dreaming in data augmentation?

texts: I looked into the deep dreaming concepts and feel like this has the potential for data generation. But i'm not sure how possible this concept is. Any thoughts regarding this?


texts: Neural Network Prediction regression task, output is a multiple factor of input with same peaks

texts:
When I missed some details please point this out. I made a simple sequential LSTM model for regression. The model loss is 3.2145e-06. The data is scaled between 0 and 1. I tried different variations of layers etc but the same happend, the simpliest model I used is the following one:

from keras.models import Sequential
from keras.layers import LSTM
model = Sequential()
model.add(LSTM(16,return_sequences=True,input_shape=(None,1))
model.add(LSTM(1,activation='sigmoid', return_sequences=True))
model.compile(loss='mae', optimizer='adam', metrics=['mae','mse'])
history=model.fit(train,train, epochs=10, batch_size=78, shuffle=False)

predicted=model.predict(test)

My train data looks like:

When I compare the test and predicted, predicted looks likes being many times smaller than the test data and showing the exact peak values. In my case I expect the values to be small because I trained with totally different data (kind of anomaly regression), what is surprising are the peak values being identical. testdata plot above and predicted below. So my question why this with the peaks are happening and how can I make the network not be able to detect the peaks and predict as trained ?

Edit:
After using MAE as loss function and 500 epochs training I reached 0.0180 loss. The prediction looks like mirrored of the mse prediction (see below). After 2000 epochs I reached a minium of 0.0030 and th

After 2000 epochs I reached a minium of 0.0030 and the prediction looks like this:

Edit: my data is on supposed to work with LSTM, I needed an other model.


texts:
I would like to find how to change my data structure to make it compatible with a ML model, with the following constraints :

A Product is defined by {id, price, continuousVariable1, ..., continuousVariable10}
The training set is train.csv. Each line contains a date, a Product, and a list of Products (competing products)
For each row of train.csv, the size of the list of Products may change.

It would be easy if I had only one competing products : I could put all columns in the same file. But I have many competing products.
The output variable is the product price.


texts: Looking for example infrastructure stacks/workflows/pipelines

texts:
I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used for different types, but I'd like to get to know more about their interaction in applications, e.g. thinking machine learning for an app, webapp, online shop.

I have vistors/session, transaction data etc and store that; but if I want to make recommendations on the fly, I can't run slow map/reduce jobs for that on some big database of logs I have. Where can I learn more about the infrastructure aspects? I think I can use most of the tools on their own, but plugging them into each other seems to be an art of its own.

Are there any public examples/use cases etc available? I understand that the individual pipelines strongly depend on the use case and the user, but just examples will probably be very useful to me.


texts:
Can someone explain RMSprop in layman's terms? I have tried reading various resources but they don't talk about the intuition on couple of things:

1) Why is is different from SGD with momentum?
2) Why does it actually work?

The only literature I could find were the slides of the professor who wrote this optimization technique and the slides didn't make a lot of sense to me.

Thanks


texts: Machine Learning algorithms and Panel data

texts:
I have a large panel dataset composed of $N$ stocks, $T$ quarterly dates and $K$ features for each stock. The dataset looks like the following:
            symbol  stockPriceD numberOfShares  marketCapitalization   ...    label
2002-06-01  A       -4.91       1000000.0       -2.254640e+09          ...    0
2002-09-01  A       -9.08       1000000.0       -4.203510e+09          ...    1
2002-12-01  A       4.27        0.0             1.985550e+09           ...    1
...
2009-06-01  BA      3.19        732600000.0     3.167762e+10           ...    1
...
2019-12-01  ZTS     10.43       -700000.0       4.896220e+09           ...    0
2020-03-01  ZTS     -8.72       -2400000.0      -4.478504e+09          ...    0

I would like to do a forecast task on this dataset but I cannot assume independence among the features (almost all are autocorrelated), and splitting the dataset into $N$ different ones for each stock will leave me with very small datasets (max 72 instances/rows).
How can I handle this problem? Am I allowed to assume independence among the instances in any case forgetting about the autocorrelation? Are there Machine Learning algorithms that can handle these kinds of problems (panel data)?
I read about using RNN and LSTM algorithms to address these issues, but how the data should be treated?


texts: Plotting two pandas data frame columns against each other

texts:
I have a data frame like this:
timestamp                  vote_count
00:12:30                   350
05:05:00                   50
01:10:35                   200
05:05:00                   175
..............................
..............................
00:12:30                   900

Now I want to visualize the vote_count for the timestamps and do some analysis on that further. How can I plot the two columns against each other using matplotlib or seaborn?
Note: The timestamp is in 24hr format. Also, at any timestamp, there can be multiple vote counts.


texts: Techniques for Cluster Analysis of a Very Large (n=140000) Binary Dataset in Python?

texts:
In essence: what techniques in Python are possible to find clusters/trends in a very large categorical dataset?

My very large dataset (140000 rows/observations, 80 variables) of categorical data has been re-coded using one-hot encoding, so they are all binary (e.g ethnicity_black, ethnicity_asian). This dataset is for instances of police use of force in the UK.

I was planning on doing agglomerative hierarchical clustering on it to find patterns (via cluster description) in use-of-force events, but I cannot accomplish this as the distance matrices are always too large and keep crashing. I was using Gower's distance since they are all dummy variables, and the gower package.

I have tried doing dimensionality reduction via MCA (like PCA but for categorical variables) but that simply reduces the number of columns, and the distance matrix is still too huge (140000x140000 requires 72.7Gigabytes of RAM).

I then tried reducing to the first 10000 rows, but that takes up 10000x10000=100000000 (100 million cells), and exceeded the recursion depth limit when constructing the dendrogram. Not sure if it is appropriate to simply take a tiny random subset of the 1400000 rows? Since that might be unrepresentative in terms of revealing patterns.

As such, I'm looking for other clustering methods besides hierarchical clustering, since K-means only works for continuous data, as does DBSCAN, since they use Euclidean distances. Was wondering if you guys have encountered similar situations before in your work, and could share about other techniques or possible workarounds in this case?

tldr; K-Means and DBSCAN seem to be for continuous data, and hierarchical clustering isn't very scalable to a large dataset due to the dissimilarity matrix required. I'm exploring K-modes but not sure how to interpret the output as well...

I did this MCA - Hierarchical Clustering technique before but in R, following the methodology used in this paper closely: https://erf.org.eg/wp-content/uploads/2019/11/Fateh-Belaid_Manuscript_FB_10-09-2019_FB.pdf

However, even that workflow doesn't work for my current dataset as it's simply too big.

Any advice would be appreciated...

Thank you very much!


texts: Suggestions for good data science certified online courses

texts:
I am wondering if someone here can recommend some good data science certified online courses.

A simple google research brings hundreds of online academy and I am lost.

It is been one year since I am working in this field and I want to validate my knowledge and acquire more.


texts: how to do regression and classification in a same time?

texts:
I was thinking of doing a job including regression firstly, then doing the classification. I read lots of sources that are saying it is Semi-Supervised learning while I think it is not.
Imagine after doing regression we will reach to the graph below:

we know that there are 2 classes and now the goal is for prediction of the possibility of locating in class 1 or 2. It means that, for the next data that we have, we need to make a classification.


texts: Converting wave to vector and vice versa creates noise

texts:
I’m a long time data scientist that works mainly with texts. I’m now trying to apply my knowledge into voice and I’m struggling with some simple tasks in python.
As far as I understood I can manipulate voice (converting voice to vectors for neural networks) using mfcc of the voice file.

However, when I try to convert it back to voice I got tons of noise.
My pseudo code:

class wav2vec:
    mfcc = None
    org_shape = None
    wave_shape =  None
    sr = None

max_len=50
feature_dim_1 = 20
channel = 1
feature_dim_2 = 50
filename = 't1.wav'

def wav2mfcc(file_path, max_len):
    wave, sr = librosa.load(file_path, sr=None)
    mfcc = librosa.feature.mfcc(wave, sr=16000)
    org_shape =  mfcc.shape

    if (max_len > mfcc.shape[1]):
        pad_width = max_len - mfcc.shape[1]
        mfcc = np.pad(mfcc, pad_width=((0, 0), (0, pad_width)), mode='constant')
    else:
        mfcc = mfcc[:, :max_len]

    objWav2vec = wav2vec()
    objWav2vec.mfcc = copy.deepcopy(mfcc)
    objWav2vec.result = copy.deepcopy(mfcc)
    objWav2vec.org_shape = org_shape
    objWav2vec.wave_shape = wave.shape
    objWav2vec.sr = sr
    return objWav2vec

def mfcc2wav(objWav2vec, file_path, max_len):
    mfcc = objWav2vec.mfcc.reshape(objWav2vec.org_shape[0],max_len)
    mfcc = mfcc[:, :objWav2vec.org_shape[1]]
    n_mfcc = mfcc.shape[0]
    n_mel = 128
    dctm = librosa.filters.dct(n_mfcc, n_mel)
    n_fft = 2048
    mel_basis = librosa.filters.mel(objWav2vec.sr, n_fft)

    bin_scaling = 1.0/np.maximum(0.0005, np.sum(np.dot(mel_basis.T, mel_basis),axis=0))

    recon_stft = bin_scaling[:, np.newaxis] * np.dot(mel_basis.T,invlogamplitude(np.dot(dctm.T, mfcc)))

    excitation = np.random.randn(objWav2vec.wave_shape[0])
    E = librosa.stft(excitation)
    recon = librosa.istft(E/np.abs(E)*np.sqrt(recon_stft))

    librosa.output.write_wav(file_path, recon, objWav2vec.sr)

t = wav2mfcc(filename,max_len)
t.mfcc = t.mfcc.reshape(1,feature_dim_1, feature_dim_2, channel)
mfcc2wav(t, 't2.wav', max_len)

Can anyone explain why I don’t get the same sound as the original file?


texts: How to apply SelectKBest in azure machine learning

texts: I am very new to machine learning, I started to understand fundamental and using it in azure studio, I see that there is an algorithm in scikit for feature selection called SelectKBest, then I searched azure studio and I coudn't find it by name, I was wondering if is it supported there?


texts: Product classification in hierarchical categories based on multiple parameters and non-standard descriptions

texts:
I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the description on the purchase order is free text and people can write just about anything (often the bare minimum for the order to be approved)

Some importan considerations are:

As the company is global, the language of the description is local (german, spanish, english, russian, chinesse, etc)
There is no standard for the structure of the sentences
People can input abbreviations
For certain products, the buyers don't define if the part is a repair part or a brand new product

I have read about sklearn libraries for text analysis but I really new at machine learning algorithms (I have mostly completed basic tutorials). Most of the examples are for analyzing tweets or complains, but I don't know where to start if I want to consider variables other than text, such as quantity, unitary price, provider and other parameters; and additionally, my categories are a hierarchy of four levels based on certain rules (such as, if the unitary price is over X, then it belongs to category A)

An example of the database could be:

DESCRIPTION                             uom    Mgroup   Provider     Category   Unit    Spend     Total Quantity
CATALOG: A6-CJR-45 XRE:    N/A C-RING   FT     A        Prov1        31000000   5.1     $5.10     1
Contactor iec, 9a, 24v dc, 50/60hz (    FT3    B        Prov2        32131000   82      $164.00   2
Ducto ranurado de 2 x 3, color gris,    BAL    C        Prov1        32131000   24.34   $486.80   20
Modulo de 8 salidas aisladas 5-235 vc   ST     B        Prov2        32131000   254.74  $764.22   3
Selector no-il. plastico 2 pos. mant.   ST     B        Prov3        32131000   6.46    $32.30    5
(ELEC-L2GEW4) TERMINAL TIPO LENGÜETA P SER    D        Prov2        39120000   3.77    $56.55    15
2 Position Selector Switch - Plastic,   M      E        Prov2        23161500   9.89    $69.23    7

So my question is, where could I start to investigate? Which algorithm would be best suited to tackle this problem?

Thank you!


texts: How can I map line spatial data in Power BI without ArcGIS?

texts:
I'm looking for a way to map line shapefiles in PowerBI. The native maps only handle polygons or points and have limited functionality. The ArcGIS for PowerBI tool requires a subscription that is far too expensive. I am willing to use Python but an example script would help.
Ideally, I would import a shapefile from a file or an API, then be able to use it to select other features on the map or conduct geoprocessing tasks like buffering.
Any tips?


texts: Is there a straightforward way to run pandas.DataFrame.isin in parallel?

texts:
I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of individual users for each of a few thousand specific pages. This is the most time-consuming part of the program, more so than the modeling or scoring pieces, simply because it only runs on one core while the rest runs on a few dozen simultaneously.

Though I know I could manually break up the dataframe into chunks and run the operation in parallel, is there any straightforward way to do that automatically? In other words, is there any kind of package out there that will recognize I'm running an easily-delegated operation and automatically distribute it? Perhaps that's asking for too much, but I've been surprised enough in the past by what's already available in Python, so I figure it's worth asking.

Any other suggestions about how this might be accomplished (even if not by some magic unicorn package!) would also be appreciated. Mainly, just trying to find a way to shave off 15-20 minutes per run without spending an equal amount of time coding the solution.


texts: Access keys of pandas dataframe when using groupby

texts:
I have the following database:

And I would like to know how many times a combination of BirthDate and Zipcode is repeated throughout the data table:

Now, my question is: How can I access the keys of this output? For instance, how can I get Birthdate=2000101 ZipCode=8002, for i = 0?
The problem is that this is a 'Series' object, so I'm not able to use .columns or .loc here.


texts: How to retrieve Twitter username with UUID

texts:
I am trying to analyze FakeNewsNet which is a fake news dataset for Social Network Analysis.

There are UUID(I am not sure about the entity name)s of users in the dataset but I am not sure how to retrieve metadata from it.

Can someone explain to me where I can find the information on how to use user_id of User.txt?

Is it even possible to retrieve the metadata of a user with this id? (may using Twitter Search API)

Reference: https://github.com/KaiDMML/FakeNewsNet/blob/master/Data/BuzzFeed/User.txt


texts:
I have used apriori to mine roles on a training set and the result is similar to the below

I have been looking around for a kind of validation or measure of such association but it seems pretty vague and there no specific method yet, I come across an opinion which suggests using precision
which is defined as TP / TP + FP
my questions are

is using precision would be a sufficient ?
is it correct to assume that ...

TP = transactions having both A and B
FP = transactions not containing A and B + transcation not containig A + transactions not containing B

would I just calculate the coinefidence on the test set ?


texts:
I have a masters in pure statistics. I have no knowledge of any programming language. Which is the best programming language that I should learn about and will help enhance my career in data analysis.

I only have a theoretical knowledge of concepts. Kindly suggest a skill that I should develop in order to become a data analyst.


texts: Trained BERT models perform unpredictably on test set

texts:
We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label indicates that the token does not belong to any class.
Generally speaking, this works well: loss decreases with each epoch, and we get good enough results. However, if we compute precision, recall and f-score after each epoch on a test set, we see that they oscillate quite a bit. We train for 1,000 epochs. After 100 epochs performance seems to have plateaued. During the last 900 epochs, precision jumps constantly to seemingly random values between 0.677 and 0.709; recall between 0.729 and 0.798. The model does not seem to stabilize.
To mitigate the problem, we already tried the following:

We increase the size of our test data set.
We experimented with different learning rates and batch sizes.
We used different transformer models from the Huggingface library, e.g. RoBERTa, GPT-2 etc.
Nothing of this has helped.

Does anyone have any recommendations on what we could do here? How can we pick the “best model”? Currently, we pick the one that performs best on the test set, but we are unsure about this approach.


texts: Are there some research papers about text-to-set generation?

texts:
I have googled but find no results.

Text-to-(word)set generation or sequence-to-(token)set generation.

For example, input a text and then output the tags for this text:

'Peter is studying English' --> {'good behavior','person','doing something'}

Thank you!


texts: Pandas datetime error when reading from excel file

texts:
I am trying to read an excel file that has two columns using pandas.

This is how the data looks in excel file:

DT                    Values
2019-11-11 10:00      28.9
2019-11-11 10:01      56.25
2019-11-11 10:02      2.45
2019-11-11 10:03      96.3
2019-11-11 10:04      18.4
2019-11-11 10:05      78.9

This is how it looks when I read using pandas:

DT                         Values
2019-11-11 10:00:00.000    28.9
2019-11-11 10:01:00.000    56.25
2019-11-11 10:01:59:995    2.45
2019-11-11 10:02:59:995    96.3
2019-11-11 10:03:59:995    18.4
2019-11-11 10:04:59:995    78.9

I have tried creating a new DateTime column, putting the data in a new excel file, converting the DT column to DateTime format in both pandas and excel. Nothing has worked yet!

Why does this happen?

EDIT - 1

I already tried the following code but forgot to mention the snippet,

df= pd.read_excel('data.xlsx', parse_dates = ['DT'])
df.head()

df['DT'] = pd.to_datetime(df['DT'])


texts:
I have a tif file that shows a map. On white background the border of ditricts are drawn with black, nothing else is on the picture. I want to color the districts according to certain values. Is it possible to do that in R? (since the picture doesn't "know" the name of districts in advance I suppose I have to do some manual work but it's ok for me)

UPDATE: the picture is similar to this one: http://www.geoindex.hu/wp-content/uploads/evk_106_20140101.png


texts: Understanding lag plot ,ACF plot and auto-correlation plots

texts:
I have a data-frame of 2809 rows  which is an unevenly spaced time-series

    date        adate

0   2012-03-30  0.0
1   2012-03-30  1.0
2   2012-03-31  19.0
3   2012-04-19  1.0
4   2012-04-20  1.0

which I interpolated linearly so that i could fit an ARIMA on it

df_rs = t111.set_index('date').resample('D').mean().interpolate('linear').round(1) #resample according to Days and linear inetrpolate and round values

Following are the ACF, auto-correlation and lag plots of now evenly spaced the data.
I need to understand what properties and parameters can be safely inferenced/understood about my data from these plots.

Notice I've made some comments in the pictures as well,are they apropiate?
I a real noob at this, any and all insights are welcome. ELI5? Thanks a bunch :D


texts: How does the validation set get used in the training phase?

texts:
I am confused about how the validation set is used during the training phase (neural network like CNN)?
In a platform like Matlab or python(Keras), I split my dataset into train set, validation set and test set.
I knew that validation set is used to tune hyperparameters(like the number of neurons and learning rate), suppose SDG optimizer is used, how the tuning hyperparameters happened based on validation set?
Does the validation set just give an indicator of how the neural network performs on unseen data(validation set) then based on that I manually set hyperparameters? or something automatically(the optimizer) tune hyperparameters?


texts: Choice of f1 score for highly imbalanced dataset?

texts:
I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation.
Given my dataset is highly imbalanced(600:100000)


texts: Lasso Regression for Feature Importance saying almost every feature is unimportant?

texts:
I have a metric (RevenueSoFar) that is a great predictor of my target FinalRevenue as you'd expect - it is a metric where we tend to get 90-95% of revenue so far on day 1 and then it can increase over the next 6 days. Therefore i'm also using DaysData (1,...,6) as a feature in the training set to attempt to get the model to learn about how confidence and revenue grows as we get closer to the final day.
In my pre-build analysis I have noticed correlations with several other metrics such as visits to our website, DOW of day1 etc. although not as strong as the correlation with RevenueSoFar. I've therefore ran Lasso Regression for feature importance on the features however my model has reduced every feature's coefficient to zero apart from RevenueSoFar (even the DaysData feature).

Does this mean i should remove all of these from my model and only
use the RevenueSoFar as the predictor?
In the training data set there are a large proportion of products that have
zero revenue and zero revenue so far (however the vast majority of these are
for countries with low demand which is separated out as a feature in the
training set). Could the zeros be skewing the model to think that RevenueSoFar
and FinalRevenue mostly matches perfectly and therefore there is no need for
other features?

Below is my code incase anybody is interested.
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Lasso
from sklearn.feature_selection import SelectFromModel

#import data

DF = pd.read_csv("LearningDataSet.csv")

DF_categorical = DF[["DOW","Market",”Country”]]
DF_numerical = DF[["DaysData","D0Visits","D0MobVisits","D0DesVisits","D0Carts","D0MobCarts", "D0DesCarts", "RevenueSoFar"]]
DF_target = DF[["RevenueFinal"]]

#Scale numerical
scaler = StandardScaler()
DF_numerical_scaled = pd.DataFrame(scaler.fit_transform(DF_numerical))

#Dummy Categorical
DF_categorical_dummy = pd.get_dummies(data=DF_categorical, drop_first=True)

#Merge categorical and numerical
DF_final = DF_categorical_dummy.merge(right=DF_numerical_scaled, how='left', left_index=True, right_index=True)
#Split into training and test
X_train, X_test, y_train, y_test = train_test_split(DF_final, DF_target, test_size=0.25, random_state=0)

selection = SelectFromModel(estimator=Lasso()).fit(X_train,y_train)

print(selection.get_support())

selected_feat = X_train.columns[(selection.get_support())]
print(selected_feat)
```


texts: Machine Learning validation data returns 100% accuracy

texts:
I'm Testing a Machine Learning model with validation data returns that return 100% correct answers, is it overfitting or the model works extremely well, do I need to continue training on more data?
I'm not sure how to interpret the result, any guidance please?


texts: Reinforcement learning: negative reward (punish) illegal actions?

texts:
If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state?

I guess over time if you only select from between the legal actions, the illegal ones would eventually drop out, but would punishing them cause them to drop out sooner and possibly cause the agent to explore more possible legal actions sooner?

To expand on this further; say you're training an autonomous vehicle, and the output is drive direction (forward or reverse) and speed. Say for the scenario you're in, the vehicle must drive between a speed range, e.g. 20mph min, 40mph max, what do you do in the scenario where the agent gives an action to drive forward but gives a speed below the minimum speed? Or another example, say you're training to play a game, and the agent proposes an illegal action which it cannot perform.

I can't proceed with the action because it's illegal, so what do I do?  How do I proceed with training in that situation? I will of course enter the min/max speeds as part of the state given to the agent, but how do I prevent it from proposing actions that are illegal, and how do I proceed with training when it does?


texts: Getting error while performing late fusion of audio and video feature vectors. Please help me resolve this

texts:
I am trying to build Multimodal emotion classifier for which I have created CNN based Audio and Video models separately. Below are the implemented CNNs:
    ##Model for audio

input_audio_ = Input(shape = (40, 1,))

output_audio_ = Conv2D(64, 5, activation='relu')(input_audio_)
output_audio_ = Dropout(0.2)(output_audio_)
output_audio_ = Flatten()(output_audio_)
output_audio_ = Dense(8,activation='softmax')(output_audio_)

model_aud = Model(inputs=[input_audio_], outputs=[output_audio_])
model_aud.summary()

    ##Model for Video

input_frame_ = Input(shape=(256,512,3))
output_frame_ = Conv2D(64, 3, padding='same', activation='relu')(input_frame_)
output_frame_ = Flatten()(output_frame_)
output_frame_ = Dense(8, activation='softmax')(output_frame_)
model_img = Model(inputs=[input_frame_], outputs=[output_frame_])
model_img.summary()

However while performing late fusion, I am stuck at resolving the dimensions of the input. I built the fused model:
> by removing the last output layer from audio and video models then
> concatenating the information in penultimate layers of both the models
> added the final output layer afterwards

I was able to compile the model and this is how the structure looks like :

But while fitting the model like this -
base_history = model_main.fit({'input_frame_': img_train, 'input_audio_': X_train_aud},
                             {'output_frame_': img_train_labels, 'output_audio_': y_train_aud},epochs=50,
                               validation_data= ({'input_frame_': img_test, 'input_audio_': X_test_aud},
                             {'output_frame_': img_test_labels, 'output_audio_': y_test_aud}))

I am getting the below error:
ValueError: Input 0 of layer conv1d_5 is incompatible with the layer: expected axis -1 of input
shape to have value 1 but received input with shape [None, 256, 512, 3]

The shapes of the inputs are as below:
X_train_aud.shape, X_test_aud.shape,y_train_aud.shape, y_test_aud.shape

((1715, 40, 1), (736, 40, 1), (1715,), (736,))

img_train.shape, img_test.shape

((1715, 256, 512, 3), (736, 256, 512, 3))

img_train_labels.shape

(1715,)

I am confused how to fuse these feature vectors of audio and video now, to make it ready for processing. Any help in this regard would be highly appreciated. Thanks in Advance!


texts: Mask RCNN detecting object but mask is inaccurate

texts:
I am trying to detect the inner region of a object. Currently I am using the mask rcnn implementation provided by tensorflow in the models zoo.

The object is similar to a hula hoop that is square in shape.

I tried the inception_V2 backbone and have about 3000 images from different angles and lighting conditions.

After about 20,000 steps the model seems to converege to ~0.2 (seems reasonable based on other examples of mask rcnn on the internet) and stay there for as long as I run it. I have tried upto 200,000 steps.

The problem I am having is that while the bounding box is correctly drawn, the mask is inaccurate. Instead of the ground truth mask which has well defined edges, I get wavy edges and overshoots/undershoots. (

My goal is to find the 4 corner points of the inner area and am unable to. Any suggestions? Is there a way to post process these masks?

Is it also possible to find the inner edges maybe from the bounding box? I tried classical methods but could not manage to get the inner area.


texts:
I'm solving a problem of ranking classes for each unique id based on the utilization quantity. I have 6 unique classes in the training and test data. My neural net mode predicts the utilization coressponding to each class. So if there are 10000 test samples, I have 10000X6 prediction array and 10000X6 true value array. I want to validate the model performance using NDCG metrics.
I followed https://www.kaggle.com/davidgasquez/ndcg-scorer to compute NDCG. In there, the shapes for the parameters are as follows: y_true : array, shape = [nsamples] Ground truth (true relevance labels). y_score : array, shape = [nsamples, n_classes] Predicted scores. k : int Rank.

In my case the ytrue is an array of shape [nsamples,nclasses]. I have aggregated the true values for individual classes for each unique id and y_true is now [n_unique_id,1].  y_score is of shape [unique_id,n_classes]. I've used Rank = 6 since there are 6 distinct classes.

ndcg score for the above data [score = ndcg_score(df_true, df_pred_array, k=6)] is 19.6 and I know the score cannot be greater than 1. Am I missing something here?


texts: Unexpected Error Message in RStudio; while using 'twitterR' Package

texts:
I'd install an R package twitterR in my RStudio for accessing Twitter data for a particular handle. But whenever I do try to access the Twitter data it's showing me an unexpected message:

In doRppAPICall(“search/tweets”, n, params = params, retryOnRateLimit = retryOnRateLimit, :
100 tweets were requested but the API can only return 0

Any thoughts on that?

I tried different parameters, handles and functionalities but still getting the same issue.

Here's my code for the particular functionality--

my_tweets <- searchTwitter('@jabhij', geocode='20.593684, 78.96288, 2000km', n = 100)
my_tweets_df <- do.call("rbind", laaply(my_tweets, as.data.frame))
View(my_tweets_df)

Any thoughts on that will be very helpful.
Cheers! :)


texts: Train/ Test split on small dataset along with SMOTE

texts:
I have a binary classification imbalanced dataset with 1000 samples ( 15% of class 1, 85% of the rest). My main goal is to build a robust classifier using the following approach. Wanted to know if there are potential downsides for the same when using a Tree classifier or if there is another way to tackle the same:

Do a 40/60 split of the data, use SMOTE to increase the Training dataset, and train the model
Test it out on the 60% test data( the main reason is that we are trying to make recommendations and larger sample size on the test data will invoke more confidence)

There is not a way to collect more samples for the experiment here. Is there another approach that could solve the issue of low sample on the test data. We just want to be sure the results are robust enough on the bigger test dataset.
Thank you


texts: GridSearch mean_test_score vs mean_train_score

texts:
I am working with scikit learn and GridSearch in order to find the best parameters in my classifiers.

I have a map of different hyperparameters and I want to print out GridSearch results, but I do not understand one thing - what is the difference between mean_test_score and mean_train_score?

As I understand, GridSearch performs cross-validation in order to find the best classifier, but how do these 2 params differ from one another? I always thought that cross-validation gives only one mean, which is a mean of the performance from trained models using N subsets of given data. For example, if I perform a cross-validation with X subsets, I will have X different accuracy scores and then I will have only one mean value.

So, how can I interpret these 2 parameters and what is the difference between them?


texts: Why is predicted rainfall by LSTM coming negative for some data points?

texts:
I have used supervised learning with LSTM network using tanh activation function and 0.1 dropout for time series prediction.my loss='mean_squared_error', optimizer='adam'. The predicted time series is shown below where x axis shows future months and y axis shows rainfall in mm.Orange line is predicted and blue line is actual. Some points are being predicted below 0 (negative) even though training dataset has all points having values >=0. My training dataset is of 64 years (all positive data)  and I am predicting for 12 years (144 data points shown)

Model code using ReLU activation:

model = Sequential()
# LSTM model
model.add(LSTM(128, batch_input_shape=(batch_size, look_back, 1),activation='relu', stateful=True, return_sequences=False))
model.add(Dropout(0.1)) #for better regularization
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')


texts: How do I implement Dual-encoder model in Pytorch?

texts:
I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model.

Here the encoder and decoder share the same weights but I am unable to put it in code. Any links ?


texts:
How do we use a correlation score between two variables for analysing data?

I have a set of 20 features and need to predict 21st feature. Now is it necessary that correlation between any two features should be close to 1 ? If I have 2 features with corr score close to -1, then does this mean that they are contradicting and thereby decreasing the accuracy ?

So how do we use a correlation score in analysis ?


texts: PCA for dimensionality reduction with simultaneous clustering

texts:
so, let's say I have a set of 3D points. Let's say these points lie more or less on a plane that is embedded in the 3d space, then I can use PCA to 'compress' these 3D points to 2D coordinates on that plane, such that they still aproximate the original data well.
let's say half of the 3d points don't lie close to that plane, but instead close to some other plane.
If I just do PCA and reduce to 2 dimensions, I won't get a good aproximation.
If the algorithm however would 'see' that some of the 3d points compress well onto one plane, and others compress well on another plane and label each point and do PCA separately for each set (and compress them to points with 2 coordinates plus one bit that says which set it belongs to) it will aproximate the original data much better.
What's the name for such a PCA algorithm that is also capable of splitting the input data into maximally N sets (probably with some penalty on the number of sets), such that for each set dimensionality reduction yields a much better fitting than if all data points would be reduced together?
// Edit:
adding an example. If one would only cluster by distance in the high-dim space one would arrive at the bad clustering where there are more clusters and each cluster would have a higher error when projected down.
the good example uses fewer clusters and they project better on their 2 dimensional sub-spaces (the green cluster being able to even compress to a 1D space)


texts: Classification affected by a lot of outliers in features? How do you deal with outliers?

texts:
I  am working on a classification problem and I found my data having  a lot of outliers which has resulted in reduction in my recognition rate. I have tried rescaling, normalization techniques like min max, box cox and even log transformation.
I am considering of eliminating outliers from box plots but I am afraid I might be eliminating useful features/data required to define the model.

Are there any suggestion on how to deal with such cases.
Also further analysis of data revealed that my data constitutes of features belong to dfferent process like web application, apps. I segregated the data based on the processes and I  do see that large variation of process  resulted in different accuracy ranging from 60-95%

Any tips on how to deal with such cases? In the end I want my classifier to classify irrespective of the process type. So with my current issue, does this imply that my features defined are not good enough or is there something else I can do?


texts: Suggest text classifier training datasets

texts:
Which freely available datasets can I use to train a text classifier?

We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified our content based on a predefined bag of words we can recommend to him engaging content by getting his feedback on random number of posts already classified before.

We can use this info to recommend for him pulses labeled with those classes. But we found If we used a predefined bag of words not related to our content the feature vector will be full of zeros, also categories may be not relevant to our content. so for those reasons we tried another solution that will be clustering our content not classifying it.

Thanks :)


texts: Why is my CNN model not learning anything? (tensorflow)

texts:
I've been trying to make an image classifier base on this tutorial

I have changed a few things in it, which I didn't think would completely break the learning pattern but they did... The accuracy calculations are consistently about as good as random guesses, with no upward curve, not even a slow one.

As you could see in the code, the only major changes I made was using my own data-set instead of the MNIST data-set, and my data is pictures of characters of 89 classes, not just 10. I apply random noise and distortions before batching them to the trainer, and I have about 100,000 examples of each of the 89 classes.

I have commented next to the only changed lines to help you read the changes

Code (python):

def model(features, labels, mode):

    #Change from the tutorial - 36*36 images instead of 28*28
    input_layer = tf.reshape(features, [-1, 36, 36, 1])

    conv1 = tf.layers.conv2d(inputs=input_layer, filters=32, kernel_size=[5, 5], padding='same', activation=tf.nn.relu)
    pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)
    conv2 = tf.layers.conv2d(inputs=pool1, filters=64, kernel_size=[5, 5], padding="same", activation=tf.nn.relu)
    pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)

    # Change From the tutorial - 9*9*64 instead of 7*7*64 to match different image size
    pool2_flat = tf.reshape(pool2, [-1, 9 * 9 * 64])

    dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
    dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=mode == learn.ModeKeys.TRAIN)

    # Change from tutorial = 89 classes instead of 10
    logits = tf.layers.dense(inputs=dropout, units=89)
    onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=89)

    loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=logits)
    train_op = tf.contrib.layers.optimize_loss(
            loss=loss,
            global_step=tf.contrib.framework.get_global_step(),
            learning_rate=0.001,
            optimizer='SGD'
    )
    # Generate Predictions
    predictions = {
        "classes": tf.argmax(input=logits, axis=1),
        "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
    }

    return model_fn.ModelFnOps(mode=mode, predictions=predictions, loss=loss, train_op=train_op)

I thought that the network parameters are maybe to small to handle 36*36 pictures and 89 classes, so I tried giving the conv layers more features, or increase the dense layer nodes, but nothing seems to change. The only results in tensorboard I see are of 1%-1.5% accuracy, which fit well with random guesses on 89 classes.

I'm a real newbie in tensorflow and machine learning so please be gentle with me, and tell me please what i'm doing wrong.

Edit: Training code:

def train(batch_size, steps):
    config = get_config()
    config.mode = 'train' # just for the example
    # Generate training data
    if config.mode == learn.ModeKeys.TRAIN:
        x, y = generate(batch_size * steps)
    print(y) # prints correct labels in range 0-89
    logging_hook = tf.train.LoggingTensorHook(tensors={"Probabilities": "softmax_tensor"}, every_n_iter=50)
    char_classifier = learn.Estimator(model_fn=model, model_dir=mydir)

    # Train!
    if config.mode == learn.ModeKeys.TRAIN: #assume this is always true in the example here
        char_classifier.fit(x=x, y=y, batch_size=batch_size, steps=steps, monitors=[logging_hook])
    accuracy = learn.MetricSpec(metric_fn=tf.metrics.accuracy, prediction_key="classes")

    # Evaluate!
    x, y = generate(1000)
    eval_results = char_classifier.evaluate(x=x, y=y, metrics={'accuracy':accuracy})

Also, sample images:


texts: How to prevent a neural network from choosing the 'easiest' solution

texts:
I have a neural network that takes in roughly twelve values, and outputs a singly probability. The issue is that the network appears to be smart enough to realize there is a very significant correlation between three of the values and the output, and doesn't seem to be recognizing any correlation with the remainder of the inputs. In other words, the model has figured out that if the and of three specific inputs are non-zero, there is an ~80% chance the output is 1.

This seems like expected behavior, but I am looking to have the network enhance it's 80% prediction with data from the other nine inputs. Is this possible?

My proposed solution requires a bit more information regarding the nature of the data. The twelve inputs can be categorized into four groups of three variables each. For instance, we have three metrics: purchase cost, warranty length and average maintenance costs. These three metrics are then applied to four different appliances in a household: the boiler, washing machine, dishwasher and refrigerator. The network is then supposed to output the probability of the owner of the household moving out in a few months. In this example, there happens to be a 80% correlation between refrigerators and the homeowner moving: if they own a refrigerator, there is an 80% chance they will move out within the next few months.

As mentioned before, the network is smart enough to recognize the correlation between the refrigerator and the moving out, but I am hoping to get it to augment this prediction with the inclusion of the other inputs.

My first strategy for addressing this would be to split the single network into four separate networks, each handling one appliance. These will output four probabilities total, which I can then either weight and compute by hand, or feed into a fifth network for final processing. My worry is that this would result in the same issue occurring - the final network would determine that ignoring the first three inputs and only returning the fridge-based probability is the best way to minimize error.

My second possible solution is to simply up the amount of iterations over the dataset, in the hope that the network happens to locate a slightly better solution by chance, then works to optimize it. This seems unlikely, and overly dependent on the problem.

My third and final possible solution is to initially only train the network on input where the homeowner doesn't own a fridge (all fridge values are zero) such that the network optimizes it's solution without having to worry about the fridges, and then gradually introduce inputs from homeowners with fridges.

Will any of these work? Is there a better solution?

I am using the Tensorflow Estimator API.

Thanks!


texts: Strategy for dealing with giant sample size

texts:
There's a not well-known fact in statistics. That as your sample size increases, more p-values become significant.

I'm working with a massive sample size consisting of 3 million samples (~ 1% of the U.S. population). It's a Logistic Regression, and p-values of several relations are significant at 5%, 1% and ~0%.

I fielded my paper's abstract to several journals to see if there was interest. Only a German journal I submitted to (those smart Germans, really) caught on that an extremely large sample size like this might more easily produce significant p-values and that I'd need to somehow adjust for that.

To be clear, an excessively large sample size doesn't produce spurious p-values. It's that more effects with tiny effect sizes start showing significant p-values.

I want your views on strategies for how to deal with such a situation, to produce reliable research. Some strategies I've thought of are:

Conduct the research as is, and only consider as relevant results that have effect sizes that are above a certain threshold (What threshold though?)
Do what they do in Data Mining. Split up my data into train, validate, and test sets. (Is this done with Logistic Regression though? Can anyone point me to past papers where the technique is illustrated?)
Skip the usual Logistic Regression and instead use a Data Mining technique such as CART (which may also use Logistic Regression internally).

I'd like your views on the relative worth of these strategies. Other suggestions, too, would be welcome. Especially welcome will be pointers to prior papers where the authors illustrate how to deal with a similar problem.


texts: Incremental learning for Gradient Boosting Classifier & Random Forest Clssifier

texts:
I have been trying to get a method to incrementally learn additional data which my model for both random forest and gbm to consume as and when we get it. I can of course retrain the model everytime I get new data, but it will take too much time.
Basis my current resesarch:

warm_start: skitlearn provides us with a hyperparameter called warm_start, however, it cannot be used to incrementally learn with new data. This will simply add new esitimators and give me the result for old+new estimators. I would want more weight to the new data as that is the latest one and trend might have changed. Can someone confirm if my understanding is incorrect and I can simply use this? while experimenting, I did not get great results with this (sometimes accuracy was better, but most of the times this was worse off)

partial_fit: this is more like running batch process. Not incremental learning.

This is the closest post I have found
Incremental Learning with sklearn: warm_start, partial_fit(), fit()
Do let me know in case more details are req. Code is pretty straight forward:
rf_model=RandomForestClassifier(n_estimators=300)
gbm_model=GradientBoostingClassifier()

def train_rf(train_data,y):
    global rf_model
    rf_model.fit(train_data,y)

def train_gbm(train_data,y):
    global gbm_model
    gbm_model.fit(train_data,y)

As of now, I call the above function whenever new data arrives. I have hardcoded so that only past 500 entries are taken (trend changes with time). Any Idea how I can implement incremental learning to this?


texts: What is the formula to calculate the precision, recall, f-measure with macro, micro, none for multi-label classification in sklearn metrics?

texts:
I am working in the problem of multi-label classification tasks. But I would not able to understand the formula for calculating the precision, recall, and f-measure with macro, micro, and none. Moreover, I understood the formula to calculate these metrics for samples. Even, I am also familiar with the example-based, label-based, and rank-based metrics.

For instance,

import numpy as np
from sklearn.metrics import hamming_loss, accuracy_score, precision_score, recall_score, f1_score
from sklearn.metrics import multilabel_confusion_matrix
y_true = np.array([[0, 1, 1 ],
                   [1, 0, 1 ],
                   [1, 0, 0 ],
                   [1, 1, 1 ]])

y_pred = np.array([[0, 1, 1],
                   [0, 1, 0],
                   [1, 0, 0],
                   [1, 1, 1]])

conf_mat=multilabel_confusion_matrix(y_true, y_pred)
print("Confusion_matrix_Train\n", conf_mat)

Confusion matrix output:

 [[[1 0]
  [1 2]]

 [[1 1]
  [0 2]]

 [[1 0]
  [1 2]]]

Macro score

print("precision_score:", precision_score(y_true, y_pred, average='macro'))
print("recall_score:", recall_score(y_true, y_pred, average='macro'))
print("f1_score:", f1_score(y_true, y_pred, average='macro'))

Macro score output:

precision_score: 0.8888888888888888
recall_score: 0.7777777777777777
f1_score: 0.8000000000000002

Micro score

print("precision_score:", precision_score(y_true, y_pred, average='micro'))
print("recall_score:", recall_score(y_true, y_pred, average='micro'))
print("f1_score:", f1_score(y_true, y_pred, average='micro'))

Micro score output:

precision_score: 0.8571428571428571
recall_score: 0.75
f1_score: 0.7999999999999999

Weighted score

print("precision_score:", precision_score(y_true, y_pred, average='weighted'))
print("recall_score:", recall_score(y_true, y_pred, average='weighted'))
print("f1_score:", f1_score(y_true, y_pred, average='weighted'))

Weighted score output:

precision_score: 0.9166666666666666
recall_score: 0.75
f1_score: 0.8

Samples score

print("precision_score:", precision_score(y_true, y_pred, average='samples'))
print("recall_score:", recall_score(y_true, y_pred, average='samples'))
print("f1_score:", f1_score(y_true, y_pred, average='samples'))

Samples score output:

precision_score: 0.75
recall_score: 0.75
f1_score: 0.75

None score

print("precision_score:", precision_score(y_true, y_pred, average=None))
print("recall_score:", recall_score(y_true, y_pred, average=None))
print("f1_score:", f1_score(y_true, y_pred, average=None))

None score output:

precision_score: [1.         0.66666667 1.        ]
recall_score: [0.66666667 1.         0.66666667]
f1_score: [0.8 0.8 0.8]

Thanks in advance for your help.


texts: Why in this case are gradient steps not perpendicular to contour lines?

texts:
There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point.
Why in this picture it seems that this rule is not respected?

source: http://www.deeplearningbook.org/contents/numerical.html page 89

And also in this case:

source: http://www.deeplearningbook.org/contents/optimization.html page 310

In second case it is even weirder because the steps don't seem to be perpendicular to contour lines but to previous gradient vector? Am I missing something or treat images to serious? Thank in advance and regards.


texts:
Imagine, there is a service, providing credit history for customers in form of list of his loans. Let's call it my-loan-service. For the sake of simplicity - I can GET http://my-loan-service/42 (where 42 is my customer id) and get back json
{
   "loans": [
      {
         "loanId": 1,
         "creditLimit": 1000
      },
      {
         "loanId": 2,
         "creditLimit": 2000
      }
   ]
}

Also, this service provide it's data for data analysis to some OLAP data storage. Shortly - we have a table

customerId
loanId
creditLimit

42
1
1000

42
2
2000

There is also a data scientist - Bob - working with this data storage. And his task is to implement and serve a model, deciding to accept or reject new loan applications from customers based on data from my-loan-service.
So far so good, Bob came up with 2 features - number of loans and total credit limit. Bob used plain numpy or pandas to transform raw data into his features and some popular framework (sklearn, tensorflow or any other) to train and test the model. Now, he can serialize his model and serve it using favorite model serving solution.
And here is the culmination followed by the question:
Any model serving instruments/frameworks or SaaS/PaaS solutions I looked into expects to get features as an input. But in our case I can't get features from my-loan-service - only raw-like data. Obviously, I can create another service - feature-calculation-service - just to get loans, count them, sum up credit limits and then pass these features to served model. This way, every time someone wants a new feature, I'll be forced to duplicate logic of its engineering in feature-calculation-service. And python currently not used for production web-services - only for data analysis.
So I am asking for any tips, which can help me avoid duplicating logic of feature engineering for model serving.


texts:
My training accuracy was better than my test accuracy, hence I thought my model was over-fitted and tried Cross-validation. The model further degraded. Is that my input data need to be sanitised further and of better quality?

Please share your thoughts what could be getting wrong here.

My data distribution:

Code snippets...

My function get_score:

def get_score(model, X_train, X_test, y_train, y_test):
    model.fit(X_train, y_train.values.ravel())
    pred0 = model.predict(X_test)
    return accuracy_score(y_test, pred0)

Logic:

print('*TRAIN* Accuracy Score => '+str(accuracy_score(y_train, m.predict(X_train)))) # LinearSVC() used
print('*TEST* Accuracy Score => '+str(accuracy_score(y_test, pred))) # LinearSVC() used

print("... Cross Validation begins...")

y0 = pd.DataFrame(y)
y0.reset_index(drop=True, inplace=True)

print(X.shape)
print(y0.shape)

kf = KFold(n_splits = 10)

e = []
for train_index, test_index in kf.split(X):
    X_train, X_test, y_train, y_test = X.iloc[train_index], X.iloc[test_index],y0.iloc[train_index], y0.iloc[test_index]
    print(train_index, test_index)
    e.append(get_score(LinearSVC(random_state=777),X_train, X_test, y_train, y_test))

print("Finally :: "+str(np.mean(e)))

Output:

*TRAIN* Accuracy Score => 0.9451327433628318
*TEST* Accuracy Score => 0.6597345132743363
... Cross Validation begins...
(9040, 6458)
(9040, 1)
[ 904  905  906 ... 9037 9038 9039] [  0   1   2 ... 901 902 903]
[   0    1    2 ... 9037 9038 9039] [ 904  905  906 ... 1805 1806 1807]
[   0    1    2 ... 9037 9038 9039] [1808 1809 1810 ... 2709 2710 2711]
[   0    1    2 ... 9037 9038 9039] [2712 2713 2714 ... 3613 3614 3615]
[   0    1    2 ... 9037 9038 9039] [3616 3617 3618 ... 4517 4518 4519]
[   0    1    2 ... 9037 9038 9039] [4520 4521 4522 ... 5421 5422 5423]
[   0    1    2 ... 9037 9038 9039] [5424 5425 5426 ... 6325 6326 6327]
[   0    1    2 ... 9037 9038 9039] [6328 6329 6330 ... 7229 7230 7231]
[   0    1    2 ... 9037 9038 9039] [7232 7233 7234 ... 8133 8134 8135]
[   0    1    2 ... 8133 8134 8135] [8136 8137 8138 ... 9037 9038 9039]
Finally :: 0.32499999999999996
>>>

Edit -1- Adding values of "e"

[0.08075221238938053, 0.413716814159292, 0.05752212389380531, 0.15376106194690264, 0.14712389380530974, 0.4668141592920354, 0.6946902654867256, 0.7112831858407079, 0.33738938053097345, 0.18694690265486727]

Edit -2- Adding shuffle=True parameter to KFold()

Result:

[   0    1    2 ... 9037 9038 9039] [   4    5   10 ... 9007 9011 9024]
[   0    1    2 ... 9037 9038 9039] [  21   43   44 ... 9018 9035 9036]
[   0    2    3 ... 9037 9038 9039] [   1   20   60 ... 9023 9031 9034]
[   0    1    2 ... 9036 9037 9038] [   6   25   27 ... 9010 9025 9039]
[   0    1    2 ... 9037 9038 9039] [  15   16   28 ... 9029 9030 9033]
[   0    1    2 ... 9037 9038 9039] [   3   12   40 ... 9015 9017 9028]
[   0    1    2 ... 9037 9038 9039] [   7    8   23 ... 9013 9014 9027]
[   0    1    3 ... 9035 9036 9039] [   2   18   19 ... 9019 9037 9038]
[   0    1    2 ... 9037 9038 9039] [  24   37   39 ... 9012 9016 9026]
[   1    2    3 ... 9037 9038 9039] [   0    9   14 ... 9020 9021 9032]
[0.6504424778761062, 0.6736725663716814, 0.6969026548672567, 0.6692477876106194, 0.6769911504424779, 0.6382743362831859, 0.6692477876106194, 0.6648230088495575, 0.6648230088495575, 0.6814159292035398]
Finally :: 0.6685840707964601


texts: Examples for multi-input Convolutional Neural Network

texts: I want to create a multi inputs Convolutional Neural Network (cnn) that takes two inputs and produces one output of the inputs class by using Keras. I searched for resources that explain multi inputs cnn, but only found brief explanations. I need examples for multi-inputs cnn for images classification.


texts: How to implement a basic query management and recommendation system

texts:
I'm trying to prototype a system where given a textual query (e.g. a question), I get a list of most relevant documents/questions among a pool of available documents/questions (similar to what we see here in Stack Exchange when submitting a question).

One simple idea coming to my mind is to extract important keywords from query, and do matching based on keywords. Or any other ways, maybe through machine learning or data mining approaches.

I am very new to data science, and I believe this is a very basic system. Is there any available light-weight open source project on this? If not, any hints how to do this?


texts: Why could my SVR prediction be offset, but still follow the actual pattern?

texts:
I am running an SVR prediction on some time series data, and I am receiving this weird offset between my actual and predicted values.

I found this SVM Regression lag post, that mentions adding a lag of 2 data points behind, instead of one.  However, I am not sure how to incorporate that into my code (which I've included below).

Does anyone have any ideas on why my predicted vs. actual is offset in this manner?

My code is as follows:

#! /usr/bin/python

import math
import statistics
import visualizer
import numpy as np
from datagen import constructData
from sklearn import svm

# Applies Support Vector Regression to the electricity dataset,
# prints out the accuracy rate to the terminal and plots
# predictions against actual values
def suppVectorRegress():

    kernelList = ["linear","rbf",polyKernel]
    names = ["linear","radial basis","poly"]
    preds = []

    # Retrieve time series data & apply preprocessing
    data = constructData()

    cutoff = len(data)-30
    xTrain = data[0][0:cutoff]
    yTrain = data[1][0:cutoff]
    xTest = data[0][cutoff:]
    yTest = data[1][cutoff:]

    # Fill in missing values denoted by zeroes as an average of
    # both neighbors
    statistics.estimateMissing(xTrain,0.0)
    statistics.estimateMissing(xTest,0.0)

    # Logarithmically scale the data
    xTrain = [[math.log(y) for y in x] for x in xTrain]
    xTest = [[math.log(y) for y in x] for x in xTest]
    yTrain = [math.log(x) for x in yTrain]

    # Detrend the time series
    indices = np.arange(len(data[1]))
    trainIndices = indices[0:cutoff]
    testIndices = indices[cutoff:]
    detrended,slope,intercept = statistics.detrend(trainIndices,yTrain)
    yTrain = detrended

    for gen in range(len(kernelList)):

        # Use SVR to predict test observations based upon training observations
        pred = svrPredictions(xTrain,yTrain,xTest,kernelList[gen])
        # Add the trend back into the predictions
        trendedPred = statistics.reapplyTrend(testIndices,pred,slope,intercept)
        # Reverse the normalization
        trendedPred = [np.exp(x) for x in trendedPred]
        # Compute the NRMSE
        err = statistics.normRmse(yTest,trendedPred)

        print ("The Normalized Root-Mean Square Error is " + str(err) + " using kernel " + names[gen] + "...")

        preds.append(trendedPred)

    names.append("actual")
    preds.append(yTest)

    # Change the parameters 2017,2,1 based on the month you want to predict.
    visualizer.comparisonPlot(2017,2,1,preds,names,plotName="Support Vector Regression Load Predictions vs. Actual",
        yAxisName="Predicted Kilowatts")

# Construct a support vector machine and get predictions
# for the test set
# Returns a 1-d vector of predictions
def svrPredictions(xTrain,yTrain,xTest,k):
    clf = svm.SVR(C=2.0,kernel=k)
    clf.fit(xTrain,yTrain)
    return clf.predict(xTest)

# A scale invariant kernel (note only conditionally semi-definite)
def polyKernel(x,y):
    return (np.dot(x,y.T)+1.0)**0.95

if __name__=="__main__":
    suppVectorRegress()


texts: Backpropogation "The hardest derivation I have seen in machine learning"?

texts:
In week 3 of Andrew Ng's "Neural Networks and Deep Learning" on Coursera there is a video titled "Backpropogation Intuition" during which he gives the full vectorized backpropogation formula, and ends the lecture suggesting that students who are good at calculus try to derive these from first principles, saying "it is one of the hardest derivations I have seen in the whole machine learning".

Please can someone post a link to a source that gives this derivation, ideally, using the same notation as Andrew Ng:


texts: How to figure out what metrics I can monitor while setting Keras Callbacks?

texts:
I am trying to figure out whether I have a host of metrics available to monitor out of the box while writing callbacks such as
ModelCheckpoint(monitor='val_loss')

or

ModelCheckpoint(monitor='val_accuracy')

or whether the metrics i can monitor only depend on what i initialize in the compile method,
model.compile(metrics=['val_accuracy', 'val_loss'])

vs

model.compile(metrics=['val_loss'])

or

model.compile(metrics=['loss'])

Question: Does initializing the callback come with a host of metrics i can monitor out of the box or do I have to manually set the metrics in the model compilation step, which constrains me to only those that I wrote?
Side Question: How can I find out the whole host of metrics I can type in whether it is in the compile method or not.. because I can't seem to find a source that tells me.


texts: K-means clustering for one class classification

texts: I want to know if I can use the k-means clustering algorithm for a one class classification (as in the case of one class SVM), which means I have data for 2 classes, and I labelled only the one class that I used for training?


texts: Changes in the standard Heatmap plot - symmetric bar colors, show only diagonal values, and column names at x,y axis ticks

texts:
I have a heatmap image (correlation between all matrix columns) and I'm straggling to preform all the changes below within the same image:

bar colors should be symmetric around zero (e.g., correlation of 1 and -1 should be with the same color)
change the correlation matrix to a diagonal matrix, since correlation values are symmetric - and show only upper matrix triangle (mask out the lower triangle )
show the correlation values in every cell of the diagonal matrix
x,y axis ticks - show the column names (instead of a serial number)

This is the code:

def generate_heatmap(X):

    """
    Pearson Correlation Heatmap Plot

    :return:
    """
    print("Start Pearson Correlation Heatmap Plot  .. ", datetime.now())

    plt.figure(figsize=(10,8))
    plt.title('Pearson Correlation of miRNAs', y=1.05, size=15)

    # Correlation matrix for heatmap
    corr = np.corrcoef(X.transpose())

    plt.imshow(corr, cmap='BuPu', interpolation='nearest')
    plt.colorbar()
    plt.show()


texts: Bagging with Neural Networks Best practices

texts:
I am trying to build a majority vote system for 3 Neural Networks, and I came across the concept of Bagging method. Actually, I want to use neural networks as weak learners (I know it's debatable, but some papers have tried it and I want to try it too).
For more information about the voting system I tried to construct/constructed, please read the following thread (The softmax Layer is better the the function in the thread, because the majority function only gives equivalent accuracies of the 3 NNs, but doesn't improve the overall accuracy).
I read that bagging can improve the overall accuracy of the weak learners, but as you can see I only have 3 learners and there aren't any clear information about bagging with Neural Networks. I only found after some reading that I can use ensemble learning with Neural networks by using the output of the trained NNs in a linear fashion. If I want to detail this into steps, I would write:

Divide Dataset into training and validation sets
From the Training sat, construct 3 bootstraps samples ?, Or I need more?
Train/Develop the 3 neural Networks on the 3 bootstraps samples?
Test the Neural Networks on the validation/test set
How can I join decisions of the NNs by bagging?

If possible, I need your insight on these implementation steps and to know if they follow the best practices of bagging with NNs.
PS: I just started reading about bagging and boosting, so I apologize for any conceptual mistakes and contradictions I might have said.
Best regards,


texts: Usual decision trees are directed acyclic graphs. Are there generalizations of decision trees that contain cycles analogously to recurrent neural networks? If such trees exist, can they be applied to sequences of variable length? I could not find information about such trees. Or their existence is impossible of infeasible?


texts: Extracting the feature map in Tensorflow Object Detection API

texts: I would like to know how can I extract the feature map of a mobilenet trained on tensorflow object detection API. I want to take that feature map in order to feed another classifier. Thanks!


texts: How to reduce an unknown size data into a fixed size data? Please read details

texts:
Example:
Given n number of images marked 1 to n where n is unknown, I can calculate a property of every image which is a scalar quantity. Now I have to represent this property of all images in a fixed size vector (say 5 or 10).

One naive approach can be this vector- [avg max min std_deviation]
And I also want to include the effect of relative positions of those images.


texts: Book about distances for data science (can't remember the name)

texts:
I saw a book somewhere that listed a big amount of mathematical distance functions (the usual euclidean norms, the discrete distance, the hamming distance, etc) used for data science.

The name of the book might be similar to "101 metrics for data science". I cannot find it. Does anyone know about it (or a similar book)? Thanks!


texts: Measure the "aggregate preference" of points on a 2D plane

texts:
Consider some points on a 2D plane, we know corrdinates of all
points, how to measure the "aggregate preference" of them.

I don't know what is the exact terminology of
"aggregate preference".

What I want to express is that,
for example, there are 20 points on a 2D plane,
some time these points are
distributed uniformly in the space like (1).
We can say that, they have a relatively small "aggregate preference".
Some time, they are distributed more aggregate in space,
form some clusters, like (2).
They have a bigger "aggregate preference" than (1).

Is there are some method(like some statistics or algotithm)
to measure this "aggregate preference"
quantificationally?


texts: Why Self-Organizing Map (SOM) units are called neurons?

texts:
In the self-organizing map method, each of the output units is called neurons. But after reading the details about the algorithm, I could not find any logistic regression unit which is the neuron in a neural network architecture. I find it closer to the clustering algorithms like K-NN.

Is there any explanation why SOM units are called neurons? And why is it linked to neural networks (while ANN does a completely different job i.e., classification)

P.S.: In this paper (https://ieeexplore.ieee.org/document/6007170/), the SOM is called as an ANN. Here is a sentence from the abstract:

  the SOM is a type of artificial neural network that is trained using unsupervised data to produce a low-dimensional, discretized representation of an input space of training samples in a cell map.


texts: ANN : a posteriori variable selection based on weights

texts:
I have trained a vanilla NN, with m explanatory variables and p neurons in the first layer. Hence I have a m x p matrice of weight of the first layer.

I have standardised my input variables. So I expect the weights to be comparable in some ways.

The number of my variable m is a bit too high for practical reasons. So I'd like to reduce it.

Reading my weight matrice by line (weight of a variable trough different neurons) wouldn't make any sense.

Reading this matrix by columns, however, allows to see the weight of different variables for a given neuron. So for a given neuron it allows to compare variables weigths (because they are standardised) and highlight which variable are not so important in terms of weights. For practical purpose I consider a variable as not important if the absolute value of its weight is less than 1% of the absolute value of the maximum weight for a given neuron. I repeat this identification of variable of lesser importance trough all my p neurons.

I then tried to drop variable that weren't important for any neuron with inconsistent results. (sometimes the performance drops, sometimes not) I also tried other thresholds instead of 1%... but lower treshold will rarely highlight enough variables and higher threshold will highlight to many of them.

Any way to make this kind of a posteriori variable selection work ?


texts: CNN - How does backpropagation with weight-sharing work exactly?

texts:
Consider a Convolutional Neural Network (CNN) for image classification. In order to detect local features, weight-sharing is used among units in the same convolutional layer.
In such a network, the kernel weights are updated via the backpropagation algorithm.

An update for the kernel weight $h_j$ in layer $l$ would be as follows:

$h_j^l = h_j^l - \eta \cdot \frac{\delta R}{\delta h_j^l} = h_j^l - \eta \cdot \frac{\delta R}{\delta x_j^{L}} \cdot \frac{\delta x_j^{L}}{\delta x_j^{L - 1}} \cdot ... \cdot \frac{\delta x_j^{l}}{\delta h_j^l}$

How can the kernel weights be updated and still be the same (=shared)?

I have 2 possible explanations:

Weights of the same layer, which are initialized to the same value, will stay the same (independently of the input). This would imply that the expression $\frac{\delta R}{\delta h_j^l}$ is the same for all of these weights $h_1^l$ to $h_J^l$. This does not make sense, since $x_j^l$ is different for different j's. Or am I missing something here?
There is a trick, e.g. after the back-propagation update, the shared weights are set to their mean.

EDIT
The confusion I had was that I didn't consider that if a weight is shared, its parameter $h_j^l$ appears several times in the loss function. When differentiating for $h_j^l$, several terms (considering the according inputs) will "survive". Therefore the updates will be the same.


texts: Sklearn Aggregating Multiple Fitted Models Into A Single Model? (binary classification)

texts:
My problem context:

dataset too big to fit into memory.
binary classification [0,1]
30 csv files in a directory with exactly 30,000 samples (rows)
each file contains 15,000 0 class and 15,000 1 class (no unbalance)
model is xgboost.XGBClassifier()

If I iterate through these 30 files, fit a model, and pickle the model to disk; is it possible to aggregate these 30 fitted models into a single model? I am aware of sklearn.ensemble.VotingClassifier but that is used for:

sklearn.ensemble.VotingClassifier
Soft Voting/Majority Rule classifier for unfitted estimators.


texts:
I'm using Tensorflow 1.1.0 with GPU support and I have this function:

def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """
    # Basic LSTM cell
    # lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size)
    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size, forget_bias=0.0, state_is_tuple=True, reuse=tf.get_variable_scope().reuse)

    # Add drop to the cell
    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

    # Stack multiple LSTM layers
    rnn_cell = tf.contrib.rnn.MultiRNNCell([drop for _ in range(layers)], state_is_tuple=True)

    # Getting an initial state of zeros
    initial_state = rnn_cell.zero_state(batch_size, tf.float32)

    # Set the name
    initial_state = tf.identity(initial_state, 'initial_state')

    return rnn_cell, initial_state

The previous code works perfectly with Tensorflow 1.0.0 if I use lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size) but now I have changed the way I get the lstm cell. And I get this error:

  ValueError: Attempt to reuse RNNCell
   with a different variable scope than its
  first use.  First use of cell was with scope
  'rnn/multi_rnn_cell/cell_0/basic_lstm_cell', this attempt is with
  scope 'rnn/multi_rnn_cell/cell_1/basic_lstm_cell'.  Please create a
  new instance of the cell if you would like it to use a different set
  of weights.  If before you were using:
  MultiRNNCell([BasicLSTMCell(...)] * num_layers), change to:
  MultiRNNCell([BasicLSTMCell(...) for _ in range(num_layers)]).  If
  before you were using the same cell instance as both the forward and
  reverse cell of a bidirectional RNN, simply create two instances (one
  for forward, one for reverse).  In May 2017, we will start
  transitioning this cell's behavior to use existing stored weights, if
  any, when it is called with scope=None (which can lead to silent model
  degradation, so this error will remain until then.)

I have changed get_init_cell with this one:

def lstm_cell(rnn_size, keep_prob=0.75):
    # Basic LSTM cell
    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size, reuse=tf.get_variable_scope().reuse)

    # Add drop to the cell
    return tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """
    # Stack multiple LSTM layers
    rnn_cell = tf.contrib.rnn.MultiRNNCell([lstm_cell(rnn_size, keep_prob) for _ in range(layers)])

    # Getting an initial state of zeros
    initial_state = rnn_cell.zero_state(batch_size, tf.float32)

    # Set the name
    initial_state = tf.identity(initial_state, 'initial_state')

    return rnn_cell, initial_state

And I still getting the same error.

I have also try it with the same error:

def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """

    def lstm_cell():
        # Basic LSTM cell
        return tf.contrib.rnn.BasicLSTMCell(rnn_size, reuse=tf.get_variable_scope().reuse)

    def attn_cell():
        return tf.contrib.rnn.DropoutWrapper(lstm_cell(), output_keep_prob=keep_prob)

    # Stack multiple LSTM layers
    rnn_cell = tf.contrib.rnn.MultiRNNCell([attn_cell() for _ in range(layers)], state_is_tuple=True)

    # Getting an initial state of zeros
    initial_state = rnn_cell.zero_state(batch_size, tf.float32)

    # Set the name
    initial_state = tf.identity(initial_state, 'initial_state')

    return rnn_cell, initial_state

Any idea?


texts:
I work with texts where there is a dialogue between two people (a client and a call center employee, the beginning and end of each person’s phrase is not defined). My goal is to classify texts in which a call center employee names words from my list.
If the texts are manually marked up, can such a classification problem be solved?
Are there any tricks to solve this type of problem?

Sample data:
"hello hello my name is Sam Chin I'm calling for pizza delivery Okay now check your order wait a minute Sam"


texts: What is the use of having shared weights in later layers of a CNN?

texts:
In a CNN, all the neurons in a single layer use the same weights and bias. As a result, all the neurons detect the same feature. The early layers of a CNN detect simple features like edges and hence it makes sense to use the same feature detector in all parts.

But the later layers are shown to detect complex features. Since these need not occur in all regions of an image, what is the use of having shared weights and bias in later layers?

Instead isn't it better to have separate weights and bias so that different features can be learnt in a single channel? Advantage of this over Fully Connected layers is that we are still making use of local receptive fields and the number of parameters are still less.

For eg, in a face recognition network, suppose one of the feature detectors is to detect a certain type of eye. What is the use of using that detector at the bottom of the image? Since we already know eyes cannot occur at the bottom of a face.


texts:
We have a pet-project for generation of words out of symbols found on the image.
Currently, we are processing symbols on the image using Photoshop, convert those to .png files and then they are ready to be used for word generation.
I am looking for kind of schematic solution for this task. The biggest problem - those symbols are not really basic. What I mean, it could be either symbols on old address plate, or banner symbols, or even symbols that are used in graffiti tags.
Therefore, the generated picture is not basic either, here's the example:

What I see, there should be several layers such as:

Text detection algorithm
Image-to-text retrieval algorithm
Image-to-symbol_image retreival algorithm
Symbol outline retreival algorithm

I would really like you to share some insights on how it could be done, what models should I take as the basis and what are the misconceptions that I could have done.
I am guessing that image-to-text algorithms could use some symbol outline retreival mechanisms that are useful in my case, but I don't know where to start from


texts: I have got multiple features(descriptor, vector with elements of natural or real number) from a single image, which need to be searched against many image with multiple features. It is a problem from SLAM, particularly the localization method. What are the SOTA methods for such problems. The naive approach would be to do a k-nn for each feature in the source image and then rank the image from the search space, this is going to be too slow, especially the ranking of the images.


texts: GAN loss suddenly explodes and model breaks

texts:
Almost every time I've tried to train a DCGAN using keras I find that the loss suddenly skyrockets and the model completely stops improving. I find this happens regardless of what combination of loss functions, learning rates, label smoothing, or any other "GAN hack" I employ (and I've tried a lot, including cloning and running other people's models). Here is my most recent example, from a DCGAN that uses linear activation and MSE loss (metrics are D loss, D accuracy, G loss, G accuracy):

Epoch 26: 04224/04334; D=0.147 (79.7%); G=0.646 (14.4%);
Epoch 27: 04224/04334; D=0.139 (80.6%); G=0.644 (15.2%);
Epoch 28: 04224/04334; D=0.151 (79.0%); G=0.646 (14.2%);
Epoch 29: 04224/04334; D=199583.580 (75.8%); G=328619.291 (18.6%);
Epoch 30: 04224/04334; D=262175.355 (31.6%); G=14210.427 (77.9%);
Epoch 31: 04224/04334; D=948.686 (49.5%); G=135.858 (46.9%);

Here are the corresponding image aggregates:

I understand that GANs typically fail to converge, but haven't been able to find anyone else with this specific issue (massive model-breaking spike). Any idea what could be causing it?