texts: Why does my neural net always converge on a trivial solution?

texts:
As part of a more elaborate project, I am trying to get a rather simple network to distinguish between 5 very distinct classes of 1D signals (the members of each class is just the same bit of signal copied over 1000 times with a slight bit of noise added). And failing.

Below is the network setup, and the output. As far as I can tell (through matplotlib), I haven't done anything really silly with data (such as flip the dimensions, for instance). Does anyone have a suggestion for what could be going wrong?

I apologize for not generating dummy data within the script, I'm still much more comfortable in matlab, so it was easier to just port some dummy data from there.

model = Sequential()

model.add(Flatten(input_shape=(6000,1),name='S2'))
model.add(Dense(40,activation='relu',name='F1'))
model.add(Dense(10,activation='relu',name='F2'))
model.add(Dense(5,activation='relu',name='output'))
model.add(Activation(activation='softmax',name='softmax'))

model.summary()

# load fake data
earEEG=scipy.io.loadmat('forPython_dummy.mat')
data=earEEG['data'][:,:]
data=np.transpose(data)
data=data[:,:,None]

labels=earEEG['labels']
labels= keras.utils.to_categorical(labels)

sgd=keras.optimizers.SGD()
model.compile(loss='categorical_crossentropy',
              optimizer=sgd,
              metrics=['accuracy'])

hist=model.fit(data, labels, epochs=10, batch_size=50,class_weight='auto',shuffle='True')

And output:

Layer (type)                 Output Shape              Param #
=================================================================
S2 (Flatten)                 (None, 6000)              0
_________________________________________________________________
F1 (Dense)                   (None, 40)                240040
_________________________________________________________________
F2 (Dense)                   (None, 10)                410
_________________________________________________________________
output (Dense)               (None, 5)                 55
_________________________________________________________________
softmax (Activation)         (None, 5)                 0
=================================================================
Total params: 240,505.0
Trainable params: 240,505.0
Non-trainable params: 0.0
_________________________________________________________________
Epoch 1/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 2/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 3/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 4/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 5/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000
Epoch 6/10
5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000

Clearly, the algorithm is just giving the same label to everything (this is also apparent when outputting the actual labels). The input is a 5000 x 6000 matrix, and 5000 labels. As the algorithm does not complain about dimensional mismatch, I feel pretty confident that the data is not being transposed or similar. As an easy test, I tried feeding the same dataset to a pca+decision tree in matlab, and got a perfect score.

Classes are perfectly balanced, by the way.


texts: How to prevent model from recognizing false Classes

texts:
Let's say that I have a model that can recognize Cats and Dogs. However, when I use a picture of a Cup or Human it generates a random prediction at 0.70 confidence. Should I use sigmoid instead of softmax?

Or is there any fix for preventing massive confidence on new class object?


texts: how is the linear relation between positional encoding helping attention?

texts:
I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that it is a function only of $\phi$, and not of $t$.
However, I'm still not clear on how exactly is this helping the model to attend to phrases such as "they are", no matter if they appear in position 3 or position 7 in the sentence. The attention layer gets as an input [$E(they)+PE(3)$, $E(are)+PE(4)$] where $PE(4)=T_1PE(3)$ and $T_1$ is a nice diagonal linear transformation that depends only on the position diff which is 1. How is it using those nice properties for learning and generalizing for the case where the same phrase "they are" appears in position 7? how is the fact that the linear transformation is nice and diagonal helping the model?


texts: Create word embeddings without keeping fastText Vector file in the repository

texts:
I am trying to embed a sentence with the help of Infersent, and Infersent uses fastText vectors for word embedding. The fastText vector file is close to 5 GiB.

When we keep the fastText vector file along with the code repository it makes the repository size huge, and makes the code difficult to share/deploy (even creating a docker container).

Is there any method to avoid keeping the vector file along with the repository, but reuse it for embedding new sentences?


texts: Binary classification and numerical labels

texts:
I am trying to create a sentiment analysis model using a dataset that have ~50000  positive tweets that i labeled as 1, ~50000 negative tweets that i have labeled as 0. Also i have acquired ~10000 tweets that are neutral.
Due to the low number of neutral tweets my thinking is to label neutral with 0.5 and train the model using binary crossentropy as loss function. My output layer is 1 neuron with sigmoid activation function so prediction value would be between (0,1) .
Is my thinking right or it will mess the accuracy?


texts:
I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months).  I have a sequential dataset with 50 customers i.e. the rows contain sequence of 20 states for each of the 50 customers (dataset has 50 rows and 20 columns excluding the headers). I am trying to determine the next state using markov chains and all the literature in the web is focused around examples of text strings. I am looking something specific to the kind of example I have.
Can somebody please help me come up with the initial probability matrix and then consider the 20 states to predict the next state?


texts:
I am new to anomaly intrusion detection system. I was reading the service specific anomaly detection for network intrusion detection but I could not understand section 3.2.3 payload Distribution, page 5. To better understand the author I tried to do the calculations by hand but I am lost. I have included a sample of my calculations.

How do we calculate the payload distribution if we have many character distribution (one character distribution for each request in the training set)?
How do we calculate the segments for the payload distribution?
Given a new request how do we calculate its anomaly score?


texts: Difference between segmentation and effect of explanatory variables

texts:
In the context of discrete choice models, what difference does it make in segmenting my sample based on a particular "criteria" and study the effects of explanatory variables on each segment VERSUS Just adding the "criteria" as another explanatory variable in the unsegmented population

Say, I want to build a commuter mode choice model using MNL, what difference does it make when I split my population via gender and study the effects of various explanatory variables VERSUS adding gender as another explanatory variable in the entire unsegmented population.


texts: Why doesn't overfitting devastate neural networks for MNIST classification?

texts:
I have a simple neural network(NN) for MNIST classification.  It includes 2 hidden layers, each with 500 neurons.  Hence the dimensions of the NN are: 784-500-500-10.  ReLU is used in all neurons, softmax is used at the output, and cross-entropy is the loss function.

What puzzles me is why overfitting doesn't appear to devastate the NN?

Consider the number of parameters (weights) of the NN.  It's approximately
$$784\times500+500\times 500+500\times 10=647000.$$
However, in my experiment, I used only $6000$ examples (a tenth of the MNIST training set) to train the NN. (This is just to keep the run time short.  The training & test error would both go down considerably if I used more training examples.)  I repeated the experiment 10 times.  Plain stochastic gradient descent is used (no RMS prop or momentum); no regularization/drop-out/early-stopping was used.  The training error and test error reported were:

$$\begin{array}{|l|c|c|c|c|c|c|c|c|c|c|}
\hline
\textrm{No.} & 1 & 2 & 3 &4 &5&6&7&8&9&10\\ \hline
E_{train}(\%) & 7.8 & 10.3 &  9.1 & 11.0 & 8.7 & 9.2 & 9.3 & 8.3 &10.3& 8.6\\ \hline
E_{test}(\%) & 11.7 & 13.9 & 13.2 & 14.1 &12.1 &13.2 &13.3 &11.9 &13.4&12.7\\ \hline
\end{array}$$

Note that in all 10 experiments (each with independent random parameter initialization), the test error differed from the training error only by approx. 4%, even though I used 6K examples to training 647K parameters.  The VC dimension of the neural network is on the order of $O(|E|log(|E|))$ at least, where $|E|$ is the number of edges (weights).  So why wasn't the test error miserably higher (e.g. 30% or 50%) than the training error?  I'd greatly appreciate it if someone can point out where I missed.  Thanks a lot!

[ EDITS 2017/6/30]

To clarify the effects of early stopping, I did the 10 experiments again, each now with 20 epochs of training.  The error rates are shown in the figure below:

$\qquad\qquad\qquad$

The gap between test and training error did increase as more epochs are used in the training.  However, the tail of the test error stayed nearly flat after the training error is driven to zero.  Moreover, I saw similar trends for other sizes of the training set.  The average error rate at the end of 20 epochs of training is plotted against the size of the training set below:

$\qquad\qquad\qquad$

So overfitting does occur, but it doesn't appear to devastate the NN.  Considering the number of parameters (647K) we need to the train and the number of training examples we have (<60K), the question remains: why doesn't overfitting easily render the NN useless?  Moreover, is this true for the ReLU NN for all classification tasks with softmax output and cross-entropy objective function?  Have someone seen a counter-example?


texts: Train/Test/Validation Set Splitting in Sklearn

texts: How could I split randomly a data matrix and the corresponding label vector into a X_train, X_test, X_val, y_train, y_test, y_val with Sklearn? As far as I know, sklearn.cross_validation.train_test_split is only capable of splitting into two, not in three...


texts: Does feature selection removes highly corelated variables?

texts: I know that feature selection helps in removal of irrelevant features. Do they also remove redundant features(features which convey the same info as as some other feature)? If not, which methods are used for removing these highy corelated variables?


texts: How does TensorFlow differentiate over Gaussian parameters?

texts:
When using variational autoencoders it is common to use stochastic tensors after the encoder that will have any value sampled from a distribution (Gaussian, Bernoulli, ...). How does Tensorflow calculate the derivatives with respect to the distribution parameters in order to do backpropagation?

Note: I know how to differentiate using Tensorflow API. I would like to know how it calculates the derivative internally, in other words, how could I do it using pen and paper?

Thanks!


texts: Linearly increasing data with manual reset

texts:
I have a linearly increasing time series dataset of a sensor, with value ranges between 50 and 150. I've implemented a Simple Linear Regression algorithm to fit a regression line on such data, and I'm predicting the date when the series would reach 120.

All works fine when the series move upwards. But, there are cases in which the sensor reaches around 110 or 115, and it is reset; in such cases the values would start over again at, say, 50 or 60.

This is where I start facing issues with the regression line, as it starts moving downwards, and it starts predicting old date. I think I should be considering only the subset of data from where it was previously reset. However, I'm trying to understand if there are any algorithms available that consider this case.

I'm new to data science, would appreciate any pointers to move further.

Edit: nfmcclure's suggestions applied

Before applying the suggestions

Below is the snapshot of what I've got after splitting the dataset where the reset occurs, and the slope of two set.

finding the mean of the two slopes and drawing the line from the mean.

Is this OK?


texts: Applying association rules for variable prediction in R

texts:
A have a data frame with shops with a number of independent variables (all categorical), I'm trying to find a decent model to predict a dependent variable that is ordered and categorical (ie volume of foot traffic: "low", "medium", "high"). The dependent variable is defined for about 20% of the observations, and I want to predict for the missing 80%.

What models/techniques would you suggest?

For now, I started by learning some association rules with arules (manual here):

require(arules)
train # my data frame with the complete observations
target_field = 'foot.traffic'
# filter rules to observe only the relevant ones
appearance_list = list(rhs=paste0(target_field,"=", unique(train[,target_field])))

rules <- arules::apriori(train,
            parameter = list( support=0.1, confidence=0.2 ),
            appearance = appearance_list)
rules <- sort(rules, by="lift")
inspect(rules)

Now I can inspect these rules, and they generally make sense. They mostly refer to low and high cases, indicating that rules for medium are harder to identify.

How can I apply them back to the training set to see how well they predict the foot.traffic variable? Is there a good off-the-shelf strategy to apply the rules?


texts: Help making a custom categorical loss function in Keras

texts:
I am a bit new to machine learning, and I'm trying to get the basics working towards a bigger project using a very simple encoder-decoder model. It looks like this:

embedding_dim = 300
lstm_layer_size_1 = 300
lstm_layer_size_2 = 300

model = Sequential()
model.add(Embedding(self.max_input_vocab, embedding_dim,
    input_length=self.max_input_length, mask_zero=True))
model.add(LSTM(lstm_layer_size_1)) # encoder
model.add(RepeatVector(self.max_output_length))
model.add(LSTM(lstm_layer_size_2, return_sequences=True)) # decoder
model.add(TimeDistributed(Dense(self.max_output_vocab, activation='softmax')))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])

It takes in a sequence of words encoded as integers, with 0 padding up to max_input_length. And outputs a one-hot-encoded version of the output for words up to max_output_length.

For example, with a max ouput length of 115, and an expected output of length 20, the network should predict 20 integers in the range max_output_vocab, followed by 95 predicted 0's.

My problem:

I've been running into the issue that the network trains way too much off of the zero tokens in the output, as many of the target sequences have output lengths far below the max output length. The network ends up learning it can get the most accuracy by just predicting almost all 0's for most of the output.

I want to try to make a custom loss function that won't train on any output that comes after the first 0 token, but I'm not sure how I would go about doing this properly.

I know it will look similar to the keras.backend categorical_crossentropy, but would it be as simple as continuing to use a version of that function, but only feeding it the portion of the output sequence I want (everything before the first 0 token in the expected output)?


texts: how to propagate error from convolutional layer to previous layer？

texts:
I've been trying to implement a simple convolutional neural network. But I've been stuck at this problem for over a week.

To be specific, assume there are 3 layers in a convolutional pass, marked as l-1, l, l+1 -layer respectively. And the l-1 layer is the input layer with shape (num_channels, img_height, img_width),the l layer is the convolutional layer with kernel shape (num_kernels, num_channels, filter_height, filter_width), and the l+1 layer is the pooling layer with pooling size (pool_height, pool_width).  Note that in my simple implementation, the filters are 3-dimensional, and they have the same depth as with the input, i.e., they have the same number of channels as that of the input.

On this setting, I encountered a problem in the back-propagation phase.

I've studied many tutorials in neural networks, however, it seems that all of them stop with back-propagating the error to l layer with the formular:
\begin{align}
   \delta_k^{(l)} = \text{upsample}\left((W_k^{(l)})^T \delta_k^{(l+1)}\right) \bullet f'(z_k^{(l)})
   \end{align}
The question is, how should I back-propagate error from the l layer to l-1 layer, that is, calculate $\delta_k^{(l-1)}$, in order to chain several conv-pooling pass together?

In my implementation, there is a class called ConvPoolLayer which has a feedforward method and a backprop method. The method backprop needs to propagate the error from output to input. When I chain instants of class ConvPoolLayer together, I just need train their feedforward methods and backprop methods together respectively.


texts: Embedding layers trained on Amazon Reviews

texts: I am working on research to perform sentiment analysis on Amazon reviews. My data is not labelled so I am now using Lexicon based sentiment analysis such as Vader. I am wondering if it is possible to use embedding layers to perform sentiment analysis in an unsupervised manner? Are there any pre-trained embedding layers on Amazon reviews that I can use? Thank you very much in advance.


texts: Accumulators in Spark (PySpark) without global variables?

texts:
BACKGROUND

Consider the following textbook example which uses accumulators to add vectors.

from pyspark import AccumulatorParam

class VectorAccumulatorParam(AccumulatorParam):

    def zero(self, value):

        dict1 = {i: 0 for i in range(0, len(value))}

        return dict1

    def addInPlace(self, val1, val2):

        for i in val1.keys():
            val1[i] += val2[i]

        return val1

rdd1 = sc.parallelize([{0: 0.3, 1: 0.8, 2: 0.4},
                       {0: 0.2, 1: 0.4, 2: 0.2},
                       {0: -0.1, 1: 0.4, 2: 1.6}])

vector_acc = sc.accumulator({0: 0, 1: 0, 2: 0},
                            VectorAccumulatorParam())

def mapping_fn(x):

    global vector_acc

    vector_acc += x

rdd1.foreach(mapping_fn)
print(vector_acc.value)

This prints {0: 0.4, 1: 1.6, 2: 2.2}, i.e., the element-wise sum of the vectors.

QUESTION

The use of the global scope in mapping_fn() gnaws at me, since it's usually bad practice. Is there a simple way to illustrate how accumulators work without resorting to a global variable? This is all the more confusing since, as far as I understand, the beauty of Spark lies in a share nothing philosophy, and global is precisely the opposite of that.


texts:
I am currently reading a white paper relating to Expectation-Maximisation (EM) and would like to encode a formula so I can play with it in order to help my understanding. The formula in question is a sum over values and shown below;

I am wondering what the advice on the best way to acheive this would be. I am currently thinking about using a for loop to create some kind of cumulative value as I iterate from t = 1 to m.


texts: # of iterations in Restricted Boltzmann Machine (RBM)

texts: I have a training set, I provide it (consider a data from training set) to the visible layer. Then the normal process is followed, i.e. Positive Phase-> Negative Phase-> Reconstruction of weights, bias units take place. Does it end here or should I have to iterate with the present visible unit value again (the process follows for few fixed iterations) ? Because it is not properly stated anywhere whether with each data of training set, it has to iterated again and again or with each data of training set.


texts: How to handle categorical features in K-means?

texts: I am working on clustering algorithms. I am working with titanic dataset. It contains 6 categorical features. I used k-means algorithm on this dataset. I am using label encoding for categorical features. But I found that categorical features should use euclidean distance. It should use Hamming distance. So, how to make k-means work finely on mixed features? I don't need other algorithm. I just want to work with k-means only on mixed features dataset.


texts:
Imagine we have some lists of features that are changing in time. Each row of the list corresponds to a sample (Change in space). I would like to know whether machine learning is able to determine the effect of each sample on another sample. For instance, the target value for the sample "S" is dependent on features of samples "S-4","S-3","S-2","S-1","S+1","S+2","S+3". I have seen something like Active learning and Causal Inference but still not sure each of which would be useful for my aim.
To elaborate more, imagine we have the picture below:

the red line is a result for one year and the blue one is next year. we have these results in an appropriate amount so in this manner we do not have problem. for the target that has been shown by the red circle and other samples, we have different features. But I am looking for an algorithm to tell me whether group 1 is affecting my target in the red circle point or group 2. To this aim, it is good to use Causal inference or Active Learning?


texts: optical chemical structure recognition from images

texts: I want to recognize name of the chemical structure from the image of chemical structure, like in the given image it is benzene structure I want to recognize that it is benzene from the image(I should be able to recognize all these structures as benzene) What way I could follow to achieve this? (like this given any structure I want to recognize the name of it)  .


texts: Regression model with variable number of parameters in dataset?

texts:
I work in physics. We have lots of experimental runs, with each run yielding a result, y and some parameters that should predict the result, x. Over time, we have found more and more parameters to record. So our data looks like the following:

Year 1 data: (2000 runs)
    parameters: x1,x2,x3                target: y
Year 2 data: (2000 runs)
    parameters: x1,x2,x3,x4,x5          target: y
Year 3 data: (2000 runs)
    parameters: x1,x2,x3,x4,x5,x6,x7    target: y

How does one build a regression model that incorporates the additional information we recorded, without throwing away what it "learned" about the older parameters?

Should I:

just set x4, x5, etc to 0 or -1 when I'm not using them?
completely ignore x4,x5,x6,x7 and only use x1,x2,x3?
add another parameter that is simply the number of parameters?
train separate models for each year, and combine them somehow?
"weight" the parameters, so as to ignore them if I set the weight to 0?
make three different models, using x1,x2,x3, x4,x5, and x6,x7 parameters, and then interpolate somehow?
make a custom "imputer" to guestimate the missing parameters (using available parameters)

I have tried imputation using mean and median, but neither works very well because the parameters are not independent, but rather fairly correlated.


texts: which algorithm used by sklearn.model_selection.StratifiedShuffleSplit for clustering into n_splits

texts:
which algorithm is used by sklearn class StratifiedShuffleSplit for clustering/stratification into n_split. dendrogram looks evident but can anyone suggest any reference.

class sklearn.model_selection.StratifiedShuffleSplit(n_splits=10, test_size=None, train_size=None, random_state=None)


texts: I am using Median Absolute Deviation(MAD) for outlier detection. But the problem with MAD is that if 50% or more values in a sample are identical, then MAD = 0 which is not desirable. Is there any way to solve this problem?


texts: How to view entire 5788 rows and 7 columns.?

texts:
I want to see entire 5788 rows and 7 columns of my CSV data. I used below code and it is giving little data by stating [5788 rows x 7 columns] but showing only 10 rows and 5 columns in python prompt. How to get display of entire rows and column there?

import pandas as pd
df = pd.read_csv("BusinessData.csv")
print (df)


texts:
I am wondering how the yelp spam detection system detects paid reviews? By paid, I mean the following scenarios:

I as a business owner pay people to write positive comments and give me a good rate
I as a business owner pay people to write negative comments against my opponent businesses.

I appreciate any technical response and also the introduction of good resources. Thanks in advance.


texts: How do I build a permutation invariance neural network in keras?

texts:
My question is about the structure of the network required to solve my problem with fewer data.

I have a sensor device that simply reports the color of the thing it's seeing in front of it. One sensor reports me 4 numbers: Red, Green, Blue, and Alpha.
The intensity of the color changes depends on the distance and the thing it's seeing.
I have 6 such sensors attached on each side of a small cube. The cube can be moved and rotated using the hand.

I want to predict the position of the cube in space in real-time.

My problem:

Input: 6 identical sensors, each giving 4 numbers. Total=6*4=24 numbers.

Output: 3 numbers, X, Y, Z, (Position of the cube)

I have my data ready with label XYZ.

Now, train a simple Multi-layer perceptron which takes 24 numbers and output 3 numbers.
This is working quite fine but it needs hell lots of data in a cubic meter space to predict accurately.

The problem is about the rotation. I need to rotate and cover 360 degrees for every location for it to predict well.

But I know for the fact that each sensor is identical so I want to share the weight of each sensor. I know that when you rotate the cube 90 degrees, it should not affect the output position at all. So this should mean the order of the sensors is not important. This means I should somehow use add or average to merge my sensors' layer. If I use concatenate it will preserve the order which make the output position changes.

The way I'm doing it is that I feed 4 numbers into a sensor model that are shared among all sensors, get the encoding, add them up, then connect it to Dense layers. Following is the model prototype:

from keras.layers import Input, Dense
from keras.models import Model, Sequential

sensor1 = Input(shape=(4,))
sensor2 = Input(shape=(4,))
sensor3 = Input(shape=(4,))
sensor4 = Input(shape=(4,))
sensor5 = Input(shape=(4,))
sensor6 = Input(shape=(4,))
sensor_model = Sequential([
    Dense(64, activation='relu'),
    Dense(64, activation='relu'),
])
sensor1_encoding = sensor_model(sensor1)
sensor2_encoding = sensor_model(sensor2)
sensor3_encoding = sensor_model(sensor3)
sensor4_encoding = sensor_model(sensor4)
sensor5_encoding = sensor_model(sensor5)
sensor6_encoding = sensor_model(sensor6)
sensor_encoding = average([
    sensor1_encoding,
    sensor2_encoding,
    sensor3_encoding,
    sensor4_encoding,
    sensor5_encoding,
    sensor6_encoding,
])
h = sensor_encoding
h = Dense(128, activation='relu')(h)
h = Dense(128, activation='relu')(h)
h = Dense(3, activation='linear')
model = Model(inputs=[sensor1, sensor2, sensor3, sensor4, sensor5, sensor6], outputs=[h])

Right now, when I change the average function to concatenate, the model loss is lower on both training and validation set which contradicts with my intuition. What is wrong with my thinking?
What do you think? How do I adjust this model in order for it to predict the same position if I rotate 90 degrees while also not suffering from 45 degrees rotation. And also make it in such a way that it does not remove the useful relationship between the inputs.


texts:
I'm trying to learn how to work with tensorflow.
My background is somewhat a mathematical one, but I know how to program in Mathematica, Python and R.
I'm looking for some bibliography focused only on Tensorflow. I've searched, and there are so many possibilities, but I simply don't have the knowledge to assess either the publisher, the authors, the content.


texts: Modeling of topics orthogonal to a given patterns

texts:
How to force the topics to be different from the defined ones?
Suppose I have a collection of texts about cats and dogs.There should naturally be two topics: one about dogs and one about cats. But I'm not interested in such results, I'd like to get topics that are completely different (let's call them orthogonal). In other words I would like to use topic modeling to discover a different division of documents (but I don't know what). I thought about guided LDA, but in my opinion it is not applicable in this case (maybe I'm wrong?).
One idea is to remove from the corpus the words that make up the topic about cats and about dogs. I can imagine such a solution in such a way that I run the algorithm to model topics and in the obtained results I mark which topics are not interesting for me and run the algorithm again but already conditioned with this information.
Do you think this is a good idea? Or are there any methods designed for this?
I also found a paper Probabilistic Text Modeling with Orthogonalized Topics, in which the concept of orthogonal themes appears, but it is used in a different context than I want.


texts: How to link csv file within a markdown file when using Jupyter Lab?

texts: I would like to render rapid reports using the Markdown files with links to csv files [csv](path to csv file). This part is working, I can dowload the csv file from the markdown preview, but I can't open the file in the Jupyter Lab "Table". Is there is something to do about this?


texts: How to implement a YOLO model using TensorFlow?

texts:
I want to implement a face detection model using TensorFlow. I have a dataset downloaded from Google's OID. I have prepared the annotation file as required in the YOLO. But I don't understand how to convert the annotations to the required output tensors.
Is learning Darknet a simpler option?


texts: is it ok to get 100% accuracy in random forest classifier algorithm?

texts:
while i was building the model to predict the performance of machine using the features like OEF,working time,performance/head etc...

I splitted the training data using

x_trainset, x_testset, y_trainset, y_testset = train_test_split(x, y, test_size=0.3)

then built the model

#Import Random Forest Model
from sklearn.ensemble import RandomForestClassifier

# Create the model with 100 trees
forest=RandomForestClassifier(n_estimators=100)

#Train the model using the training sets
train1=forest.fit(x_trainset, y_trainset)
y_pred=train1.predict(x_testset)

y_pred

o/p:

array(['level2', 'level5', 'level2', 'level3', 'level3', 'level2',
       'level3', 'level2', 'level4', 'level2', 'level2', 'level3',
       'level4', 'level3', 'level3', 'level2', 'level2', 'level3',
       'level3', 'level3', 'level3', 'level3', 'level2', 'level5',
       'level3', 'level3', 'level2', 'level5', 'level4', 'level3',
       'level3', 'level3', 'level2', 'level2', 'level3', 'level3',
       'level3', 'level2', 'level4', 'level2', 'level2', 'level3',
       'level4', 'level3', 'level1', 'level3'], dtype=object)

y_testset

o/p:

120    level2
104    level5
117    level2
91     level3
4      level3
10     level2
133    level3
18     level2
119    level4
61     level2
71     level2
30     level3
27     level4
103    level3
14     level3
59     level2
50     level2
55     level3
53     level3
22     level3
78     level3
114    level3
70     level2
60     level5
6      level3
1      level3
83     level2
82     level5
26     level4
67     level3
62     level3
145    level3
74     level2
11     level2
107    level3
29     level3
138    level3
49     level2
150    level4
8      level2
20     level2
95     level3
51     level4
56     level3
72     level1
102    level3
Name: levels, dtype: object

then found the accuracy and i got like,

from sklearn import metrics #Import scikit-learn metrics module for accuracy calculation
print("Accuracy:",metrics.accuracy_score(y_testset, y_pred))

o/p:

Accuracy: 1.0

In x_trainset,x_testset i have only the features(7) and in y_trainset,y_testset i have only column "levels" which is needed to be predicted

dataset size is about 153

I don't know whether this is correct.Please can anyone answer this as soon as possible


texts:
Google recently included in tensorflow's nightly builds its Eager mode, an imperative API to access tensorflow computation capabilities.

How do tensorflow eager compare to PyTorch?

Some aspects that could affect the comparison could be:

Advantages and disadvantages of eager due to its static graph legacy (e.g. names in nodes).
Intrinsic limitations of either of them that the other does not have.
Areas in which one of them needs improvement (e.g. feature completeness, computational optimizations).
Ecosystem differences (e.g. tensorboard?).

Note1: Yaroslav Bulatov wrote a review about eager's nice features.

Note2: In a previous question, I requested a comparison between PyTorch and Tensorflow Fold. At that time, it seemed to me that Fold could face PyTorch thanks to Google backing it. I was very very wrong: in the end, Google itself abandoned Fold in favour of Eager. I understand that this was due to intrinsic limitations in the normal tensorflow API that led Fold not to be very friendly, which constrained its adoption.


texts: Confusion with L2 Regularization in Back-propagation

texts:
In a very simple language, this is L2 regularization
$\hspace{3cm}$$Loss_R$ = $Loss_N + \sum w_i^2$
$Loss_N$ - Loss without regularization
$Loss_R$ - Loss with regularization
When implementing [Ref], we simply add the derivative of the new penaty to the current delta weight,
$\hspace{3cm}$$dw = dw_N + constant*w$
$dw_N$ - Weight delta without regularization
What I think - L2 regularization is achieved with the last step only i.e. the weight is penalized.
My question is -
 Why do we then add the loss in the total loss as done in the first equation. Will, it not put an additional penalty during back-prop( On $dw_N$ component) for each weight because of increased Loss. I can understand if it is for console print purpose but it is not, I believe.
I know I am missing something very simple.


texts: Generative Adversarial Networks - The simplest possible examples

texts: I'm looking for the simplest possible examples of GANs. What would be simple yet illustrative examples with, say, univariate data and in which both the generator and the discriminator are as simple as possible (potentially depending only on one or two parameters each, and not necessarily being neural networks at all)? Thanks for your help!


texts:
Is the total sample size = 30.891 and the overall percentage of “yes” = 11,3%? How I can describe the leaves predicting "yes" outcome in term of explicative variables and values?
What are the variables appearing in the second tree level?

Thank you in advance!


texts: Is it normal for some classifiers to change their prediction performance from fixed amount of data each time I run them to train?

texts:
I have a fixed amount of dataset, say text data with 100,000 records, to rank classifiers, such as gradient boosting, random forest, and other ones, according to their classification performance. The dataset is divided randomly for training and validation but the issue is that I get different prediction performance even if I use the same dataset overtime.

My understanding is that once the classifiers are ranked after the first training and validation run on the dataset, the rank should always show the same way as the previous rank. I don't add or delete the fixed amount of dataset that I use. Even though the same dataset is divided randomly, what I think is there shouldn't be a significant impact on the performance because the ratio of data used between training and validation is 80(training):20(validation). If they change their behaviors each time I run on the same unchanged fixed amount of data, how can I tell which one shows better performance on the current dataset? I understand the training data changes but its percentage is 80% of the dataset.


texts: Having trouble figuring out how loss was calculated for SQuAD task in BERT paper

texts:
The BERT Paper

https://arxiv.org/pdf/1810.04805.pdf

Section 4.2 covers the SQuAD training.

So from my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End).

For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.

I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph

"and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions."

What do they mean by "maximum scoring span is used as the prediction"?

Furthermore, how does that play into "The training objective is the loglikelihood of the correct start and end positions"?

From this Source:

https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/

It says the log-likelihood is only applied to the correct classes. So the we are only calculating the softmax for the correct positions only, Not any of the in correct positions.

If this interpretation is correct, then the loss will be

Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )


texts:
In Fuzzy c-means, we have to put how many centers(centroids) in the code. I am wondering how many centroids are suitbale ?

How to define a suitbale number of centroids for fuzzy c-means?


texts:
I have used a few regression models on the same dataset and obtained error metrics for them as shown below,

The RMSE(Root Mean Squared Error) and MAE(Mean Absolute Error) for model A is lower than that of model B where the R2 score is higher in model A. According to my knowledge this means that model A provides better predictions than model B. But when considering the MAPE (Mean Absolute Percentage Error) model B seems to have a lower value than model A. I would really appreciate it if someone could explain why it is so. Thanks in advance.


texts: Sequentially Training Certain Layers/Sub-Networks in Keras Functional API

texts:
Suppose we have a stacked neural network architecture with a
layer that is to be shared between two "sub-networks".

Example:

from keras.layers import Input, Dense
from keras.models import Model

main_input = Input(shape=(5, ))

## Model A: main_input -> A_output
layer_A1 = Dense(10, name='A1')(main_input)
layer_A2 = Dense(10, name='A2')(layer_A1)
layer_A3 = Dense(10, name='A3')(layer_A2)
A_output = Dense(1, name='A_output')(layer_A3)

## Model B: main_input -> layer_A2 -> B_output
layer_B1 = Dense(10, name='B1')(layer_A2)
B_output = Dense(1, name='B_output')(layer_B1)

model = Model(inputs=main_input,
              outputs=[A_output, B_output],
)

model.compile(optimizer='adam',
             loss={
                        'A_output':'mean_squared_error',
                        'B_output':'mean_squared_error'
                    },
             )

The goal is to train model A first so that model B can learn from the pre-trained weights of Layer A2. However calling fit in the current architecture will train both simultaneously and sum up the losses.

How can I change the architecture so that model A is trained first without creating separate models? Ultimately, I'll need to call model.predict(new_sample) where new_sample is of shape (5,) in the example.


texts:
I have "millions" of items each having N binary features. When a feature is "0" it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to have a probability of the "0" features being "1".

I am thinking this can be a Neural network with all features as input and same as output. But then I don't know how the training would work. I don't have ground truth.

I would like some help expressing my problem and hopefully not reinvent the wheel. Is this is a classical problem in ML, and what approach can be applied?


texts: The sum of multi-class prediction is not 1 using tensorflow and keras?

texts:
I am studying how to do text classification with multiple labels using tensorflow. Let's say my model is like:
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, 50, weights=[embedding_matrix], trainable=False),
    tf.keras.layers.LSTM(128),
    tf.keras.layers.Dense(4, activation='sigmoid')])
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=tf.metrics.categorical_accuracy)

I have 4 classes, the prediction function has the following results:
pred=model.predict(X_test)
pred
array([[0.915674  , 0.4272042 , 0.69613266, 0.3520468 ],
       [0.915674  , 0.42720422, 0.69613266, 0.35204676],
       [0.915674  , 0.4272042 , 0.69613266, 0.3520468 ],
       [0.9156739 , 0.42720422, 0.69613266, 0.3520468 ],
   ......

You can see that every data has 4 prediction values. But the sum of them is not 1, which I do not understand. Did I do something wrong, or how to interpret this?


texts: Does statsmodels fully support MultiIndex?

texts:
The below code snippet shows how statsmodels seems to flatten MultiIndex tuples by joining them with an underscore "_".
import numpy as np
import pandas as pd
from statsmodels.regression.linear_model import OLS

K = 2
N = 10
ERROR_VOL = 1

np.random.seed(0)
X = np.random.rand(N, K)
coefs = np.linspace(0.1, 1, K)
noise = np.random.rand(N)
y = X @ coefs + noise * ERROR_VOL

index_ = pd.MultiIndex.from_tuples([('some_var','feature_0'), ('some_var','feature_1')])
df = pd.DataFrame(X, columns=index_)
ols_fit = OLS(y, df, hasconst=False).fit()
print(ols_fit.params)

The result is
>>> some_var_feature_0    0.230474
some_var_feature_1    1.646789
dtype: float64

Because of the above flattening, the following, and similar operations relying on name matching, fail:
params_stdzd = ols_fit.params * df.std()
>>> ValueError: cannot join with no level specified and no overlapping names

Questions

Is there a way to get statsmodels to respect a pandas MultiIndex rather than flatten it?

If not:

Is there a way to set the flattening character to something other than underscore?

can I rely on OLS.params respecting the order of df.columns? If so I could just reindex OLS.params with df.columns to get a properly indexed params Series.

Are there better ways to get MultiIndex interoperabilty with statsmodels?


texts: Using regression or classification for rare events

texts:
I'm willing to make a model to predict the number of occurences of rare events.
I was wondering if classification could work better than regression for that particular case ?

The events are very rare (500 in 60000 entries) and can be either 1 or 2, all others are set to 0.


texts: NEAT XOR Example gives different results at each run

texts:
I have just started learn to use NEAT algorithm. I was thinking I understood the basics of NEAT when I read "Evolving Neural Networks through
Augmenting Topologies" paper and current python documentation. However, in practice I m in trouble.

I have run XOR examples from source (https://github.com/CodeReclaimers/neat-python.git). However, when I run any of "evolve-feedforward.py",  "evolve-feedforward-parallel.py" or "evolve-minimal" codes more than once, results are  changing. For instance,

When I run "evolve-feeedforward.py" it gives : 1

when I run same code for second time, it gives: 2

and it changes with each run. I have also deleted generated neat-checkpoint files before each run, but structure still changes. I would like to understand reason what cause this, I think I m missing some essential points here.


texts: Comparing different machine learning methods over multiple test datasets with different number of samples

texts:
Say, I have an image dataset (for example, imagenet) and I am training two image recognition models on it. I train a resnet with 10 layers 3 times on it (each time with different random weight initialization), each time for 20 epochs. For last 5 epochs of training, the accuracy on test datasets does not change very much, but oscillates around. At each of the last 5 epochs, I save the current weights (at that epoch) of the model.
I also have a resnet with 20 layers. Let's say I train it 4 times for 20 epochs on the same dataset, and simiarly save the weights at the final 5 epochs for each training.
I also have 10 test image datasets, coming from various sources, maybe from internet, web cameras, street cameras, screenshots from movies, etc. Each of the the datasets has varying number of images in them, ranging from 20 to 20000.
I evaluate all the models (2*(3+4)*5=70) on all the datasets.
Now given the above info, I have these questions: What is the probability that a resnet with 20 layers is on average better on these datasets than a resnet with 10 layers? (on average, as in calculating the accuracy on each of the ten datasets, and then taking the mean of the ten resultant values). And what are the confidence intervals (or credible intervals) around that probability value?
There are multiple sources of variance here: variance due to test dataset sizes, variance due to different weight initializations, variance due to accuracy oscillating from one epoch to next. How do you account for all these sources of variance to come up with a single number which would indicate the probability that one method is better than the other?
And finally, imagine that you did these tests, and you noticed that on one of the ten datasets the accuracy difference is the largest between these two methods. How can you quantify whether such accuracy difference is by chance or because it indeed is the case that one of the methods is better on this particular dataset? (the concern here is the multiple hypothesis testing and how to account for it, while taking care of all the other sources of variance as well).


texts: what is difference between set() and word_tokenize()?

texts:
from nltk.tokenize import sent_tokenize ,word_tokenize

sentence = 'jainmiah I love you but you are not bothering about my request,
            please yaar consider me for the sake'

word_tok = word_tokenize(sentence)
print(word_tok)

set_all = set(word_tokenize(sentence))
print(set_all)

  Actually both word_tokenize() and set(word_tokenize()) both returns
  same answers what makes the difference?


texts: How to get the weights of a linear model by solving normal equation?

texts:
In chapter 6.1 of the book Deep Learning, the author tries to learn the XOR function by using a linear model (on page 168).

Linear Model:

$f(\mathbf{x};\mathbf{w},b)=\mathbf{x}^T\mathbf{w}+b$

MSE Loss:

$J(\mathbf{w},b)= \frac{1}{4} \sum(f^*(\mathbf{x})-f(\mathbf{x;\mathbf{w},b})) $ , where  $f^*(\mathbf{x})$ is the XOR function.

Normal equation:

According to the same book on page 107, the weights can be obtained by solving the gradient of the loss function, which will result in a normal equation (5.12).

$\mathbf{w}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$

My Attempts:

Since it is a XOR function, we know that if the input is $\mathbf{X}=\begin{bmatrix}
    0 & 0 \\
    0 & 1 \\
    1 & 0 \\
    1 & 1 \\
\end{bmatrix}$, then the corresponding output will be $\mathbf{y}=\begin{bmatrix}
    0 \\
    1 \\
    1 \\
    0 \\
\end{bmatrix}$.
So I just plug everything into the normal equation as shown above.

However, the solution I get is
$\mathbf{w}=
\begin{bmatrix}
    \frac{1}{3} \\
    \frac{1}{3} \\
\end{bmatrix}$.

What am I doing wrong here? And also how to find the bias $b$?


texts: How to use random forest with large number of categorical features and categories?

texts: I have 2 features productName and productCategory , both of them are strings. I have a category named supplier. There are 4000 suppliers and 500,000 items in test data. I don't think one hot encoding will be a good approach to deal with such a big data. How should i handle these using random forest?


texts: Un-learning a single training example from a trained model

texts:
I was going through the paper "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" by google on which suggests best practices for models in production. In a section about privacy controls in the data pipeline it says:

  Finally, test that any user-requested data deletion propagates to the
  data in the ML training pipeline, and to any learned models.

I understand about the data deletion from the data pipeline but is it even possible to "un-learn" a single training example without retraining on the new data? They have mentioned in the paper that the practices are being used in google at some point or other, so there might be an efficient way but I'm unable to get any information on this.

I am looking for any literature on this or any ideas about how one would go on solving this problem.

Edit:
On further research, I found this paper which focuses on the specific problem. Though making a lot of assumptions they propose a method for k-means too. Looks like this is an upcoming research area and would require time to develop!


texts: confusing regarding to kmeans clulstering for data correlation

texts:
I am trying to think through my process before doing any real coding. However, got really confused easily.

Say I have 100 instruments and I know their price movements every day for a year. So I can create a movement matrix

A =[[I1-1, I2-1, .... I100-1],  (I1-1 is price for instrument 1 on day 1)
    [I1-2, I2-2, .... I100-2],
    ....
    [I1-365, I-2365, .... I100-365]
    ]

Then for each instrument, I can calculate a price movement correlation between other instruments for the whole year.

   C =[C1-2, C1-3,...C1-100,C2-3,....C99-100] (C1-2 is the price movement correlation between instrument 1 and 2 for the whole year)

Then I would like to apply a K-Means clustering algorithm to classify the correlation into say 10 categories. So in theory, I created 10 categories that the prices turned to move together.

However, the more I think about it, the more it is not correct. For example, if this is my Correlation result:

 C =[0.35, 0.59,...0.88(C1-100),0.48,....0.99(C99-100)]

isn't it K-Means clustering may classify C1-100, C99-100 in one cluster, and C1-2, C1-3, C2-3 in another cluster.

When I read that, it means instrument 1,100, 99 in one category, and instrument 1,2,3 in another category. But I would like each instrument only available in one category, so looks like there is a hole in my idea or maybe my idea is totally wrong?


texts: Gradient Boosting Tree: "the more variable the better"?

texts:
From the tutorial of the XGBoost, I think when each tree grows, all the variables are scanned to be selected to split nodes, and the one with the maximum gain split will be chosen. So my question is that what if I add some noise variables into the data set, would these noise variables influence the selection of variables (for each tree growing)? My logic is that because these noise variables do NOT give maximum gain split at all, then they would never be selected thus they do not influence the tree growth.

If the answer is yes, then is it true that "the more variables the better for XGBoost"? Let's not consider the training time.

Also, if the answer is yes, then is it true that "we do not need to filter out non-important variables from the model".

Thank you!


texts: Neural network - calibration strategy & cross-validation

texts:
I have a hard time articulating the different parts of a calibration process of a relatively vanilla neural network. I am mostly concerned with :

Grid search for the regularisation hyperparameter
Random weights initialisations
Cross validation

Currently I am doing n random weights initialisations for p hyper-parameters values. From those the hyper-parameter value is rather obvious. Then I usually choose the best of n random initilisations.
However I noticed that population on which the model is calibrated may have a significant impact on individual predictions (as random initialisations). It is my understanding that a k-fold cross validation could help.

How should I articulate those steps ? More specifically, do I need to do n * p * k calibrations ? Or can I get the hyper-parameter value before (from previous n * p runs) and do n * k calibrations ?


texts: Using Keras to Predict a Function Following a Normal Distribution

texts:
I am currently trying to predict a function that has a shape similar to that of a normal distribution.  It is defined as:

(4 γΩ )/((γ+2nγ)^2+(4Δ^2+2Ω^2))

I have tried to use relu, sigmoid, and tanh activation functions.

I have also tried mae, mse and binary_crossentropy loss functions.

I have also tried adam, rmsprop and sgd optimizers, and have also played around with learning rates.

Nothing seems to work. When I perform a regression analysis, I'm told that the error percentage, mse and rmse values are low.

But when I plot the predictions, it just shows a linear function.

Any suggestions?

My architecture looks like:

df=pd.read_csv('DataGen(N=1,Gamma=0.1,Omega=5).csv')
df.head()
dataset=df.values
X=dataset[:1800000,0].reshape(-1,1)
Y=dataset[:1800000,2].reshape(-1,1)
X2=dataset[1800001:,0].reshape(-1,1)
Y2=dataset[1800001:,2].reshape(-1,1)
scaler= MinMaxScaler(feature_range=(0,1))
X_min=scaler.fit_transform(X)
Y_min=scaler.fit_transform(Y.reshape(-1,1))

X_test=scaler.fit_transform(X2)
Y_test=scaler.fit_transform(Y2.reshape(-1,1))

seed=7
np.random.seed(seed)
X_tr,X_val, Y_tr, Y_val = train_test_split(X,Y,test_size=0.1, random_state=seed)

def NN():
model= Sequential()
model.add(Dense(4, input_dim=4, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(16, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(32, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(64, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(128, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(64, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(32, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(16, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(3, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(1,kernel_initializer = 'normal'))

model.compile(loss='mean_absolute_error', optimizer='adam', metrics=['MAE'])
return model

My plotted results look kind of like this:

The blue points are the expected values and the red points are predictions.


texts: How to integrate time series data into normal features for machine learning?

texts:
I confront a problem where one data source is a "normal" DF with customers as rows (each customer occurs once) and static customer features as columns. The other DF other hand is a big pile of timestamped data of the aforementioned customers with event features and event timestamps (each customer occurs many times according their activity). The task is a simple classification by customer.

My problem is how to integrate the time series data into the first DF as columns? Should I simply compute somekind of aggregations from the second DF? If so, what kind of? If not, what is the canonical way to merge these two kind of data sets?


texts: Classifying based on irregular number of features

texts:
Ok, so I am trying to classify a rather large data set where the training set has some peculiar issues...

There are a different number of features available for each row. For example, I might have 10 data points to work with for one prediction and only 3 for another. I would want to somehow null the empty data points because they aren't necessarily 0. We just don't have the data.

Each feature is a triplet of numbers (A,B,C), (A1,B1,C1), (A2,B2,C2). Should I separate these into individual features or is there a way to keep them grouped?

There are a few features/variables which all rows will have.

Is there preferred methodology?

| Y | v0        | v1        | v2      | v3     | v4 | v5 | v6 | v7 | v8 | v9 | A | B | C  |
|---|-----------|-----------|---------|--------|----|----|----|----|----|----|---|---|----|
| A | 15,0,7    | 18,0,2    | 23,0,0  | 21,0,3 |    |    |    |    |    |    | 1 | 0 | 5  |
| B | 15,.05,12 | 14,.05,12 | 9,.3,45 |        |    |    |    |    |    |    | 2 | 2 | 12 |
| A | 18,0,2    | 23,.03,0  |         |        |    |    |    |    |    |    | 0 | 1 | 3  |
| A | 23,0,0    | 18,0.07,2 | 15,0,7  |        |    |    |    |    |    |    | 3 | 1 | 1  |
| C | 4,.08,212 |           |         |        |    |    |    |    |    |    | 6 | 4 | 4  |
| B | 14,.05,12 | 9,.3,45   |         |        |    |    |    |    |    |    | 1 | 3 | 9  |


texts: Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?

texts: I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and low/under resource language. so is it possible to feed BERT to the seq2Seq encoder/decoder?


texts: GAN to generate a custom image does not work

texts:
I have been training a GAN in the cloud for some time now. I use Google's free credit. My laptop with a CPU doesn't seem to be up to the task.

The image I want to generate is this.

Even though the number of epochs is about 15000 I don't get anything close to the original.

This is the main code. I don't claim to fully understand the deep layers. It took a few days to even write this code. The rest of the code is boilerplate to train.

There is no compilation error and I look at the images using TensorBoard.

The output from the generator is (1024,1024) images. Should this be the same as my original which is a (299,299) images.

Should I calculate using formulas how each layer transforms the image to understand it better ?

How do I fix this ? I have mixed and matched API's just to create a working example assuming that doesn't create any problem.

X = tf.placeholder(tf.float32, shape=[None, 299, 299, 1], name='X')

Z = tf.placeholder(dtype=tf.float32,
                              shape=(None, 100),
                              name='Z')
is_training = tf.placeholder(dtype=tf.bool,name='is_training')

keep_prob = tf.placeholder(dtype=tf.float32, name='keep_prob')
keep_prob_value = 0.6

def generator(z,reuse=False, keep_prob=keep_prob_value,is_training=is_training):
    with tf.variable_scope('generator',reuse=reuse):
        linear = tf.layers.dense(z, 1024 * 8 * 8)
        linear  = tf.contrib.layers.batch_norm(linear, is_training=is_training,decay=0.88)
        conv = tf.reshape(linear, (-1, 128, 128, 8))
        out = tf.layers.conv2d_transpose(conv, 64,kernel_size=4,strides=2, padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)
        out = tf.nn.leaky_relu(out)
        out = tf.layers.conv2d_transpose(out, 32,kernel_size=4,strides=2, padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)
        out = tf.layers.conv2d_transpose(out, 1,kernel_size=4,strides=2, padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)
        print( out.get_shape())
        out = tf.nn.leaky_relu(out)
        tf.nn.tanh(out)
        return out

def discriminator(x,reuse=False, keep_prob=keep_prob_value):
    with tf.variable_scope('discriminator',reuse=reuse):
        out = tf.layers.conv2d(x, filters=32, kernel_size=[3, 3], padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.nn.leaky_relu(out)
        out = tf.layers.max_pooling2d(out, pool_size=[2, 2],padding='SAME', strides=2)
        out = tf.layers.conv2d(out, filters=64, kernel_size=[3, 3], padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.nn.leaky_relu(out)
        out = tf.layers.max_pooling2d(out, pool_size=[2, 2],padding='SAME', strides=2)
        out = tf.layers.dense(out, units=256, activation=tf.nn.leaky_relu)
        out = tf.layers.dense(out, units=1, activation=tf.nn.sigmoid)
        return out

GeneratedImage = generator(Z)

DxL = discriminator(X)
DgL = discriminator(GeneratedImage, reuse=True)


texts: Manual feature engineering based on the output

texts:
So, I'm working on a ML model that would have as potential predictors : age , a code for his city , his social status ( married / single and so on ) , number of his children and the output signed which is binary ( 0 or 1 ). Thats the initial dataset I have.

My prediction would be based on those features to predict the value of signed for that person.

I already generated a prediction on unseen data. Upon validation of the results with the predicted result vs the real data , I have 25% accuracy. While cross-validation gives me 65% accuracy. So I thought : Over-fitting

Here is my question, I went back to the early stages of the whole process, and started creating new features. Example : Instead of the code for the city which makes no sense to have that as input to a ML model, I created classes based on the percentage of signed, the city ,with the higher percentage of 'signed' ( the ouput ) , gets assigned to a higher value of class_city, which improved a lot in my Correlation Matrix the relationship signed-class_city which makes sense. Is what I'm doing correct? or shouldn't I create features based on the ouput?
Here is my CM
:

After re-modelling with 3 features only ( department_class , age and situation ) i tested my model on unseen data made of 148 rows compared to 60k rows in the training file.

First model with the old feature ( the ID of the departement ) gave 25% accuracy while the second model with the new feature class_department gave 71% ( Again on unseen data )

Note : First model with 25% has some other features as ID's ( they might be causing the model to have such a weak accuracy with the deparment_ID )


texts: Deploy pretrained model to simple web app

texts: As a small personal project, I wanted to try and deploy this image classification model with a simple web app (input image, output classification and heatmap). There are a couple pretrained models on that link. What is the absolute easiest way to do this (don't need production level capabilities)?


texts: While I was reading the paper "Grounded Action Transformation for Robot Learning in Simulation", I came across the term "policy". Could someone explain to me what that actually is (in general and in the particular context of the paper)?


texts: How can data science teams inside businesses measure costs and efficiency of their technical work?

texts: How can data science teams measure and improve costs of their technical work, when they often don't know the monetary value of the datasets and insights they are producing? Are they using industry based benchmarks for technical development, and some subjective measurement for business insight creation?


texts: Image classification: Strategies for minimal input count

texts:
I want to classify product images into 8 discrete classes. For several reasons the number of input images need to be as small as possible. Related to this I have 2 questions:

What strategies can I employ to reduce the number or required images
What is the magnitude of images per classes needed (A) in a simple scenario where the image is always isolated, see below or (B) in a complex scenario where the image is a scenery (also below). I know this is hard to tell in general but a rough number would be fine.

Additional information

Architecture

Currently a CNN seems to be most fitting for the problem. I am also thinking about a pre-trained network (removing the deeper layers of a powerful network such as AlexNet).

Data

Imagine I am trying to classify dinner tables by the type of legs they have. In the simple scenario images from 3 different classes could look like this:

Class: regular

Class: outward

Class: Wire

In a complex scenario they might look like this:

Techniques to reduce input data needed

There are a couple of ideas I have to reduce the need for data (please add your own):

Use a pre-trained network (good idea or not?)
Pre-process images (how? fiddle with image orientation?)
Does it make sense to change the images more deeply (such as chopping off the top 50% since I only care for the legs?)

I am looking forward for your thoughts!


texts:
I am new to Machine Learning and I have a conceptual question.

I have a scaled dataset (scikit-learn and pandas).

After training/testing my algo, I will make new predictions using new actual data which will not be scaled or normalized.

Will this discrepancy be a problem, if so, how should I resolve it?

Best,


texts:
The final step in dropout regularization is to multiply the weights by the dropout probability. This is motivated by analogy to bagging: averaging the weights of multiple nets. But it isn't truly that because of Jensen's inequality.

The effect of the averaging is basically just to shrink the weights by some factor. Why average at all? Has anyone tried simply not averaging, and controlling the magnitude of the parameters with e.g. L2 regularization?  One would still get the benefits of preventing codependence, but one would need to regularize some other way.


texts: How to tune / choose the preference parameter of AffinityPropagation?

texts:
I have large dictionary of "pairwise similarity matrixes" that would look like the following:

similarity['group1']:

array([[1.        , 0.        , 0.        , 0.        , 0.        ],
       [0.        , 1.        , 0.09      , 0.09      , 0.        ],
       [0.        , 0.09      , 1.        , 0.94535157, 0.        ],
       [0.        , 0.09      , 0.94535157, 1.        , 0.        ],
       [0.        , 0.        , 0.        , 0.        , 1.        ]])

In short, every element of the previous matrix is the probability that record_i and record_j are similar (values being 0 and 1 inclusive), 1 being exactly similar and 0 being completely different.

I then feed each similarity matrix into an AffinityPropagation algorithm in order to group / cluster similar records:

sim = similarities['group1']

clusterer = AffinityPropagation(affinity='precomputed',
                                damping=0.5,
                                max_iter=25000,
                                convergence_iter=2500,
                                preference=????)) # ISSUE here

affinity = clusterer.fit(sim)

cluster_centers_indices = affinity.cluster_centers_indices_
labels = affinity.labels_

However, since I run the above on multiple similarity matrixes, I need to have a generalised preference parameter which I can't seem to tune.

It says in the docs that it's by default set as the median of the similarity matrix, however I get lots of false positives with this setup, the mean sometimes work sometimes gives too many clusters etc...

e.g: when playing with the preference parameter, these are the results I get from the similarity matrix

preference = default # which is the median (value 0.2) of the similarity matrix: (incorrect results, we see that the record 18 shouldn't be there because the similarity with the other records is very low):

 # Indexes of the elements in Cluster n°5: [15, 18, 22, 27]

 {'15_18': 0.08,
 '15_22': 0.964546229533378,
 '15_27': 0.6909703138051403,
 '18_22': 0.12,    # Not Ok, the similarity is too low
 '18_27': 0.19,    # Not Ok, the similarity is too low
 '22_27': 0.6909703138051403}

preference = 0.2  in fact from 0.11 to 0.26: (correct results as the records are similar):

 # Indexes of the elements in Cluster n°5: [15, 22, 27]

 {'15_22': 0.964546229533378,
 '15_27': 0.6909703138051403,
 '22_27': 0.6909703138051403}

My question is: How should I choose this preference parameter in a way that would generalise?


texts: Best way to narrow down a list and rank based on attributes?

texts:
I have a mortgage/credit data set that contains a list of customers (600k rows) and has a 100 columns inclusive of the customer's general info (address, city, zipcode, etc), income, fico scores, number of current mortgages, mortgages in the past, aggregate mortgage amounts, number of bank card trades, etc. The data pertains to customers that are already good candidates to contact for issuing a credit product, however if one is to narrow down the list to 350K:

What would be the best method to rank the list to cut it down?

PS Your insights are much appreciated.


texts: Reason of use of Product rule of Probability in Predicting Error

texts:
P(x,y) = P(y|x)P(x)
Why do we use this in estimating expected prediction error?
i.e. E{(y - f(x))^2}
I researched and I came to know that it helps in figuring out noise but How?


texts: Applicable method for the clustering of time-series consisting of multiple events

texts:
I'm currently dealing with a time-series clustering problem at work and I need help with picking/suggesting the right methodology. The problem is similar to clustering of users on a website based on their actions.

Each user is uniquely identified by his IP (oversimplification) and is represented by a time series of actions. An action could be anything from going to homepage, adding an item into his shopping cart, sending a form. An action has other features - e.g. time at which it was done, or what items were added into the shopping cart.

One time series could therefore look like this:

User receives goes to https://www.example.com,
from there he goes into https://www.example.com/categories,
https://www.example.com/some-product and he adds this product into the cart.

Each action was separated by 5-10 seconds. The behaviour pattern is completely different from a crawler, which would look like this:
Homepage, Category1, Category2, Category3 ... CategoryN, each with exactly 2 seconds between any two events.

Is there a unsupervised learning method which could be used for clustering users into N groups, depending on the characteristics of their behaviour? The closest I got so far are the self-organizing maps, though they do not really put much emphasis on time.

Thanks, any suggestions are greatly appreciated.


texts: how much text data is required for a meaningful use of word2vec

texts:
how much data does word2vec require?  Are there any public data sets that are useful?

For example, could it be that 1000 newspaper articles are enough to use word2vec?

Here is a word2vec tutorial from Kaggle that uses 50,000 movie reviews.  I am trying to understand the scale of the word2vec input.


texts:
I am wondering how I can manage a test data after using PCA or normalization and another thing like that in the classification because our model works on the representation given by its input vectors. For example, suppose you have used PCA in your training dataset to gain better accuracy or you have normalized (min-max) data. Now, you have developed a model and want to install it and label the new coming samples. You need to somehow apply PCA to each coming record and normalize that record. Applying PCA to one record will not yield the same effect of the PCA  used in training phase and I think even it doesn't make sense to apply PCA to just one sample. So how can we manage these preprocessing techniques in the training phase in the test data, too?

Thanks in advance.


texts: How can I extract the reason of the legal compensation from a court report?

texts:
I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report)

Order mister X to pay EUR 5000 for compensation for unpaid wages

and

To cover damages, mister X must pay EUR 4000 to mister Y

I want to make an algorithm that is able from this sentence to extract the motive of legal compensation. For the first sentence

Order mister X to pay EUR 5000 for compensation for unpaid wages

the algorithm's output must be "compensation for unpaid wages" or "compensation unpaid wages ".

For the second sentence, the algorithm's output must be "cover damages". Output can be a string or a list of string, it doesn't matter.

As I'm not an NLP expert (but I have already worked on a project on sentiment analysis, so I know some stuff about NLP), and there are so many articles, I don't know where to start.

I'm working on French texts, but I can get away with working on English texts.


texts: What model can I build with a limited dataset?

texts:
I have a dataset consisting of purchasing history from an e-commerce website. The columns consist of customer id, product id, postal code, quantity of products purchased, date of the purchase. There are thousands of different customer ids, hundreds of product ids, and some million rows.

I ran ARIMA modelling for forecasting the purchased quantities of a given product.
Now I want to try other methods for analysing the dataset, but cannot find models which could fit the dataset well. What other models could I run in order to gain more insight on this data?


texts: Why labels are not used in outlier detection algorithms?

texts:
I read this article from sklearn: https://scikit-learn.org/stable/modules/outlier_detection.html

While these algorithms are very useful for outlier detection, I'm surprised to see that they are not using labels to find outliers.. Is it normal? Would it be not useful to use them?

  Outlier detection is then also known as unsupervised anomaly detection

Are there existing libraries which provide supervised outlier detections algorithms?

More explanations:

How to know if a data point is really an outlier without the label? Let say we want to predict the price of a house. An house having a lot of good features (huge size, big swimming pool, big garden, .) might be an outlier if the price is low, or as a normal point if the price is higher than average.


texts: How to get the weight matrices of intermediate layers in bidirectional recurrent neural networks?

texts:
I am a newbie in deep learning. I have a doubt regarding the training procedure of bidirectional recurrent neural networks using backpropagation through time. Following the original paper for bidirectional recurrent neural networks, https://ieeexplore.ieee.org/abstract/document/650093/ , I understand the forward and backward passes for training. One can access the hidden state for a layer after training the network comprising of both forward and backward recurrent networks. But, if one needs to access the intermediate weights of different layers in the bidirectional rnn, how does one do that?
I understand, there are popular deep learning APIs like Keras which can get access to layer outputs and weights, but how do they get the weights in the background when training a complex network like bidirectional RNNs through back-propagation is what causing my doubts?


texts: Not sure how I should interpret the scaling. Is it correct to convert the sparse matrix to a dense matrix by padding with 0's and scale normally?


texts: How to find the feature regions where each label is the most expected when using decision trees?

texts:
Given a decision tree for classification for example this one:

What is the way to find the feature domain (petal and sepal width and length) where a sample would most likely occur in the feature space for each class?
It is clear here that for Setosa it is when petal length is less or equal to 2.45.
However, where I am confused is how to think in more complex cases. For example, let's take Versicolor:
I am hesitating between 2 choices or take every path that leads to Versicolor or just choose the domain (considering the path) that leads to the leaf with the most samples.
I don't necessarily care about this example, I want to know the general case and how to think to solve that problem.
Thanks


texts: train_test_split : stratify can not be recognized?

texts:
I'm trying to set stratify = True, but somehow jupyter notebook says 'name 'y' is not defined'

X_train, X_test, y_train, y_test = train_test_split(
    loan.drop('Loan_Status', axis=1),
    loan['Loan_Status'], test_size=0.2, random_state=0, stratify=y)

Can anyone tell me what is the proper way to do it?

I'm using Scikit-learn v0.19.1 and have tried to set stratify = True / y / 2 but none of them worked.


texts:
Let's say I have streaming textual data coming in. I run named entity recognition software to capture the entities, then I score them using tf-idf. Tf-idf scores are unbounded positively. How can I come up with an intelligible threshold (that stays current over time) to filter out the "low scoring" tf-idf values?

Currently, I'm keeping track of the last 1000 values, then taking the 0.99 quantile and only keeping those, but it has been too conservative. I can drop the quantile to 0.9, but I would like something more principled.

Thanks!


texts: What's the relationship between an SVM and hinge loss?

texts: My colleague and I are trying to wrap our heads around the difference between logistic regression and an SVM. Clearly they are optimizing different objective functions. Is an SVM as simple as saying it's a discriminative classifier that simply optimizes the hinge loss? Or is it more complex than that? How do the support vectors come into play? What about the slack variables? Why can't you have deep SVM's the way you can't you have a deep neural network with sigmoid activation functions?


texts: Determine whether or not a company has acquired others using NLP

texts:
I'm trying to build a Python script to identify whether a company has acquired others. For this, I intend to seach Google for " acquisition", and parse the titles of the first N pages. Then, from the titles that have indicators that there was an actual acquisition, I want to extract the name of the company that was acquired, and I also want to provide a confidence metric. And obviously, it also has to make sure that it was the company I'm looking for that acquired another one, and not the other way around.

I have some experience with NER, but with this specific NLP task I don't know where to start. Is there something already implemented for such task?


texts: Neural network cost is constant never changing during training

texts:
I am trying to build a binary classifier to predict a pulsar star with Single Hidden layer Neural Network.

But the cost on training dataset after almost 100 iterations has no change, following is the implementation with python numpy.

import os
import csv
import numpy as np

def load_dataset(file):
    with open(file, 'r') as work_file:
        reader = list(csv.reader(work_file))
        total = len(reader)
        train_set = reader[:round(total * 0.8)]
        val_set = reader[:round(total * 0.2)]
        features = len(train_set[0][:8])
        x_train = np.zeros((len(train_set), features))
        y_train = np.zeros((len(train_set), 1))
        x_val = np.zeros((len(val_set), features))
        y_val = np.zeros((len(val_set), 1))

        for index, val in enumerate(train_set):
            x_train[index] = val[:features]
            y_train[index] = val[-1]

        for index, val in enumerate(val_set):
            x_val[index] = val[:features]
            y_val[index] = val[-1]

    return x_train, y_train, x_val, y_val

def activation(fun, var):
    val = 0.0
    if fun == 'tanh':
        val = np.tanh(var)
        # val = np.exp(2 * var) - 1 / np.exp(2 * var) + 1

    elif fun == 'sigmoid':
        val = 1/ (1 + np.exp(-var))

    elif fun == 'relu':
        val = max(0, var)

    elif fun == 'softmax':
        pass

    return val

def loss_calc(y, a):
    return -(np.dot(y, np.log(a)) + np.dot((1-y), np.log(a)))
    # return -(y * np.log(a) + (1-y) * np.log(a))

x_train, y_train, x_val, y_val = load_dataset('workwith_data.csv')
norm = np.linalg.norm(x_train)
print(x_train)
x_train = x_train/norm
print(x_train)
# Weights inititaed in trasponsed shape
# 0.001 is the ideal weights multiplier else log loss goes nan due to log 0 or -ve
w1 = np.random.randn(x_train.shape[1], 3) * 0.0001
w2 = np.random.randn(3, 1) * 0.01
# baises over layers
b1 = 0.0
b2 = 0.0
cost = 0.0
dw1 = 0.0
db1 = 0.0
dw2 = 0.0
db2 = 0.0
samples = x_train.shape[0]
lr = 0.01
for i in range(1000):
    # forward pass
    z1 = np.matmul(x_train, w1) + b1
    a1 = activation(fun='tanh', var=z1)
    z2 = np.matmul(a1, w2) + b2
    a2 = activation(fun='sigmoid', var=z2)
    loss = loss_calc(y_train.T, a2)
    cost =  np.sum(loss)/samples
    print(cost)
    # Backprop
    dz2 = a2 - y_train
    dw2 += np.matmul(dz2.T, a1)/samples
    db2 += dz2/samples
    tanh_diff = 1 - np.square(z1)
    dz1 = (w2.T * dz2) * tanh_diff
    dw1 += np.matmul(dz1.T, x_train)/samples
    db1 += dz1/samples
    w1 = w1 - lr * dw1.T
    w2 = w2 - lr * dw2.T
    print('iteration ' + str(i) + ' cost'+str(cost))


texts: Understanding the filter function in Convolution Neural Networks

texts:
I am trying to follow the following tutorial accessible with this link.
Under the 3rd Heading, "3. Visualize the Activation Maps for Each Filter", we can see the following function:
def apply_filter(img, index, filter_list, ax):
    # set the weights of the filter in the convolutional layer to filter_list[i]
    model.layers[0].set_weights([np.reshape(filter_list[i], (4,4,1,1)), np.array([0])])
    # plot the corresponding activation map
    ax.imshow(np.squeeze(model.predict(np.reshape(img, (1, img.shape[0], img.shape[1], 1)))), cmap='gray')

I understood what they are trying to do. They are applying the filters and trying to show the output after that. But, what I don't understand is the following line:
model.layers[0].set_weights([np.reshape(filter_list[i], (4,4,1,1)), np.array([0])])

What does it mean to assign weights here and also, why are they reshaping the filter which is of 4*4 to 4*4*1*1?


texts: Recurrent Neural Net (LSTM) batch size and input

texts:
I am working in Keras to build LSTM models. I understand that setting STATEFUL=FALSE means that the different batches are treated as independent when training the model.

Suppose I want to build a model that uses 6 weeks of hourly temperature observations in order to predict tomorrow's 24 hourly temperatures. I have been using a batch size of 24, meaning each day of training data gets used independently by the LSTM when training. But this means that I am not allowing the RNN to pick up longer correlations across days, and only within days, correct? So should I instead be using a batch size of 6*7*24=168? Would this larger batch size allow for the LSTM to pick up longer range correlations across the 6 week training period?

Secondly, I have been using the previous day's observations $y_{(t-24)}$ as predictors for the current day's temperature (at the same time period) $y_t$. Is this a valid approach, or is this redundant with the long term correlations that the LSTM will be looking for?


texts: How to create custom Activation functions in Keras / TensorFlow?

texts:
I'm using keras and I wanted to add my own activation function myf to tensorflow backend. how to define the new function and make it operational. so instead of the line of code:

model.add(layers.Conv2D(64, (3, 3), activation='relu'))

I'll write

model.add(layers.Conv2D(64, (3, 3), activation='myf')).


texts: Adding high p-value and low R square features in linear regression model to improve result

texts:
I am working on a linear regression problem. The features for my analysis have been selected using p-values and domain knowledge.  After selecting these features, the performance of $R^2$ and the $RMSE$ improved from 0.25 to 0.85. But here is the issue, the features selected using domain knowledge have very high p-values (0.7, 0.9) and very low $R^2$ (0.002, 0.0004). Does it make sense to add such features even if your model shows improvement in performance. As far I know, according to linear regression, it is preferable to only keep features with low p-values.

Can anyone share their experience? If yes, then how can I back up my proposal of new features with high p-values.


texts: Is ensemble learning using different classifier combination another name for Boosting?

texts:
For implementation I am following the Matlab code for AdaBoost. Based on my understanding, AdaBoost uses weak classifiers known as base classifiers and creates several instances of it. For example, a weak classifier is a decision tree. So, AdaBoost can create maximum N decision trees (where N = number of samples) and combine the prediction results. This is a Homogeneous boosting method. But I have seen some examples such as this one in Matlab and ensemble-toolbox which have confused me. Can somebody please explain the following concepts with respect to the implementations and what is going on in the code?

1) Does the Matlab code for AdaBoost combine different classifiers? The combination method is unclear to me - -whether they do sum or majority voting or something else.

If they are combining several classifiers, then technically it is a Heterogenous ensemble method and the term for it is stacking and not boosting. Please correct me where I am wrong.
In Boosting methods,  the based classifiers are the same. But the given in Matlab code for AdaBoost combines different classifiers, I am not sure.

2) Is ensemble learning or the example in the ensemble toolbox the same as the Adaptive Boosting Matlab code (second link)? Is ensemble learning the same as Adaptive boost?


texts:
Context: I'm trying to optimize a portfolio of assets for some arbitrary goal. Currently, I'm multiplying the matrix representing the different assets with the matrix representing the weights (taken from a cartesian product). Thereafter, each portfolio's performance is evaluated to find the optimal weights.
Unfortunately, this has been proven to be extremely time consuming for anything other than very small portfolios.
Question:
What are some fas ways to multiply relatively small matrices such as [500 x 100][100 x 1] billions of times?
What are some good libraries?
Is there any way to accelerate it with a CUDA-enabled GPU?


texts: Using Google Translate API to create a Translation Dataset

texts:
Is it a good idea? ;-)
Is it legal to do so? Is it legal to release such a dataset to public?

Say I have a language X for which I want to create a dataset for translation to/from English, for which I have no online sources to scrape out the data from. But our giant Google Translate somehow has the support for language X.

Can we just dump the translations from English datasets for my language X using Google Translate API and create a rough dataset to start with?


texts: state-action-reward-new state: confusion of terms

texts:
My question may sound like a duplicate of, for example, How is that possible that a reward function depends both on the next state and an action from current state?
but I still feel confused.

In neural network approximation of the Q function, I follow the experience replay routine. Papers and manuals suggest storing state, action, received reward, and next state information in the experience replay. However when one is about to calculate a maximum of Q value based on the next action, which state do they use: state or next state? If the next state is used to calculate the max Q, then what is the whole purpose of storing the previous state and action information?

An example (2D world):

An agent is at cell A1 (state). His goal is to get to C3 to get a positive reward. He then moves to B2 (action); a received reward is, let's say, 0. Next state is B2 (after the action was done).

Questions:

Should one rely on the B2 state to iterate over possible actions from this state (next state) to get an approximation of highest reward (max Q)?

Then, why do we store the A1 and move-to-B2 information at all in the replay buffer?

Or I am wrong and we just use the A1 and iterate over possible actions (including that to B2) to get the max Q?

Edit: I think I have found an answer ). We need to store previous state (A1) and action (move to B2) in order to create the state-action distribution, which will be met with the expected long-term reward distribution, that we get after the next state routine. Right?


texts: Correct Way of Displaying Features in Decision Tree


texts: LONG SHORT-TERM MEMORY Hochreiter's paper BPTT

texts:
I was trying to read paper about LSTM, and I am stuck with mathematical problem.

http://www.bioinf.jku.at/publications/older/2604.pdf

page 4.

see,

|$f'_l$$_m$($net_l$$_m$)$w_l$$_m$ $_l$$_m$$_-$$_1$| is part of total error back flow.

|$f'_l$$_m$($net_l$$_m$)$w_l$$_m$ $_l$$_m$$_-$$_1$| is used for explaining about gradient vanishing, and exploding.
(If |$f'_l$$_m$($net_l$$_m$)$w_l$$_m$ $_l$$_m$$_-$$_1$| > 1.0, gradient will explodes.)

$f_l$$_m$ is the logistic sigmoid function, then the maximal value of $f'_l$$_m$ is 0.25(which is alright), then |$f'_l$$_m$($net_l$$_m$)$w_l$$_m$ $_l$$_m$$_-$$_1$|  takes on maximal values where

$w_l$$_m$ $_l$$_m$$_-$$_1$ = $1/y^l$$^m$ $^-$ $^1$  $coth$(1/2net$_l$$_m$)

goes to zero for |$w_l$$_m$ $_l$$_m$$_-$$_1$| -> ∞

here is what I don't understand.

1.where is $coth$ coming from?

if |$w_l$$_m$ $_l$$_m$$_-$$_1$| is infinite, then why is |$f'_l$$_m$($net_l$$_m$)$w_l$$_m$ $_l$$_m$$_-$$_1$| = 0?


texts: How to plot centroids and clusters resulting from a KMean model based on a text variable

texts:
I hope you can help as after several attempts, I'm no longer sure I can get a decent result.
I have a text corpus made of several documents, like the one below (which has been simplified for the sake of the post, bear with me but I didn't test this specifically):
corpus = ["is this good?", "this is bad", "some other text here", "i am hero", "blue jeans", "red carpet", "red dog", "blue sweater", "red hat", "kitty blue"]

I have applied a TD-IDF from sklearn also using the tokenizer parameter to which I have attached my cleaning/parsing text routine.
The resulting vector has been ran past to identify the elbow and finally to the KMean model to get the unsupervised clusterization.
vectorizer = TfidfVectorizer(....)
X = vectorizer.fit_transform(corpus)

model = KMeans(n_clusters=11, random_state=1021)
clusters = model.fit_predict(X)

All in all it seems to work, though I may still have doubts on the quality of the output, likely due to the limited numbers of documents (50 in my case).
ISSUE
Where I'm struggling is the visualisation part. I'd like to scatterplot both the clusters and the centroids in a unique graph to show somehow where these nodes where and what the center is.
By doing a cluster_distance = model.transform(X), the resulting 11 dimension array contains a single value that is not of any purpose, because scatterplots require both an X and Y.
If I have a look at X, my 50 X text_features matrix, is just a weight map of terms in the documents.
As far as I can see there is no one method that does what I need.
I therefore moved into considering some sort of data transformation to reduce the dimensionality.
I've tried two different approaches:

TSNE (from sklearn.manifold import TSNE)
PCA (from sklearn.decomposition import PCA)

But I have some perplexities.

With the TSNE I was able only plot the clusters, thought there is not such an evident split (in. my case).

See an extract of the code:
from sklearn.manifold import TSNE

tsne_init = 'pca'  # could also be 'random'
tsne_model = TSNE(n_components=2, random_state=1021, init=tsne_init, perplexity=20,
         early_exaggeration=4, learning_rate=1000)

centroids = model.cluster_centers_
transformed_centroids = tsne_model.fit_transform(centroids)
# print(transformed_centroids)

# plt.scatter(transformed_centroids[:, 0], transformed_centroids[:, 1], marker='x')
Y = tsne_model.fit_transform(X.toarray())
# fitted = tsne_model.fit(Y_sklearn)

plt.scatter(Y[:, 0], Y[:, 1], c=predicted_values, s=60, cmap='viridis')
plt.scatter(transformed_centroids[:, 0], transformed_centroids[:, 1],c='red', s=150, alpha=0.4)

With the PCA I was more or less able to get what I want, but to achieve the output, the input has to be fitted and transformed again, hence the cluster assignment changes from the original one.

# FROM https://towardsdatascience.com/k-means-clustering-8e1e64c1561c

from sklearn.decomposition import PCA
sklearn_pca = PCA(n_components=2)

Y_sklearn = sklearn_pca.fit_transform(X.toarray())

fitted = model.fit(Y_sklearn)

predicted_values = fitted.predict(Y_sklearn)
plt.scatter(Y_sklearn[:, 0], Y_sklearn[:, 1], c=predicted_values, s=60, cmap='viridis')

centers = fitted.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1],c='red', s=150, alpha=0.4)

My questions in using the latter are:

Would that be ok discharging the results of the original Kmean clusterisation and take the PCA ones (simply because of my visualization needs)?
Or would that be "safe" plot something but report the clusterization of something different?

Thanks for your help.


texts: Does Sampling size matters in Multi classification Model

texts:
I am working on a multi class classification model where few of the class are with less data compare to other classes.
I used random sampling technique to create a sample from the population keeping the proportion of each class equal to that of population.
For example, class A has 400 records in the population and class B has 100 records in the population then when doing random sampling I am creating a sample where records of class A and class B are in proportion of 4:1.
The trend I have observed is by changing the sample size (keeping inter class proportion constant) of one class leads to change in model performance (accuracy,precision,recall).

What technique do i need to apply in order to make my model stable irrespective of sample size?


texts: LabelEncoding selected columns in a Dataframe using for loop

texts:
I have certain columns in my dataset that are "object" type, so first I found them and now I want to transform from categorical to numerical data.
How can I do it in multiple columns using a for loop? I've been struggling with it. I don't want to repeat code.

I've been trying this, but it doesn't work

for column in object_columns:
    df.loc[:,column] = labelencoder.fit_transform(df.loc[:,column])


texts: How to do feature analyzing : pandas groupby(). mean

texts:
I'm analyzing Titanic data from Kaggle1 with a kind of guiding book. In the book, feature analyzing about the relationship between Pclass data and Survived data is done like below.

train_set = pd.read_csv('train_csv)
fig = plt.figure(figsize=(12,4))
ax1 = fig.add_subplot(121)
PclassPlot = train_set['Survived].groupby(train_set['Pclass]).mean()
ax.bar(x=PClassPlot.index, height=PClassPlot.values)

Why do you need to use mean data?
I thought I didn't need to culcalate mean to see the relationship between Pclass and Survived data.

Please give me tips for analyzing data.
Thank you for your help.


texts: Which outlier detection algorithms give a breakdown of the contribution from each feature?

texts:
I am looking for an algorithm that outputs a breakdown of which features contributed the most towards a data point being labelled as an outlier.

It can be supervised or unsupervised.

At the moment, I'm using isolation forest which is really good at spotting the outliers in my data set but it's really hard to figure out why they were picked when you have hundreds of features.

I'm using python/pandas/etc.


texts: Is it appropriate to use random forest not for prediction but to only gain insights on variable importance?

texts:
For a binary classification problem, I have a small data set with 200 observations. There are around 20 potential variables but based on variance importance I think only 2 or 3 are important for classification. This data set is too small to train a random forest model for prediction purposes, but is it okay to run a random forest model and use the variable importance feature to understand which variables are important? If only a handful of features are by far the most important, then, I think despite the small data size, it's an appropriate approach in that it'll tell me what I want. The results can then be used to perhaps build a simple decision tree or as a preliminary analysis for a future larger test.
Any flaws with this thinking?


texts: What is the interpretation of the expectation notation in the GAN formulation?

texts:
I'm confused about the expectation notation in the context of GAN loss functions.

The GAN loss for the discriminator is binary cross-entropy. ie: is this real or not.

real = $D(x)$ (ie: give the discriminator a real image)
fake = $D(G(z))$  (ie: generate a fake image and ask discriminator what it is)

Then the binary crossentropy is:
$$log(p) - log (1-p)$$

When used as a GAN loss we replace p with either "real class" or "fake class".

$$log(real) - log (1-fake)=\\
log(D(x)) - log (1-D(G(z)))$$

So far, this is ok (i think haha).

But the actual formulation adds an expectation sign... which I don't understand why it's there.

$$E_{x~data}log(D(x)) - E_z log (1-D(G(z)))$$


texts: convert list of tuple of tuple to list of tuple in pySpark


texts: Sharing parameters of an activation across layers of a neural network

texts:
Keras now provides advanced parametric activation layers like Leaky-ReLU PReLU.
Each time I add this layer to a sequential model, an additional trainable parameter is added to graph.
How can I make sure the trainable parameter of activation is shared across all layers.
Thanks