texts: Is there an intuitive interpretation of precision always higher than recall?

texts: I have a multiclass-classifier whose macro-precision is always greater than macro-recall. I suppose it means false negatives outnumber false positives in general. Is there an intuitive interpretation about this?


texts: Why the sigmoid activation function results in sub-optimal gradient descent?

texts:
I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford.  She says that because the output of sigmoid is always positive, that any gradients flowing back from a neuron following a sigmoid will all share the same sign as the upstream gradient flowing into that neuron.  She then says that a consequence of these weight updates sharing the same sign is a sub-optimal zigzag gradient descent path.
I understand this phenomenon when zoomed in on a single neuron.  However, since upstream gradients flowing into a layer can be of different signs, it's still possible to get a healthy mixture of positive and negative weight updates in a layer.  Therefore, I'm having trouble understanding how using sigmoid results in this zigzag descent path, except for in the case where the upstream gradients are all of the same sign (which intuitively seems uncommon).  It seems to me that if this suboptimal descent is important enough to be highlighted in the lecture, that it must be more common than that.
I'm wondering if the issue is "reduced entropy" among the weight updates, rather than all weight updates in the network sharing the same sign.  That is, zigzagging in a subset of the dimensions.  For example, say a network using sigmoid has four weights in a layer with two neurons: w1, w2, w3, and w4.  The updates to w1 and w2 could be positive, while the updates to w3 and w4 could be negative if the two upstream gradients differ in sign.  However, it wouldn't be possible for w1 and w3 to be positive, and w2 and w4 to be negative.  Is this the limitation of sigmoid that the Stanford lecture is referring to, assuming the second combination of weight updates was the optimal one?


texts: How to interpret axis of Histogram and distribution curve?

texts:
After doing statistical analysis of data I got the below figures:

The data contains a line length in meters which is represented with Line(m). The purpose here is to confirm the distribution behavior of line lengths(m). Now, based on above two figures:

What kind of information x-axis and y-axis of both figures convey? Particularly for the distribution curve (pdf)?. From the first figure, it is somehow clear that it shows the percentage of the data fall within that region.
The x-axis of second figure shows the values from [0 - 80]. Does this means line length (m)?


texts: Selecting audio pre-processing parameters for ASR

texts:
As far as I understand, audio in many speech recognition systems goes through the following pre-processing steps:

Slice audio stream into frames of size frame_size with step frame_step.

Compute spectrograms for each frame using STFT.

Compute Mel-scaled filter banks and compress them using the log function.

The above can be represented with the following code:
sample_rate = 16000 # 16kHz
frame_size = int(round(sample_rate * 0.025)) # 25ms
frame_step = int(round(sample_rate * 0.010)) # 10ms

fft_length = 1024

power_spectro = tf.math.abs(tf.signal.stft(audio,
    frame_length=frame_size, frame_step=frame_step, fft_length=fft_length
))

num_mel_bins = 40
lower_edge_hertz = 80
upper_edge_hertz = 8000

mel_weight_matrix = tf.signal.linear_to_mel_weight_matrix(
    num_mel_bins=num_mel_bins, num_spectrogram_bins=fft_length // 2 + 1,
    sample_rate=sample_rate,
    lower_edge_hertz=lower_edge_hertz, upper_edge_hertz=upper_edge_hertz
)

filter_banks = tf.tensordot(power_spectro, mel_weight_matrix, axes=1)
logmel_spectro = tf.math.log(filter_banks + 1e-8)

I can think with some guidelines for selecting sample_rate, frame_size and frame_step, as well as lower_edge_hertz and upper_edge_hertz.
My question is: what are the rules/guidelines for selecting fft_length and num_mel_bins?


texts:
I am currently trying to get a vocabulary for BoW-vector generation out of a set of 200k scientific abstracts.

I do some basic filtering of tokens already like lowercasing, stop-word-removal, stemming, not taking tokens with size < 2, leaving tokens out that can be converted to a number and so on. But still I count more than 121k distinct tokens, which seems like a lot to me.

As I am quite new to all this I am wondering if there exist guidelines for how big such a vocabulary should be in average, maybe even depending on the originating field.


texts:
I have a questions regarding the training of my final machine-learning model.
I trained the model using k-fold cross validation and reduced the learning rate with keras ReduceLROnPlateau function by a factor of 0.2.

Now I want to train my model on the whole dataset and I am wondering how I should use reduce on plateau now, as my validation set has gone.

The only set left is my test set. Should I monitor the validation loss of the test set instesd during training of the whole model?

Thanks in advance,

Best regards,

Michael


texts: neural networks: activation with tanh is an angle?

texts:
I heard that the activation could be thought of as a unnormalized angle between input and nodes.

How is this true?

My idea is that:

We have an input vector $\vec{x}$ per feature (?). This then gets multiplied with a dot product to a weight vector connecting the input to the neuron $\vec{w}$ and thus we have $\vec{x} \cdot \vec{w}$.

Now the dot product has to do with directions, such that it is minimized when they are opposite, and it is maximized when they are in the same direction. Thus it also has to do with the angle:

$\vec{x} \cdot \vec{w} = |\vec{x}| |\vec{w}| \cos{\theta}$

where $\theta$ si the angle between them. After the dot product we can take the $\tanh$ activation function and we would have
$\tanh(\vec{x} \cdot \vec{w}) = \tanh(|\vec{x}| |\vec{w}| \cos{\theta})$

This is the further I could get. Any idea?

Also, is it correct that in the neural network we put a vector per each feature? I mean I know we should put an input neuron per each feature, so does that mean that each input  neuron holds a vector? Or is the vector formulation a shortcut for the entire network?


texts: Why does averaging over vector values cause errors?

texts:
As a way to improve my model, I want to average GloVe vectors over a sentence. However, I can't get np.mean to work. The following code works when not averaging over words. (copied from other code)

embeddings_dict = {}
with open("glove.6B.50d.txt", 'r') as f:
    for line in f:
        values = line.split()
        word = values[0]
        vector = np.asarray(values[1:], "float32")
        embeddings_dict[word] = vector

g = open("input.txt", 'r')

vector_lines = []

for line in g:
    clean_line = line.translate(str.maketrans('', '', string.punctuation))
    array_line = clean_line.lower().split()
    vec_line = []
    i = 0
    for word in array_line:
        i += 1
        try:
            vec_line.append(embeddings_dict[word])
        except:
            vec_line.append(embeddings_dict["unk"])
    while i < 30: //pad up to thirthy words with zero vectors
        vec_line.append(np.zeros(50))
        i += 1
    vector_lines.append(np.asarray(vec_line))

X = np.asarray(vector_lines)

To average over words, I'm modifying a small part of the code

        try:
            vec_line.append(embeddings_dict[word])
        except:
            vec_line.append(embeddings_dict["unk"])

    #padding is not necessary anymore
    #while i < 30:
    #    vec_line.append(np.zeros(50))
    #    i += 1

    vec_mean = np.mean(vec_line, axis=0, keepdims=True)[:,None]
    vector_lines.append(vec_mean)

X = np.asarray(vector_lines)

This gives me the error "ValueError: could not broadcast input array from shape (1,50) into shape (1,1)". It feels as if I have tried every possible modification to this code as I could, but I keep on getting shape issues. What is causing all of these issues?


texts: Combinatoric system which using in bookies "S3" system

texts:
I am interested what type of combinatorics is using for following bookmakers system called "S3":

We have  N={1..8} events
We build express pairs C=8 like

[(1, 2, 4), (1, 3, 6), (1, 6, 8), (2, 3, 5),
 (2, 5, 8), (3, 7, 8), (4, 5, 7), (4, 6, 7)]

Each event repeats 3-times: 1 =>  (1, 2, 4), (1, 3, 6), (1, 6, 8)..

It's not permutation, combination... Please advice what is it?

I want as example generate in similar way 4 from 10 if this possible. After I'd like to predict the optimal way for building the `pairs'


texts: Is there any way to calculate the true,false positives and negatives for a regression problem

texts: I am trying to predict the glucose values of the patients for example values like 45,256,115 etc. based on some features. Currently I am calculating the accuracy in means of RMSE,MSE,R². Is there any way to calculate the accuracy in means of a confusion matrix by setting a threshold value like ±10 for the value predicted. For example if the actual value is 110 and my prediction is 100 then we can tell that's it's kind of accurate since 100+10=110 where 10 is the threshold value. Any suggestions and some code to elaborate it will be very helpful.


texts: How to measure the similarity between two text documents?

texts:
Assume, I have 100 text documents, and I want to cluster those documents.

The first step is the construct pairwise similarity matrix 100X100 for the documents

My question is:

what are common way to measure similarity between two documents?

Thanks,


texts: anomaly detection - conceptual help on machine learning

texts:
I am working on Anomaly detection model problem for a finance data set -  set of card transactions. My team member suggested an idea that
" First train the model with normal instances of data and to the trained model pass the anomalous data during testing to see whether the model will be able to detect the anomaly or not."
Is it possible like this with any Machine Learning algorithms. If so, can anybody suggest what algorithms can be used for this idea?

An additional question: How about the situation when there is no properly labelled fraudulent data and we have just transactional information. Does semi-supervised learning works? I do not have experience in this type of ML approach. Can anybody suggest good resources for this and also suggest can Semi-supervised learning or hybrid supervised learning works for anomaly/ fraud detection?


texts: Is there a quick way to check for multicollinearity between categorical variables in R?

texts: I have a large amount of categorical and dummy variables (36) and I would like to remove a number of them based on their multicollinearity (or just collinearity). Instead of using Chi Square tests over and over again, are there any functions that can check for (multi)collinearity in my variables and return variables with multicollinearity (or collinearity)?


texts: Metrics to evaluate features' importance in classification problem (with random forest)

texts:
I want to evaluate the importance of each of the features of a 2000x60 dataset  in a classification problem with random forest.

The most widely used ones apparrently are:

Cross Entropy-Information Gain
Gini Importance (SkLearn implementation with feature_importances_)
Mean Squared Error (H2O implementation with h2o.varimp)

I have also found a rather concise overview of some other metrics for variables' importance at random forests at this research paper.

These are the following:

Altmann
Boruta
Permutation
Recurrent relative variable importance
Recursive feature elimination
Vita
VSURF

Has anyone used these and which one was the most informative for his/her model?

Do you have any other metrics of this kind for variable importance at random forests?


texts:
I am a beginner to Orange GUI data mining software and I have some questions related to that.
First, I would like to know if there is any search function so that I can search if a column contains a specific word.
Second, does Orange has any functions for dimension reduction?


texts: Errorbars on a weighted average (of a non gaussian distribution)

texts:
I have some datapoints $x_i$ and errors $\sigma_i$ which are the absolute value of some datapoints $w_i$ with (supposed) gaussian errorbars (the $\sigma_i$). The $x_i$ are amplitudes so we did not care the sign.

I need to compute the average (and weighted by $1/\sigma_i^2$ average) of them. And I need the errorbars on this mean.

How can I compute it if I don't have the values $w_i$ but just $x_i$?

I mean if this was just the error of gaussian distributions (i mean for the  $w_i$ ) I would use the  weighted mean for this formula:

$$ \mu = \frac{\Sigma w_i \sigma_i^{-2}}{\Sigma \sigma_i^{-2}}$$

And for the error $$ \sigma = \sqrt{\frac{1}{\Sigma \sigma_i^{-2}}}$$ (as in https://ned.ipac.caltech.edu/level5/Leo/Stats4_5.html)

But what about in this case with the $x_i$?


texts: Does building a corpus make sense on a documentation project?

texts:
I have zero to experience in data science or machine learning. Because of this I am not able to determine if building a corpus does apply to the problem I am trying to solve.

I am trying to build a reference site for cloud technologies such as AWS Google Cloud.

I was able to build structured data and identify primary entities with in a single ecosystem using standard web scraping and sql.queries.

But I wanted to have the ability to have a mechanism that can autonomously identify entities and related information that is relevant to that entity and other entities it has relationships with.

Given that a specific ecosystem documentation follows a certain style can I use few entities as training docs and then have it classify the information I mentioned above.

Is the starting point to this is to build a corpus? I tried it out nltk categorized corpus builder.

Is it fine to include a specific document in multiple categories? For example an instance in AWS can be in category ec2 and a general category Computing unit

Anyways is this problem I am trying to solve fit into the general NLP ML space?


texts: Why are character level models considered less effective than word level models?

texts: I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word embeddings need a huge vocabulary while character level models need very less vocabulary, so why is it not considered?


texts:
I am currently learning the concept of neural networks by myself. I am working with a very good pdf from http://neuralnetworksanddeeplearning.com/chap1.html

I also did a few exercises but there is one exercise I really don't understand.

Task: There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01.

I found also the solution, as can be seen in the second image

I understand why the matrix has to have this shape, but I really struggle to understand the step, where the user calculates

0.99 + 3*0.01
4*0.01

PS: I know that the question was answered in Extra output layer in a neural network (Decimal to binary). However, the specific step I am looking for was not answered.


texts: What is the correct procedure when "joining" data takes ~6 hours?

texts:
I am dealing with bike-share data. I have 2 DataFrames:

trips_df (subset shown), total entries = 1,048,568

weather_df (subset shown), total entries = 2,654

I am trying to calculate and attach the total_precipitation for each trip, as a column. I do this by looking up the start_timestamp and end_timestamp datetime for each trip from trips_df, in the weather_df, and summing the precipitation_amount within those times, then attaching that value back in the trips_df under the new column. I can attach the code if it's helpful.

I ran the code on a subset of 65 entries and it took ~1.3s. (CPU times: user 1.27 s, sys: 8.77 ms, total: 1.28 s, Wall time: 1.28 s). Extrapolating that performance to my entire data, it would take (1.3 * 1048568)/65 = 20971.36seconds or 5.8hours.

What am I supposed to do in this situation? For context, this is a Kaggle style data science project so I'll have to do further data wrangling, and data extraction then apply a predictive model.


texts:
I just came through the TensorFlow Quantum library they introduced Quantum Convolutional Neural Network.

What is Quantum Convolutional Neural Network?
What is the difference between CNN vs QCNN?


texts: Distinguish randomly generated texts from reasonable for human texts

texts:
I have strings short texts of 2 types:
'23jd2032n0d2mn', 'fn830n30rn83', 'fhui29n4ok', 'qn4foml', ...
and
'sweetie23', 'king3prussia', 'maryjesus', 'lovedog4and_kitties', ...
Is there a way to distinguish one type from another?
I've tried to vectorize texts with word2vec and classify on these vectors with xgboost, but I didn't succeded to achieve got F1-score.


texts: Why is Reconstruction in Autoencoders Using the Same Activation Function as Forward Activation, and not the Inverse?

texts:
Suppose you have an input layer with n neurons and the first hidden layer has $m$ neurons, with typically $m < n$. Then you compute the actication $a_j$ of the $j$-th neuron in the hidden layer by

$a_j =  f\left(\sum\limits_{i=1..n} w_{i,j} x_i+b_j\right)$, where $f$ is an activation function like $\tanh$ or $\text{sigmoid}$.

To train the network, you compute the reconstruction of the input, denoted $z$, and minimize the error between $z$ and $x$. Now, the $i$-th element in $z$ is typically computed as:

$
z_i = f\left ( \sum\limits_{j=1..m}  w_{j,i}' a_j+b'_i \right)
$

I am wondering why are the reconstructed $z$ are usually computed with the same activation function instead of using the inverse function, and why separate $w'$ and $b'$ are useful instead of using tied weights and biases? It seems much more intuitive to me to compute the reconstructed with the inverse activation function $f^{-1}$, e.g., $\text{arctanh}$, as follows:

$$
z_i' = \sum\limits_{j=1..m} \frac{f^{-1}(a_j)-b_j}{w_{j,i}^T}
$$

Note, that here tied weights are used, i.e., $w' = w^T$, and the biases $b_j$ of the hidden layer are used, instead of introducing an additional set of biases for the input layer.

And a very related question: To visualize features, instead of computing the reconstruction, one would usually create an identity matrix with the dimension of the hidden layer. Then, one would use each column of the matrix as input to a reactivation function, which induces an output in the input neurons. For the reactivation function, would it be better to use the same activation function (resp. the $z_i$) or the inverse function (resp. the $z'_i$)?


texts: Can previous successful actions be used as input to policy model in contextual bandits?

texts:
In a recommender application, I apply contextual bandits using logged propensity scores similar to this.
The model is retrained daily.
The application recommends images on an e-commerce website. Each image can contain multiple products.
The action is displaying an image.
The context contains the following: previous product views, add-to-carts and purchases.
Is it causes any problem if I add the previously clicked images (from the recommendation panel) to the context? In other words: Is it ok to add previously clicked images to the model input?


texts: What is the best practice to test a ETL pipeline?

texts:
In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure the stability of the software.

A ETL pipeline, as a piece of code, should also go through these testing steps to build a healthy system.

However due to the nature of ETL process, traditional testing technique may not be applicable.

Is there any reference or guideline specifically focus on testing on ETL pipeline?


texts:
In order to establish an overall rating for a product from a series of user ratings (from 1 to 5), I thought that the median would be a good idea so that extreme values would not have too much influence. But in doing so, it is hard to rank products since they will all have a whole ranking.
So I thought about averaging the mean and the median. Is this a known measure? Is it relevant in this case?


texts: Is there any advantage in using Particle Swarm Optimization for clustering than K-Means?

texts: I have read some paper about using particle swarm optimization. It doesn't look give much different result than K-Means. I tried to use PSO for clustering but the result is pretty much the same with K-Means with some drawbacks like longer execution time and have a lot of different result caused by the random factor.


texts:
Consider the following code for Conv1D layer
# The inputs are 128-length vectors with 10 timesteps, and the batch size
# is 4.
input_shape = (4, 10, 128)
x = tf.random.normal(input_shape)
y = tf.keras.layers.Conv1D(32, 3, activation='relu',input_shape=input_shape[1:])(x)
print(y.shape)
(4, 8, 32)

It has been given that there are 10 vectors, with each of length 128. Then how does the output will be of shape (8, 32)?
If we apply a filter of size 3, we will then get a vector of length 126, if stride is 1. But, I cannot see 126 anywhere in the output.
How to understand the shapes of input and output?


texts: Xavier initialisation vs He initialisation

texts:
After reading the famous paper, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, I understand two things:-

He initilization borrows on the benefits of Xavier initialization except that the latter expected a linear activation and the prior accounts for ReLU non-linear activation. Infact they differ just by a factor of sqrt(2).
All the fuss is about layers having 0 mean and 1 std achieved by Xavier initialization but when ReLU is used the negative is clipped and so mean rises and thus the He initialzation introduces the sqrt(2) correction term which helps train a lot better.

So my question is as about the basic ResBlock which almost uninamously implemented as follows,

class ResBlock(nn.Module):

    def __init__(self,in_c):
        super(ResBlock, self).__init__()

        self.conv1 = nn.Conv2d(in_c,in_c, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv2 = nn.Conv2d(in_c,in_c, kernel_size=3, stride=1, padding=1, bias=True)
        self.relu = nn.LeakyReLU(negative_slope=0.2, inplace=True)

    def forward(self, x):

        identity = x

        out = self.relu(self.conv1(x))
        #out = F.leaky_relu_(out, negative_slope=0.2)
        out = self.conv2(out)
        out = out + identity

        return out

Note that conv2 is not followed by ReLU and only Conv1 is followed by ReLU. Thus conv2 should have Xavier init and Conv1 have He init. But the dominant practice is to use He init for both. Isn't this wrong?

---EDIT---

Alternatively, we can have conv2 have He initialization with 'leak parameter, a=1' and Conv1 have He initialization with 'leak parameter a=0'. The leak parameter is used to calculate the appropriate gain. But the dominant practice is to use He init for both with leak parameter set to 0. Isn't this wrong?

The 'leak parameter a' is the negative slope of the ReLU activation used after the conv layer. Check this for more information.

Thankyou


texts: Errors loading tensorflow (.pb) model into c# (Tensorflow.NET)?

texts:
I'm new to tensorflow.NET and I am currently following the documentation online (TF.NET/Example) to import my trained CNN model from python into c#.

So far I've imported my protobuf (.pb) file into c#, but I get errors when trying to set the input and outputs of the model (lines 23 and 24).

  Unhandled exception.

  System.Exception: Can not convert a String into a Operation.

  at Tensorflow.Graph._as_graph_element_locked(Object obj, Boolean
  allow_tensor, Boolean allow_operation)

  at Tensorflow.Graph.as_graph_element(Object obj, Boolean allow_tensor,
  Boolean allow_operation)

  at Tensorflow.Graph.get_operation_by_name(String name)

After looking online it appears to be something to do with multi-threading or parallelism, however, I've yet to find a solution to it. Can someone point me in the right direction?

Thanks.

Here is my current code implementation:


texts: How do I calculate the delta term of a Convolutional Layer, given the delta terms and weights of the previous Convolutional Layer?

texts:
I am trying to train an artificial neural network with two convolutional layers (c1, c2) and two hidden layers (c1, c2). I am using the standard backpropagation approach. In the backward pass I calculate the error term of a layer (delta) based on the error of the previous layer, the weights of the previous layer and the gradient of the activation in respect to the activation function of the current layer. More specifically the delta of layer l looks like this:

delta(l) = (w(l+1)' * delta(l+1)) * grad_f_a(l)

I am able to compute the gradient of c2, which connects into a regular layer. I just multiply the weights of h1 with it's delta. Then I reshape that matrix into the form of the output of c2, multiply it with the gradient of the activation function and am done.

Now I have a the delta term of c2 - Which is a 4D matrix of size (featureMapSize, featureMapSize, filterNum, patternNum). Furthermore I have the weights of c2, which are a 3D matrix of size (filterSize, filterSize, filterNum).

With these two terms and the gradient of the activation of c1 I want to calculate the delta of c1.

Long story short:

Given the delta term of a previous convolutional layer and the weights of that layer, how do I compute the delta term of a convolutional layer?


texts: How to find similarity of two series over time containing periodic trends?

texts:
Considering the data is received from a streaming source each second.How to distinguish if both the line graphs 'look' same/different in real time, statically, like the picture given below

Edit:
1.Not sure if there is something as real-time  periodic correlation mechanism then cross-correlation would perhaps be an ideal solution ?
2. Comparing slope of the two Line is the last option I would go with.
3. If statistically there is no way to solve this then I would look at machine learning to solve this


texts: Are there any deep learning models to forcast waveforms from waveforms?

texts:
I want to forcast waveforms from waveforms. Now I have collected the dataset.
The input feature is a waveform, e.g.:
input feature
The out is also a waveform, e.g.:
output feature
Are there any avaliable models to solve this problem?


texts: Instead of binarising discrete variables for NN algorithms, could I attempt to assign them a value by splitting into buckets and sorting?

texts:
Creating a boolean for each value of each discrete variable explodes the number of features. I have an idea for solving it but I wasn't able to find anything on this on the internet, so perhaps someone could either point me to a resource or let me know whether they think this is possible. And if not, maybe point to another solution (all I could find was binarisation).

So if we have training vectors $X_i$ and similarly labels $y_i$, I propose we partition the training set by some variable $j$, so that for each $X_i$ for which $j_i$ equals some value $q$, $X_i$ will be in $P(q)$.

Then for each $q$ we calculate the mean $y_i$ for $P(q)$ and sort those averages, mapping each $q$ onto its index in the sorted list. This value is used as the feature for $j$. I suspect that this ought to be better than just giving each value a random index, as any remaining non-linearity will be hopefully fixed by the hidden layers. But it seems that these non-linear patterns will be easier to pick up if the function is somewhat quasi-monotonous.

I think this could be improved, potentially instead of just finding the mean, another measure could work better and also for multiple discrete features, I am not sure if this process should be repeated for each variable separately or through a multi-dimensional partitioning.

The main question is of course, would this process be helpful at all, and if not, is there anything similar that could solve this problem?


texts: Activation function vs If else statement

texts:
The question is very naive and most of us may know the answer. I have googled it but was not able to find a satisfactory answer so posting it here. Can someone please put the right words on this question.

Activation functions like ReLU, Sigmoid etc generally work like if else statements. e.g. if value of input is less than 0 then 0 else same value ( max(0,a)  in case of ReLU)  etc. So the question is why if else statements are not being used? Are these functions less compute intensive or work better in case of multidimensional data?

The argument in favor of if else statements is, the programming of neural networks is also being done in high level languages like python, C etc. In that case what compels the use of these activation functions rather than programmatic oriented If Else syntax?


texts: How can I make a face verification system?

texts:
I'm building a face verification system using face embeddings (of 512 dimensions) from a pretrained Facenet model. For this, if I have some 4 to 5 images for a person, how can I successfully verify a new unseen images as either the same or different person?
I think this of one class classification (because of having single class) task, and googled it, but unable to find any reliable sources which suits well for this task.
Then, I've trained a SVM classifier with embeddings of a person and also with dummy embeddings of some 4 to 5 persons (so that having positive and negative classes). But it doesn't seem to work well.
Please suggest me with a ML algorithm/technique for this task, thank you.


texts:
I've read on one site that "if val_loss starts increasing and val_acc also increases, it could be a sign of overfitting".  I thought these model results fit well, however, does the results below, in and of itself; lead you to view this as overfitting?
model= classification_model()
#fit model
model.fit(X_train, y_train, validation_data= (X_test, y_test), epochs = 10, verbose = 2)
#eval model
results = model.evaluate(X_test, y_test, verbose =0)

Train on 60000 samples, validate on 10000 samples
Epoch 1/10
 - 113s - loss: 0.1839 - accuracy: 0.9449 - val_loss: 0.0984 - val_accuracy: 0.9679
Epoch 2/10
 - 104s - loss: 0.0774 - accuracy: 0.9759 - val_loss: 0.0834 - val_accuracy: 0.9742
Epoch 3/10
 - 160s - loss: 0.0526 - accuracy: 0.9834 - val_loss: 0.0869 - val_accuracy: 0.9752
Epoch 4/10
 - 133s - loss: 0.0395 - accuracy: 0.9875 - val_loss: 0.0705 - val_accuracy: 0.9789
Epoch 5/10
 - 103s - loss: 0.0285 - accuracy: 0.9913 - val_loss: 0.0808 - val_accuracy: 0.9775
Epoch 6/10
 - 99s - loss: 0.0258 - accuracy: 0.9917 - val_loss: 0.0733 - val_accuracy: 0.9804
Epoch 7/10
 - 113s - loss: 0.0222 - accuracy: 0.9927 - val_loss: 0.0934 - val_accuracy: 0.9779
Epoch 8/10
 - 111s - loss: 0.0213 - accuracy: 0.9928 - val_loss: 0.0909 - val_accuracy: 0.9792
Epoch 9/10
 - 108s - loss: 0.0165 - accuracy: 0.9945 - val_loss: 0.1161 - val_accuracy: 0.9771
Epoch 10/10
 - 143s - loss: 0.0171 - accuracy: 0.9948 - val_loss: 0.1040 - val_accuracy: 0.9752


texts: Match an image from a set of images : Combine traditional Computer vision + Deep Learning/CNN

texts:
In the application I am developing, I have about 5000 product label images.(One label per product).

One functionality of my application is that user can take a picture using his camera and get a possible match(es) against the product labels registered the system.

Since initially, my system only has one sample per product, I decided to go with traditional Computer Vision techniques. I managed to implement this using Feature extraction and Descriptor matching.(using OpenCV SIFT and FLANN techniques referring this: https://github.com/kipr/opencv/blob/master/samples/cpp/matching_to_many_images.cpp)

Now I am thinking how to improve the accuracy by combining with CNN or Deep Learning techniques since when users approve matches, it gradually add more label samples for a product.

Is it possible to build a hybrid image matching system combining Computer Vision techniques and CNN/Deep Learning techniques?

Are there any similar services already available as services?


texts: Finding the perfect algorithm for realtime optimizing of content

texts:
I am looking for an algorithm that allows me the following:

I have a webpage and I want to randomly show one "content" from a list of contents on that webpage depending on who (visitor) sees the webpage. I know my visitors' demographic features like age, gender, locale. I assume that visitors with similar demographics have a similar taste regarding content on my website. I also know that they liked my content when they share it in the end.

What I have:

A list of available contents, let's say: red, green, blue, purple,
violet
A constant stream of events of visitors with specific
demographic data that share a content

What I want:

First of all, all contents should be displayed completely randomly.
Each user should randomly get one content without any preference
As soon as the first user with a specific demographic shares the content
they got I want other visitors with similar demographic features that
end up on my webpage to see this specific content with a higher
probability.

So basically I want a self optimizing system that learns in realtime.


texts: Multi Label Classification on Data Columns in Tables

texts:
I am seeking guidance on a machine learning problem involving the tagging of data columns. Currently, I have a system where users can add multiple tags to a columns in a table. However, I want to automate the tagging of new columns by using Multi Label Classification. I have extracted 21 features from each column by doing a column analysis on the column values. The features obtained would include statistical values such standard deviation, max,min, kurtosis and etc. Am I on the right path in using these features as inputs for a Multi Label Classification model ? Right now I am focusing on numeric values in columns

Example:

As an example, the above table on the left represent some arbitrary table which consist of 3 columns. As a user I would tag the column with the appropriate tags. So the Rainfall column would have rainfall and precipitation and Temperature column would have temperature. The table on the right just represents the tags being assigned to a column in a table format.

Example sample data set in the above image

In order for me to do multi-label classification to automate the tagging of columns automatically, when tables with similar columns are ingested into the system, I would need to extract some features or properties that describes the already tagged columns to use as input for the multi label model. So I did some column analysis and placed just several example features in the table above. This includes standard deviation, maximum,minimum, median and kurtosis. I have about 21 features in total. The output labels are also represented for each column in the above image where 1 signifies the label is present and 0 is not present.

In the end the model will decide which tags are assigned to a newly discovered column based on its features.


texts: How do linear learning systems classify datapoints that fall on the hyperplane

texts: How do linear learning systems, such as the simple "closest to the class average" algorithm or SVMs, classify datapoints that fall on the hyperplane?


texts: How to prepare the varied size input in CNN prediction

texts:
I want to make a CNN model in Keras which can be fed images of different sizes. According to other questions, I could understand how to set a model, like Input =(None,None,3). However, I'm not sure how to prepare the input/output datasets.
Concretely, now I want to combine the datasets with (100,100) and (240,360).
However, I don't know how to combine these datasets.


texts: Conv1D relu: negative values or additional sequence?

texts:
If I have sequences that can have negative values with occasional spikes in both positive and negative directions I'd like to preserve, what is the best practice to handle those sequences:

should I experiment with leaky-RELU and that kind of stuff, or
should I just add an additional sequence indicating the direction (1 and -1/0) and have all values positive. Would it work though?

Thanks. Also, references to research on the subject are appreciated.


texts: What does the limit of xgboost max_depth=1 represent?

texts:
In my mind, this means that each tree just takes one feature, and produces a step function based upon it.

In the limit of n_estimators being very large and max_depth=1, does xgboost become a linear model?

On my dataset, a gridsearch found max_depth to be 1, so I'm wondering if I should be building a linear model instead.


texts: ELMo - How does the model transfer its learning/weights on new sentences

texts:
Word2vec and Glove embeddings have the same vector representation for every word in the corpus and does not take context into consideration.
For eg:

The dog does bark at people
The bark of the tree is hard.

In the above examples, Word2vec and Glove create one vector for the word "bark". But using Elmo, there would be two different representations for the word "bark" as it considers context. So, I am trying to the mechanism behind how Elmo gives a vector for a new sentence in the data set?
Say, we have a pre-trained model and it has different vectors for various words. When I have to use this model on a new sentence, does Elmo produce a new vector with completely new vectors? Then is the fine-tuning of the model always happening as it is applied to new data? If that is the case, when is it considered to be a completely trained model?


texts: How to interpret curve of regularization loss during CNN training?

texts:
I am fine-tuning a single shot detector (SSD) in tensorflow object detection api.
I didn't freeze the backbone (mobilenet), I programmed the learning rate to go from e-3 to e-4 to e-5.
In the paper the loss function is the sum of a weighted l1 norm and a weighted softmax. But is the tensorflow object detection api, the default configuration uses a weighted sigmoid instead of a weighted softmax.
When I visualize training progress on tensorboard, I get a curve showing the progress of the regularization loss. Here are 2 different examples of curves I got (first image : when I used weighted softmax, second image : when I used weighted sigmoid):

How to interpret the progress of the regularization loss, is it bad when it is high?


texts: Why do we need for Shortcut Connections to build Residual Networks?

texts: Why do we need for Shortcut Connections to build Residual Networks, and how it help to train neural networks for classification and detection?


texts: Policy gradient - and auto-differentiation (Pytorch/Tensorflow)

texts:
In policy gradient, we have something like this:

Is my understanding correct that if I apply log cross-entropy on the last layer, the gradient will be automatically calculated as per formula above?


texts: How to resolve IndexError: too many indices for array

texts:
I am attempting to load my dataset of images using the following code.

def load_dataset(train_dataset= 'img_pixels1.csv', test_dataset= 'img_pixels.csv', samples_per_digit=1000):

   """
    Loads and returns train and test datasets.
    """
    np.random.seed(7)
    data_train = np.genfromtxt(fname=train_dataset, delimiter=',')
    y = data_train[:,1].astype(np.int)
    x = data_train[:,1:]
    x = 255.0 - x
    x = x / 255.0
    labels = np.unique(y)
    for l in np.arange(len(labels)):
        index = np.where(y==labels[l])[0]
        np.random.shuffle(index)
        if l == 0:
            x_train = x[index[0:samples_per_digit]]
            y_train = y[index[0:samples_per_digit]]
        else:
            x_train = np.vstack((x_train, x[index[0:samples_per_digit]]))
            y_train = np.hstack((y_train, y[index[0:samples_per_digit]]))

    data_test = np.genfromtxt(fname=test_dataset, delimiter=',')

    y = data_test[:,0].astype(np.int)
    x = data_test[:,1:]
    x = 255.0 - x
    x = x / 255.0
    labels = np.unique(y)
    for l in np.arange(len(labels)):
        index = np.where(y==labels[l])[0]
        np.random.shuffle(index)
        if l == 0:
            x_test = x[index[0:samples_per_digit]]
            y_test = y[index[0:samples_per_digit]]
        else:
            x_test = np.vstack((x_test, x[index[0:samples_per_digit]]))
            y_test = np.hstack((y_test, y[index[0:samples_per_digit]]))

    return (x_train, y_train, x_test, y_test)

and yet I keep getting the following error.

----------

IndexError     Traceback (most recent call last)
<ipython-input-22-181aa814c982> in <module>
----> 1 x_train, y_train, x_test, y_test = load_dataset()
<ipython-input-21-bd181ccba363> in load_dataset(train_dataset, test_dataset, samples_per_digit)
      5     np.random.seed(7)
      6     data_train = np.genfromtxt(fname=train_dataset, delimiter=',')
----> 7     y = data_train[:,1].astype(np.int)
      8     x = data_train[:,1:]
      9     x = 255.0 - x

IndexError: too many indices for array

Assisstance please.


texts: Why does discriminiator accuracy falls to 0%, and is there a fix around this?

texts:
I am training a Vanilla-GAN(or original GAN 2016) on a pokemon dataset https://www.kaggle.com/kvpratama/pokemon-images-dataset, for few epochs the discriminator has 100% accuracy over the real examples, but as the epochs pass it falls to 0% around 4-5 epochs.

One interesting effect however is that when I increase my batch size from 32 to 64, this effect is seemed to have a delayed effect. i.e It kicks in around 400-500 epochs.

Please help.


texts: How to plot multiple columns with ggplot in R?

texts:
I do have a data frame with different categorical and numerical columns with the following schema:

Id | num_col_1 | num_col_2 | num_col_3 | cat_col_1 | cat_col_2

Now I want to draw a combined plot with ggplot where I (box)plot certain numerical columns  (num_col_2, num_col_2) with boxplot groups according cat_col_1 factor levels per numerical columns. Along y axis is the spread of the respective selected columns (not other column). So far I couldn' solve this combined task.

Thank you.


texts: What is the meaning of the term "pipeline" within data science?

texts:
Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here

People often refer to pipelines when talking about models, data and even layers in a neural network. What can be meant by a pipeline?


texts: How to deal with changing video frame sizes in a CNN?

texts:
How to deal with videos where the frame sizes are not the same frame to frame?

For example this video moves up and down and when it does, the video part of the screen has a different amount of pixels vertically.

How to deal with different frame sizes in a CNN?


texts: feature importance between two data distributions

texts:
I have two textual datasets collected from different domains (Twitter and Reddit).

I extracted a set of features in the same way from these two datasets, one of these features as an example called X_positive.

How could I know if the feature X_positive is more representative (exists more) in one of the datasets?


texts: What kind of machine learning algorithm can I use?

texts:
I have a data set of tweets regarding vaccines.  They have been collected from an API because they have keywords like "flu, measles, MMR, vaccine" etc.

I need to find tweets specifically about measles and the outbreak that occurred in California this past February.  It isn't enough to search the data set for words like "California" and "Measles" because tweets like "MMR vaccination rates in Palo Alto on the rise" are about measles and California, but wont be captured by a naive search.

Are there any unsupervised algorithms that could help me out?


texts: Needed: Java library to calculate text readability/complexity

texts:
In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall readability score, linsear write formula, gunning fog etc).

I guess there must be plenty of libs but I just cant find them ...


texts: How to predict variables based on multiple samples?

texts:
The problem
I tried to do some ML models but everytime I had some weird plots and I couldn't understand these scores. I'm relatively new to ML and maybe someone can help me with this data with some example (DecisionTreeClassifier/SVM/RandomForestClassifier/KNN) how to deal with it and interpret.
The dataset
I have two datasets:

"seq.csv" with frequency of every aminoacid in specific protein:

A,C,D,E,F,G,H,I,K,L,M,N,P,Q,R,S,T,V,W,Y
17,0,6,14,5,11,12,9,19,18,3,2,4,5,4,6,5,9,2,3
17,0,10,8,5,11,1,10,13,16,5,12,3,5,13,6,12,9,3,6
18,0,6,14,6,11,11,9,19,18,3,2,4,5,4,6,5,8,2,3
16,0,11,8,5,11,1,10,13,17,5,12,3,5,13,7,12,9,3,6

"secstr.csv" with frequency of every bond of secondary structure in specific protein:

H,G,I,B,E,S,T,C,
108,12,0,0,0,3,12,0,19
102,3,0,1,14,12,15,0,18
109,9,0,0,0,2,14,0,20
103,3,0,1,12,9,13,0,26

Code example
import numpy as np
from sklearn.model_selection import train_test_split

X = np.asarray(seq_df)
Y = np.asarray(sec_str_df['H'])

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)
X_train.shape, X_test.shape
>>>((67, 20), (33, 20))

##########################

from sklearn.linear_model import LinearRegression

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
print('Score: ', lineReg.score(X_test, y_test))
print('Weights: ', lineReg.coef_)

plt.plot(lineReg.predict(X_test))
plt.plot(y_test)
plt.show()
>>>Score:  0.6800436491044164
Weights:  [-0.87378736 -0.13286578 -4.03961593  2.71956276 -1.6088727  -1.655681
 -3.381233    1.89215565  1.47324114  5.08374842  1.97302355 -2.77426402
 -0.92489368 -1.24430955  3.09726346 -0.41345148  0.11391256 -1.95393996
 -0.71682087  0.05350237]

##########################

clf = DecisionTreeClassifier(max_depth = 4,
                             random_state = 0)
clf.fit(X_train, y_train)
>>>DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',
                       max_depth=4, max_features=None, max_leaf_nodes=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, presort='deprecated',
                       random_state=0, splitter='best')

The goal
I'm trying to predict likelihood of occurence specific bond in protein based on aminoacid frequency.


texts: How do I build a DQN which selects the correct objects in an environment based on the environment state?

texts:
I have an environment with 4 objects in it. All of these objects can either be selected or not selected. So the actions taken by my DQN should look like - [1,0,1,1],[0,0,0,1],[1,1,0,0]...etc

Where 1 denotes that the object was selected and 0 denotes that the object was not selected.
The environment state being given as input to the DQN consists of attributes for each of the object and other factors of the environment. The DQN would get rewards based on the selection it made.
I'm new to reinforcement learning and I've only built DQNs that had to select one action out of the entire action space. But how do I build a DQN or a Reinforcement learning network for this particular environment?


texts:
I'm coming across a metrics for model evaluation which I had never seen before and I don't know how to further research (since I don't know its proper name).

I'm using someone else's code, whose goal is to perform cross-validation to choose the best tree-based algorithm for a binary classification. It is probably worth saying that classes are highly skewed (93% / 7%).
The metric which is used is the following: the classifier is trained and then the probability associated with each test element is computed.

probas = probas[:,list(clf.classes_).index(1)]

Then, these probabilities are ordered from the highest to the lowest and put in the x-axis. On the y-axis, being y the cumulative sum of entries associated with each probability.
Then, they compute the area under the obtained curve, as in:

joint = zip(probas, truth)
joint = sorted(list(joint), key=lambda x:x[0], reverse=True)
probas = [x[0] for x in joint]
truth = [x[1] for x in joint]

# Calculate accumulated number of true labels at each probability point.
# Also calculate Area Under Curve (AUC) (higher is better model)
truth_cumulative = np.cumsum(truth) / np.sum(truth)
area = np.trapz(truth_cumulative, dx=1) / len(truth)

Can anyone give me an intuition of what this metric is about and a pointer to some resources to better understand it?

Thanks.


texts:
I am trying to implement an autoencoder for prediction of multiple labels using Keras. This is a snippet:
input = Input(shape=(768,))
hidden1 = Dense(512, activation='relu')(input)
compressed = Dense(256, activation='relu', activity_regularizer=l1(10e-6))(hidden1)
hidden2 = Dense(512, activation='relu')(compressed)
output = Dense(768, activation='sigmoid')(hidden2) # sigmoid is used because output of autoencoder is a set of probabilities

model = Model(input, output)
model.compile(optimizer='adam', loss='categorical_crossentropy') # categorical_crossentropy is used because it's prediction of multiple labels
history = model.fit(x_train, x_train, epochs=100, batch_size=50, validation_split=0.2)

I ran this in Jupyter Notebook (CPU) and I am getting loss and validation loss as:
loss: 193.8085 - val_loss: 439.7132
but when I ran it in Google Colab (GPU), I am getting very high loss and validation loss:
loss: 28383285849773932.0000 - val_loss: 26927464965996544.0000.
What could be the reason for this behavior?


texts: Feature importance varying a lot using same data with same features

texts:
I have trained two boosted classifiers on same data with same features, but I am getting two different set of feature importance from the models.

Few high ranked features (suppose top 10) from one boosted classifier is almost 5-6 ranks below when compared with the top 10 of other boosted classifier.

Is there anything wrong with this, or its totally okay to have this kind of behavior as it depends on how a model learns.

Thanks in advance.


texts:
I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new observation only and predict future values.

Question

Should I be reading the entire dataset each time I get a new Observation, normalize the entire dataset, create the ML model, then predict?

How I got stuck:

Let's say I have 1 feature and at t-1 all of the values have min/max = [0, 1000]
At t, a new observation comes in with value = 1001
How should I normalize the new value given that the ML model has been trained with different min/max?

Thank you


texts: Difference between 1x1 Convolution and TimeDistributed(Dense())

texts:
Are these lines of code equivalent in Keras? From a few runs, they seem to be, and also intuitively since the channels dimension of my data is 1, my understanding is that a fully connected acts like a convolutional layer. Is one better computationally (they have the same number of parameters)?

Conv1D(filters=1, kernel_size=1, strides=1, padding='causal')
TimeDistributed(Dense(1))

For example, I have a regression problem, where I have (batches,64) points coming in and (batches,64) points coming out of the model.

My model is:

model = Sequential()
model.add(Reshape([64, 1], input_shape=(64,)))
model.add(BatchNormalization())
model.add(Conv1D(filters=4, kernel_size=8, strides=1, padding='causal', activation = 'relu'))
model.add(BatchNormalization())
model.add(Conv1D(filters=1, kernel_size=1, strides=1, padding='causal'))
model.add(Flatten())
model.compile(loss=loss, optimizer=optimizer)

My thinking for the layers is:
reshape dimensions to 3D.
variance scale.
conv layer, activation.
variance scale.
final layer: my question is whether is matters if its a TimeDistributed(Dense) or Conv in my situation.
reshape back to a 2D waveform.

Thank you in advance!


texts: Why does logistic regression in Spark and R return different models for the same data?


texts: Gaussian Process for Classification: How to do predictions using MCMC methods

texts:
Problem

I was reading about Gaussian Processes for regression in the "Gaussian Processes for Classification" textbook and in a few other online resources. Everywhere I look people seem to avoid talking about one would go about doing this. Can anyone provide a simple answer to this?

Mathematics and Context

$X\in\mathbb{R}^{n\times d}$ is a matrix whose rows ${\bf{x}}_i$ are the $n$ training observations living in $d$-dimensions.
${\bf{y}}$ is an $n$-dimensional vector containing training labels $0$ and $1$ for each training input.
${\bf{f}}$ is an n-dimensional vector whose elements $f_i={\bf{x}}_i^\top {\bf{w}}$ are the so-called linear predictors.
${\bf{x}}_*$ is a testing input.

Formulation of Gaussian Process for Classification

Inference is performed in two steps:

Compute
$$
p(f_*\mid X, {\bf{y}}, {\bf{x}}_*)=\int p(f_*\mid X, {\bf{x}}_*, {\bf{f}})p({\bf{f}}\mid X, {\bf{y}})d {\bf{f}}
$$
Squish the value using the sigmoid function to find the class probability.
$$
\overline{\pi}_* = p(y_*=1\mid X, {\bf{y}}, {\bf{x}}_*) = \int \sigma(f_*)p(f_*\mid X, {\bf{y}}, {\bf{x}}_*)d f_*
$$

What I don't understand

How does one go about solving this using sampling methods? My idea is that the first integral might be similar to an expectation so maybe we can do something like this.

Get samples ${\bf{f}}_1, \ldots, {\bf{f}}_N$ from $p({\bf{f}}\mid X, {\bf{y}})$ and then approximate the first integral like this
$$
\mathbb{E}_{p({\bf{f}}\mid X, {\bf{y}})}\left[p(f_*\mid X, {\bf{x}}_*, {\bf{f}})\right] \approx \frac{1}{N}\sum_{i=1}^N p(f_*\mid X, {\bf{x}}_*, {\bf{f}}_i)
$$
but then how do I compute $p(f_*\mid X, {\bf{x}}_*, {\bf{f}}_i)$ ?


texts:
I am looking for an algorithm that clusters a directed graph into a set of clusters that form a directed acyclic graph.

For example, given:
nodes: {A, B, C, D}
with edges: {(A,B), (B,A), (A,C), (C,D), (D,C)}

A valid output would be: {{A,B},{C,D})
An invalid output would be: {{A,C},{B,D}} (since the node edges imply a cycle between the clusters)

Thanks for any pointers or suggestions!


texts:
I have a huge dataset of addresses. I have another data stream that contains addresses that I need to match against those in the original dataset. As all the addresses are user-provided, matching them is not trivial. For example, 10 John Smith Square could be represented as 10 J.S. Square, J. Smith Sq 10, possibly even as only John Smith (which obviously misses the number and on top of that can collide with John Smith Street).

I am wondering if there are some known ways to reducing this ambiguity. My intuition is that some clustering should be possible, allowing me to match incoming addresses from the second dataset to the clusters of the first. I want to get to clusters that separate different street numbers, rather than put all addresses on a single street together.

I'm recently considering employing some sort of (possibly sparse) autoencoders, but was not able to find any literature to suggest that anyone else has tried a similar approach. So my question is, how can I employ autoencoders to cluster street address labels if anyone has tried anything of the sort? Otherwise, what alternative approaches would you suggest.

I've already have a dictionary to reduce the vocabulary by trivial substitutions (e.g. Square to Sq), but it remains to be tested whether this actually improves results or not, depending on what further analysis is applied after that. My current representation of the address strings is with bag-of-words, as I don't think word order changes much.


texts: Why is it bad to use the same test dataset over and over again?

texts:
I am following this Google's series: Machine Learning Crash Course.

On the chapter about generalisation, they make the following statement:

Good performance on the test set is a useful indicator of good performance on the new data in general, assuming that:

- The test set is large enough.

- You don't cheat by using the same test set over and over.

Why exactly is the second point a bad one? As long as one does not use the test set for the training stage, why is it bad to keep using the same test set to test a model's performance? It's not like the model will get a bias by doing so (the test set is not updating any of the model's parameters internally).


texts: Is there an inherent recency bias in deep learning?

texts:
When working with very large models within Deep Learning, training often takes long and requires small batch sizes due to memory restrictions.
Usually, we are left with a model checkpoint after training has commenced. I am wondering whether the exact time at which we take that checkpoint significantly factors in to the statistical properties of a model's outputs.
For example:
Within text generation, lets assume that just before we extract the checkpoint, the model learns statistically anomalous batches with longer sentences than the mean.
Would that result in our model generating longer sentences, overrepresenting that recent batch of anomalous texts?
As training batches are often randomly generated from the dataset, such unrepresentative batches may certainly occur, sometimes right before we save the checkpoint.
Has there been any research regarding such, potentially unwanted, recency bias in slower deep learning scenarios?
The only references I could find were intentionally trying to employ such biases, but I have not found any literature on unwanted recency bias.


texts:
I have a dataset  where for each date I have two values say A and B, there are dates when one of the values is missing

Date       C1   C2
1/1/2019    A   10
1/1/2019    B   20
1/2/2019    A   25
1/2/2019    B   30
1/3/2019    A   23
1/4/2019    A   32

I want to make sure for every date, I have the date for A and B.
One way I can think is to have driver table with all distinct dates and data for A and B and then do left join.
Is there any better way of doing this with pandas merge or iterating through rows? I can't think of any but feel like there might be better way of doing it. Thanks!


texts: Semantic segmentation in high-resolution images with high variance - cannot avoid underfitting

texts:
I am working on a dataset of 2K images for a semantic segmentation problem.
I want to detect and localize small objects, with the smallest mask to be 5x5 pixels.
The images include 5 different textures, quite different from each other.
I am using Unet and EfficientNetb0-3 as backbones, but whatever I do, I get underfitting.
I've reached the conclusion that it is probably due to the fact that the resolution is too high and the variance as well.
Do you have any advice on how I could overcome this underfitting problem?


texts:
What are the downsides of using TPUs instead of GPUs when performing neural network training or inference?

From what I read on https://www.predictiveanalyticsworld.com/machinelearningtimes/should-you-choose-a-gpu-or-a-tpu-to-train-your-machine-learning-models/10460/ there are only upsides:


texts: how to match a sentence to a cluster of keywords?

texts:
I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below.

year of education               Education
years education                 Education
years of educational            Education
two years of education          Education
years of education beyond       Education
education four year             Education
..........
of proven sales experience      Ability
knowledge of and                Ability
experience or education high    Ability
assigned knowledge skills       Ability
accountable for driving         Ability
..........
administrative and leadership skills    Experience
advanced negotiations skills            Experience
must have keyboarding skills            Experience
must have skills                        Experience
activities preferred skills             Experience
of clinical skill                       Experience

I have to give a string and find it whether it belongs to either Experience or Education or Ability based on the trained model. The examples of strings .

string1 = "There is a requirement of four year professional degree"
string2 = "Able to drive the teams to higher levels"
string3 = "Must have programming experience in C, C++"

when I test these string it should be able to classify the string in to either of the clusters.

What are the possible ways to train my model ? Refering to word2vec and doc2vec, will these models work? I couldn't find any relevant examples to train model on words and test on strings. Any ideas on how to work?


texts: Why do we want the variance of the layers to remain the same throughout a deep network?

texts:
I've been reading the literature on vanishing/exploding gradients and specifically how they connect to weight initialization. An idea I've come across a few times, which seems very important in this area, is that we want the variance to remain the same throughout the layers of a neural network, that is, if $v_n$ is the variance of the $n$-th layer, we want all the $v_n$ to be about equal. For example in Kumar 2017:

In this paper, I revisit the oldest, and most widely used approach to the problem with the goal of resolving some of the unanswered theoretical questions which remain in the literature. The problem can be stated as follows: If the weights in a neural network are initialized using samples from a normal distribution, $N(0,v^2)$, how should $v^2$ be chosen to ensure that the variance of the outputs from the different layers are approximately the same?

The paper goes on to claim that "the first systematic analysis of this problem was conducted by Gloriot and Bengio", citing Gloriot & Bengio (2010). But that paper seems to just assume that the reader already accepts keeping the variance stable is a good idea, I can't find anything like an explanation in the paper. They just make this claim:

From a forward-propagation point of view, to keep information flowing we would like that
$$\forall(i, i′), Var[z^i] =Var[z^{i′}]$$
From a back-propagation point of view we would similarly like to have
$$\forall(i, i′), Var\left(\frac{∂Cost}{∂s^i}\right)=Var\left(\frac{∂Cost}{∂s^{i′}}\right)$$

What variance is actually being taken here? Variance with respect to what?
What exactly is helped by the variance being stable in this way?
Whatever the answer to (2) is, what is the proof or evidence that that's the case?


texts: How to use decode_predictions() for non-Imagenet models..?

texts:
I know that decode_predictions() works for only imagenet dataset(1000 classes) for the models like VGG16 etc.
But condiser my scenario.

My Scenario:

I used vgg16 pretrained model, and added my own weights.

So this turns out to be a non-Imagenet model. I have mentioned classes=9 as i trained my previous model with 9 classes only.

So now to find the predictions, i could use predict() method and then my print(answer) would give the corresponding class label.
But actually i need the class name to be printed.
Is that possible to get class name ?? If it is, can anyone explain me how.?


texts: Machine Learning algorithms and Cross Validation, the best practice

texts:
I'm new in Machine Learning, and I'm studying the main concepts behind algorithms from the mathematical point of view.
I'm also trying to start implementing some algorithms for regression purposes using sklearn library.
I would like to summarize the main steps in ML and provide the question:

I understand the dataset, clean and prepare, identifying a set of suitable algorithms for such dataset.
Train the algorithms and see the performances, choosing the best one.
Hyperparameter tuning of the best performing algorithm using Random or Grid Search with cross-validation between 5-10
I have the best parameters for training the algorithm, I train the algorithm with such params.

Now the question that arises for which I am not clear what to do.
May I consider to train the algorithm with the best parameters also performing the cross-validation? or Have I  to train the algorithm without the cross-validation?
In short, The model puts into production is the model trained with cross-validation (for instance using KFold for training in step 4) or the model trained without (step 4)?


texts: Model performance in different snapshots varying

texts: I am trying to solve this problem. A medical representative needs to visit some doctors' clinics and for that a model will generate probability scores for visiting a clinic. I ma using a tree based model RandomForest/XGboost. The data has to be scored every week and based on that he will decide whether to visit. What I observed is that the behaviour of the representative is mostly dependent on his past one month. So I decided to train the model each week on the past one month of his visit information. But the AUC is varying too much across different weeks. What might be the reason for this? The visit rate is not same across different weeks, but still is there any way to gain some statbility in terms of performance?


texts: Can we use feature selection and dimensionality reduction together?

texts:
I have a dataset having about 10,000s of features. The features have a hierarchy inherent to them. I found an algorithm performing feature engineering, taking the hierarchy of the features into consideration. After the procedure the feature space will be changed and the original features may not exist. This algorithm will reduce the number of features to about 2000 features. As the next step I am planning to use autoencoders(to perform dimensionality reduction) and obtain a latent representation to perform the classification task. The reason I didn't use the original dataset for the autoencoders is because I want to use the information on the hierarchy of the features for my model. Is this a meaningful model? Is it pointless to compress the feature space twice?
Thank you!


texts: How to summarize(add) a column according to same year and plot in R?

texts:
I have following type of data. But I would like to have graph with X-axis as Year and add all the sales added up in that year on Y- axis.

The Data frame name is salesdata. I have nearly 8000 rows of data. and the column headers are in the picture below.

Thanks!


texts: Algorithm for gesture classification in a wearable

texts:
I'm fairly new to machine learning so please pardon me if I mess up terms. I'm working on a wearables project that involves hand gesture detection, looking for any pointers on potential algorithms that would be applicable.

Basically there will be sensors that pick up tendon movement at the wrist to provide input, so for simplicity I'm assuming that would mean 5 features (one for each finger), each with values ranging from say 0-100.

There are also 5 classes that the gesture could be classified as. The user would do each type of gesture several times to provide training data, and then for each gesture class I'd need an algorithm to determine the best model for it based on that training data.

After initial training, whenever the user does a gesture it would input the sensor readings into each model to see which one is the closest match.
Additionally, since this is a wearable anything that optimizes for low memory and low processing power would be best.

If anyone has any input on either the type of models or the algorithm with which to generate them that I should look into I'd appreciate it!

Edit: Here is some sample data from 4 sensors during 3 different gestures. These are basically just pressure readings from sensors lying along tendons on the wrist.


texts: Extracting patterns from the spatiotemporal movement of objects

texts:
Say I have a database with x and y coordinates (30 times per second) for some objects in a 2d plane.
Is there a way to extract patterns from these coordinates?
If so, is there a way to visualize these patterns using for example vectors?


texts: Keras intuition/guidelines for setting epochs and batch size

texts:
I'm using Python with Keras to make a convolutional neural network (CNN) for an image classifier. I took about 50 images of documents and 150 images of non-documents for training. I shrunk the resolution, using scipy.misc.imresize(small, (32,32)) to standardize the images, when looking through these pixelated images I thought that I could still tell the difference between photos and documents, so I figured that the ML algorithm should be able to as well.

My question is in regards to the number of epochs and batch size. I'm currently using model.fit(X_train_temp, y_train_temp, epochs=N_epochs, batch_size=batch_size, verbose=verbose) . I've seen this question regarding the topic here:

How to set batch_size, steps_per epoch and validation steps

But the answer was very definition-based; I'm looking for intuition. I would like to know if there are general guidelines as to what values to set the number of epochs and batch size to for a given problem. I performed a crude parameter sweep across the number of epochs and batch size. Here is the CNN model:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape, activation='sigmoid'))
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape, activation='sigmoid'))
model.add(Flatten())
model.add(Dense(num_classes, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

And here are the results:

Here are some observations that I've made so far. The circled region seems to be pretty good for training since high accuracy is achieved relatively early on and it does not seem to oscillate much as further epochs pass. It seems that batch_size = 1 is generally a bad idea since training does not seem to improve the model. Additionally, it seems that batch_size >~ N_epochs seems to be desirable.

Question 1: is batch_size = 1 generally a bad? Or is this just for my particular parameter sweep?

Question 2: Is it generally the case that N_epochs >~ batch_size is recommended?

Question 3: Can anyone speak to the robustness of the generalizations in questions 2?

Thank you.


texts: Correct dimensions of a Siamese network Input array

texts:
I have an image dataset where the folder structure is as follows-
there are 900 folders (all of which will be classes) and in each folder, we have a varying number of images(each image has been resized to the same pixel intensity).
Now when we construct the 4d matrix to be fed to the deep learning model it should be of the dimension-
  (900,A,105,105)

900 because that's the total number of classes
105 x 105 because I have resized each image to that
now A should be the number of images in each folder.

In my case the number of images in each folder is varying so my question is-
If I keep the original varying number of images in each folder that means in my 4d array A would be the dimension of the largest folder size which means for folders of lesser number of images it will have a zero 2d matrix. So is that technically right or should I make all the folder sizes same?


texts:
How to get sentence embedding using BERT?
from transformers import BertTokenizer
tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')
sentence='I really enjoyed this movie a lot.'
#1.Tokenize the sequence:
tokens=tokenizer.tokenize(sentence)
print(tokens)
print(type(tokens))

2. Add [CLS] and [SEP] tokens:
tokens = ['[CLS]'] + tokens + ['[SEP]']
print(" Tokens are \n {} ".format(tokens))

3. Padding the input:
T=15
padded_tokens=tokens +['[PAD]' for _ in range(T-len(tokens))]
print("Padded tokens are \n {} ".format(padded_tokens))
attn_mask=[ 1 if token != '[PAD]' else 0 for token in padded_tokens  ]
print("Attention Mask are \n {} ".format(attn_mask))

4. Maintain a list of segment tokens:
seg_ids=[0 for _ in range(len(padded_tokens))]
print("Segment Tokens are \n {}".format(seg_ids))

5. Obtaining indices of the tokens in BERT’s vocabulary:
sent_ids=tokenizer.convert_tokens_to_ids(padded_tokens)
print("senetence idexes \n {} ".format(sent_ids))
token_ids = torch.tensor(sent_ids).unsqueeze(0)
attn_mask = torch.tensor(attn_mask).unsqueeze(0)
seg_ids   = torch.tensor(seg_ids).unsqueeze(0)

Feed them to BERT
hidden_reps, cls_head = bert_model(token_ids, attention_mask = attn_mask,token_type_ids = seg_ids)
print(type(hidden_reps))
print(hidden_reps.shape ) #hidden states of each token in inout sequence
print(cls_head.shape ) #hidden states of each [cls]

output:
hidden_reps size
torch.Size([1, 15, 768])

cls_head size
torch.Size([1, 768])

Which vector represents the sentence embedding here? Is it hidden_reps  or cls_head ?
Is there any other way to get sentence embedding from BERT in order to perform similarity check with other sentences?


texts: Evaluating likelihood of egg breaking when falling in random container on concrete

texts:
I am working on a project where I would like to predict whether an egg will break if it is put in a container that is then dropped on concrete.

I am looking at the different factors that play a role in whether the egg will break. So far, I have come up with the following:

mass
friction
padding
shape of container

What other variables could help predict the outcome?


texts: Specifics of Input data format and processing in LSTM models

texts:
I'm busy writing a LSTM from scratch(just started) and I just want to clarify something.
If I have a data shape as 1,10,2 which looks like:
double array[10][2]:
{ 0.52 0.82 }
{ 0.98 0.91 }
{ 0.76 0.32 }
{ 0.63 0.78 }
{ 0.12 0.21 }
{ 0.99 0.76 }
{ 0.43 0.65 }
{ 0.2  0.32 }
{ 0.45 0.87 }
{ 0.32 0.17 }

So this means number of samples = 1, 10 timesteps, 2 features (correct me if I'm wrong).
Does this mean that I have 10 Individual LSTM's and 2 input neurons for each timestep?
So basically with Keras, you feed this 3 dimensional array only so that the keras library can define the amount of individual layers within the LSTM net and the amount of input neurons feeding all the gates?
Is this correct?


texts: Which classification model to use on large, high-dimensional dataset?

texts:
I face a classification task: with several features a target features is to be predicted. I'm working with python.

My dataset includes 60 features from which I picked 16 which I think could be relevant (many others are time stamps, for example). The problem is that most of these 16 features are categorical - encoding them with get_dummies generates 886 features.

The data also includes about 17 million observations.

I am now wondering about how to tackle this problem and what to research and try next. I summarized this in these two questions and I'd love to hear some opinions!

First. If possible, I'd like to reduce the number of features. I tried using SelectFromModel with a RandomForestClassifier which worked okay, I think, as features were reduced drastically without much loss of prediction power. However, as my categorial features are split into several new features, only parts of the original features are selected (it's not einther all or none of the features that originated from one). Is this a problem? If so, can this be avoided?

Second. As one model can be tuned a lot by playing with parameters or input representation, I would like to focus on a few promising models. For faster results, I used only 200,000 observations to train KNeighborsClassifier, LogisticRegression, LinearSVC, DecisionTreeClassifier, RandomForestClassifier, GradientBoostingClassifier and MLPClassifier (neural network); all from sklearn.

I would not continue to pursue KNeighborsClassifier and GradientBoostingClassifier as they already took a lot of time to train for this small subset.
As RandomForestClassifier is supposed to perform nearly always better than DecisionTreeClassifier, I would drop the latter, too, but stick to RandomForestClassifier.
The two linear models LogisticRegression and LinearSVC worked well with penalty = l1, but very badly with penalty = l2, so that I would continue to pursue both with penalty = l1 (altough I expect similar results).
My first neural network performed very badly, but I guess there a plety of things to try for improving; however, I expect a very long training time with the full dataset (as this small subset took quite a while already).
I did not try a naive bayes classifier (as it is supposed to perform worse than linear models anyway) or a support vector machine (as they are supposed to perform badly with many observations).

Summary: I would continue my work by looking at RandomForestClassifier, LogisticRegression/LinearSVC and (if you think this is a good idea) neural networks. Is this reasonable?

Thanks a lot!


texts: The robustness of the Frobenius and L2,1 norms to the outlier

texts:
I have a question about the properties of the Frobenius and L$_{2,1}$ norms. Why is the L$_{2,1}$ norm more robust to the outlier than the Frobenius norm?
PS: For a matrix $A\in\mathbb{R}^{n\times d}$, it can be easily seen that
$$
\text{Frobenius norm:}\qquad\Vert A\Vert_F=
\left(\sum_{i=1}^{n}\sum_{j=1}^{d}\vert a_{i,j}\vert^2 \right)^{\frac{1}{2}}=\sum_{i=1}^{n}\Vert A(i,:)\Vert_2^2,
$$
and
$$
L_{2,1}\,\,norm: \qquad\Vert A\Vert_{2,1}=\sum_{i=1}^{n}\left(\sum_{j=1}^{d}\vert a_{i,j}\vert^2 \right)^{\frac{1}{2}}=\sum_{i=1}^{n}\Vert A(i,:)\Vert_2,$$
where $A(i,:)$ is the $i$-th row of $A$.
I would be very grateful if some could answer my question.


texts:
I tried to implement LSTM model for time-series prediction. Below is my trial code. This code runs without error.
metrics = ['mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error']

# define model
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_past, 6), return_sequences=True))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(LSTM(50, activation='relu'))
model.add(Dense(2, activation='relu'))
model.compile(optimizer='adam', loss='mse', metrics=metrics)
model.summary()

)
The forecast correctly predicts the peaks, but the constant values should be 0 and you cannot predict it.
However, prediction is extremely poor. How to improve the predictin? Do you have any ideas to improve it?
Any ideas for improving prediction by re-designing architecture and/or layers?


texts: How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

texts:
Sorry for a long story, but it is a long story. :)
I am using h2o library for python to build a decision tree and to extract a decision rules out of it.
I use some data for training where labels get TRUE and FALSE values.
My final goal is to extract the significant path (leaf) of the tree where the number of TRUE cases significantly exceeds that of FALSE ones.

treemodel=H2OGradientBoostingEstimator(ntrees = 3, max_depth = maxDepth, distribution="bernoulli")
treemodel.train(x=somedata.names[1:],y=somelabel.names[0], training_frame=somedata)
dtree = H2OTree(model = treemodel, tree_number = 0, tree_class = False)

def predict_leaf_node_assignment(self, test_data, type="Path"):
    if not isinstance(test_data, h2o.H2OFrame): raise ValueError("test_data
    must be an instance of H2OFrame")
       assert_is_type(type, None, Enum("Path", "Node_ID"))
    j = h2o.api("POST /3/Predictions/models/%s/frames/%s" % (self.model_id,
    test_data.frame_id),
        data={"leaf_node_assignment": True, "leaf_node_assignment_type":
    type})
    return h2o.get_frame(j["predictions_frame"]["name"])
dfResLabH2O.leafs = predict_leaf_node_assignment( dtree,test_data=dfResLabH2O , type="Path")

In sklearn there is an option to explore the leaves by using tree.values
There is not such an option for h2o as I understand.
Instead of that there is an option in h2o to return predictions on leaves.

When I run dtree.predictions
I am getting pretty weird results

dtree.predictions
Out[32]: [0.0, -0.020934915, 0.0832189, -0.0151052615, -0.13453846, -0.0039859135, 0.2931017, 0.0836743, -0.008562919, -0.12405087, -0.02181114, 0.06444048, -0.01736593, 0.13912177, 0.10727943]***

My questions (and somebody has already asked it, but no clear answer was provided so far)

What's the meaning of negative predictions? I expect to get a proportions p of TRUE to ALL or FALSE to ALL, where 0<=p<=1. Anything wrong with my model.
I ran it in skitlearn and I can point out the certain significant paths and extract rules.
For positive values : Is it TRUE to ALL or False to ALL proportion? I am guessing it so FALSE as I mentioned Class=False, but I am not sure.
Is there any method or solution for h20 trees to reveal the sample size of the certain leaf and the [n1,n2] for TRUE and FALSE cases respectively in  a similar way that sklearn provides?
I found in some forums a function def predict_leaf_node_assignment that aims to predict on a dataset and to return the leaf node assignment (only for tree-based models), but it returns no output and I cannot find any example how to implement it.
The bottom line : I'd like to be able to extract the sample size values of the leaf and to extract the specific path to it, implementing [n1,n2] or valid proportions.

I'll appreciate any kind of help and suggestions.
Thank you.


texts:
I read a couple of posts (like this one) about the encoder-decoder architecture and their implementation. But I still don't understand a couple of things.

What is the difference between a basic CNN or RNN and encoder decoder ?
Are there some properties that the encoder and decoder need to satisfy ? As far as I understood the encoder encodes the input in another dimension and creates a context vector. Later this context vector is decoded by the decoder.

I know that their are different types of encoders and decoders, what is the most simple architecture to implement ?

Should I use predefined word embeddings like word2vect in the encoder(if i have text as an input) ?


texts: Why are optimization algorithms slower at critical points?

texts:
I just found the animation below from Alec Radford's presentation:

As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of it. Regular SGD itself is simply stuck at the saddle point.

Why is this happening? Isn't the "movement speed" constant value that is dependent on the learning rate?

For example, weight for each point on regular SGD algorithm would be:

$$w_{t+1}=w_t-v*\frac{\partial L}{\partial w}$$

where $v$ is a learning rate and $L$ is a loss function.

In short, why are all optimization algorithms slowed down by the saddle point even though step size is constant value? Shouldn't a movement speed be constantly same?


texts: Finding out the scale when using MinMaxScaler()

texts:
I am using the MinMaxScaler() of sklearn to scale my features before using kmeans.I needed to find the scale used.

from sklearn import preprocessing
scaler = preprocessing.MinMaxScaler()
scaler= scaler.fit_transform(finance_features)
print scaler.scale_

However on using the line

print scaler.scale_

It shows an error saying no such attribute


texts: How to process multivariant questions in Orange

texts:
can you explain how to process multivariant questions in Orange? For example:
I have sociological survey database. Among the others there is question:

How would you like to spend your weekend? (pick only 3):

At home
With friends
With family
At work

And here is database example:

Database logic: max three options to pick - 3 columns. Respondent answer's order doesnt matter. Is there a possibility to make frequency histogramm in Orange based on these three columns?


texts: Segment 5-7 min audio into sentence wise audio clips for creating speech recognition dataset

texts:
I am trying to create a speech recognition dataset, especially for Indian Accents.
I am taking from colleagues to build this.
Daily I send an article link and ask them to record and upload it to google drive.
I have a problem with this approach.
All audio recordings of length 5 -7 min.
I am using the DeepSpeech model for this and it requires 10-sec audio sentences.
Suggest me any approach if possible to segment audio files into corresponding sentence phrases or to build a better with 5 min length audio files.
Suggestions are more than welcome on a better way to create a speech-to-text dataset.


texts: Transfer learning for a regression problem

texts:
if my understanding is correct, in case of image classification and NLP, if I have a pre-trained model, to train on new data, I can reshape the data according to the pre-trained model. So there is no problem even if the new data is slightly different from the previous data.  I am trying to use transfer learning for a regression problem. Consider I train a base model with 15 parameters and 1 million rows. I train a model. Now if I want to use this model for a similar problem case but I have only 14 parameters, one parameter is missing. Will the pre-trained model be of any use. Is there a way I can use transfer learning in such cases?

Thank you.


texts: Does LSTM without delayed inputs work as a deep net?

texts:
I want to predict a multivariate time series. My time series is $a_1(t),...,a_k(t)$ and I want to predict $a_k(t)$. I use the following keras LSTM:
model = Sequential()
model.add(LSTM(90,return_sequences=True,input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(LSTM(90))
model.add(Dense(1))

I use $a_1(t),...,a_{k-1}(t)$ as input and $a_k(t)$ as output to train it. So I don't use delayed inputs like $a_s(t-l)$. My qestion is in this situation, Does LSTM work as a deep neural network? i.e. Is it same as the following keras net?
model.add(Dense(90, input_dim=12))
model.add(Dense(90))
model.add(Dense(1))


texts: How to use a cross entropy loss function for each letter/digit in a captcha?

texts:
I'm trying to develop a captcha solver using a simple fully-connected neural network in TensorFlow. All captchas have 5 digits/letters, each character can be a number 0-9 or a letter A-Z. They all look like this:

Each captcha has a label like this for example: "AB4D1". I take these labels, split them ['A', 'B', '4', 'D', '1'], convert each of the characters to a number between 0 and 35 (26 letters and 10 digits) and convert each of those to one-hot representations [0, 0, 0, 1, 0,..]. Then I concatenate the one-hot arrays for all the letters in a captcha so I end up with a single 5*36-length array for each image. (This array has 5 ones and the rest zeros)

Now I'm wondering what loss function to use. I tried mean-squared-error but that didn't work at all for some reason. Now I'm trying to use cross-entropy (with softmax first) but as far as I understand it, I would need to apply it to each letter separately and then add up the loss for each letter since softmax requires exclusive categories.

I tried splitting the model output and labels:

split_logits = tf.split(logits, num_or_size_splits=5, axis=1)
split_train_labels = tf.split(tf_train_labels, num_or_size_splits=5, axis=1)

This works afaik. Now I just add the cross-entropy losses for each letter together.

loss =
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[0], logits=split_logits[0])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[1], logits=split_logits[1])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[2], logits=split_logits[2])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[3], logits=split_logits[3])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[4], logits=split_logits[4]))

But this is where I have a problem. As long as I only have the line for the first (0th) digit/letter in there it works really well (But obviously only learns recognizing the first letter). As soon as I add the term for another letter, or or just have a single other letter instead of the first one, the network immediately plateaus after training starts and never improves to above-random performance.

I'm not sure how to fix this. I'm thinking the problem is either in my loss function or in my network structure.

Is the problem that all the letters except the first one are varying a lot in their position due to the different font sizes/widths? I understand that convolutional networks would be much better for this but I really want to try it with fully-connected networks first.


texts: Using Iterative Hard/Soft Thresholding in autoencoder with non linear activation

texts:
Can someone please give an intuitive explanation of the difference between the Iterative Hard Thresholding VS Iterative Soft thresholding algorithm?
And if we can use these algorithms in an autoencoder with a non-linear activation function?


texts: Deep Reinforcement Learning for dynamic pricing

texts:
I am trying to implement a Deep Q Network model for Dynamic pricing in Logistics. I can define

State Space (Origin, Destination, type of the shipment, customer, Type of the product, Commodity of the shipment, AVAILABILITY of capacity etc.
Action Space (price itself, can range from 0 to inf) we need to determine the price itself.
Reward Signal (Rewards can be based on a similar offer to other customers, seasonality, remaining capacity.

I am planning to use Multi-Layer Perceptron for getting inputs from the state space and the outputting the price.

I am not sure how to define a reward function. Please help me in defining the mathematical formula for the reward function based on the price as an action space?

-- UPDATE --

State space that evolves over the time is the remaining capacity (Logistics).
Consider at the initial time step is 10,000 kgs capacity and at over a period of time the capacity decreases and when the capacity is full and it cannot take anymore shipments, then the episode completes.

The agent will have to find an optimal price based on the following rewards.


texts: What is the correct way to apply KNN to a time-series using a rolling window?

texts:
I have a time-series. The index is weekly dates and the values are a certain indicator that I made. I think I understand how to apply KNN in this situation but I'm not sure how exactly to do it.

This is what I currently have:

1. Set lookback period to 200 rows (which is 200 weeks)
2. Set the KNN value to 10 Nearest Neighbors
3. Start with the 201 st row
4. Get the previous 200 days
5. Do Knn.fit(x = prev_200_row, y = profit_after_each_row, neighbors = 10)
6. Do Knn.predict(current_row)
7. Save the prediction to a list
8. Go to the next row and repeat steps 4, 5, 6, 7, 8 until no more rows.

Is this is a correct use of the process? Eventually, I would like to compare different values for K (10 nearest neighbors or maybe 50 nearest neighbors).
I am going to build on this so I want to make sure that I am starting correctly. I am wondering if it is correct that the Knn.fit is being called after every row but it seems correct to me.


texts:
I have a classification that has to predict three different classes: gcc,icc, clang. The prblem is that if I use a blind test set to do a submission, when I look athe the prediction I have on it I find that most of the predictions are only of one type. So I have the following situation:

My code is the following:

#pakages
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

print('Libraries imported.')

dataFrame = pd.read_json('training_dataset.jsonl',lines = True)
dataFrame

dataFrame['opcodes'] = dataFrame['instructions'].apply(lambda x:[i.split()[0] for i in x])

df = dataFrame[["opcodes", "compiler"]].copy()
df

df['opcodes'] = [" ".join(opcode) for opcode in df['opcodes'].values]

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer=TfidfVectorizer(ngram_range=(2,2),min_df=2,max_df=0.5)

df_x = df['opcodes']

X_all = tfidf_vectorizer.fit_transform(df_x)
y_all = dataFrame['compiler']

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all,
    test_size=0.2, random_state=15)

from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=100).fit(X_train,y_train)

y_pred = clf.predict(X_test)

acc = clf.score(X_test, y_test)
print("Accuracy %.3f" %acc)

Now, using a random forest classifier I get an accuracy of 0.89, but the problem is that the model predicts mostly the class gcc, so I guess something is wrong.

I was almost soing a submission when I noticed this.

Does this happen because I didn't consider the fact that I have a multiclass classification problem? I really don't understand.

Can somebody please help me? Thank's in advance.

[EDIT] The dataset is balanced.

[EDIT 2] When try to do the prediction on the blind test set I do the following:

test = pd.read_json('test_dataset_blind.jsonl',lines = True)
test

test['instructions'] = [" ".join(opcode) for opcode in
test['instructions'].values]

tfidf_vectorizer=TfidfVectorizer(ngram_range=(2,2),min_df=2,max_df=0.5)
df_x_new = test['instructions']

X_new = tfidf_vectorizer.fit_transform(df_x)

new_pred_class = clf.predict(X_new)[:3000]
len(new_pred_class)

sub_compiler = pd.DataFrame({'instructions':
test['instructions'],'compiler':new_pred_class})
sub_compiler.to_csv('sub_compiler.csv')
sub_compiler.head()

[EDIT 4]anctually the problem seems to be on how I select the data to test. Infact, if I do new_pred_class = clf.predict(X_new)[-3000:], I get:

`
so this time mostly the clang are predicted.

I have 30000 rows in the original dataset and 3000 in the blind test set.

[EDIT 5]So I think I have to reorganize the select the new_pred_class taking random values from clf.predict(X_new)[:3000], so what I mean is that I should take 3000 random values. How could I do that? Thank's again.

EDIT 6] if I do type(X_new)  I have scipy.sparse.csr.csr_matrix


texts: What's the difference of stateless LSTM and a normal feed-forward NN?

texts:
From what I understand, the whole point of LSTM is for the network to establish long-term dependencies in the data, i.e. an event happening now may be in some way determined by something that happened sometime in the past, and that may not be in the batch of data currently being presented, but many sequences or batches previously.

So I understand that a stateful LSTM does this, and the network will retain the state until reset_states() is called.  But with stateless LSTM, where the state isn't retained between batches, then how does that differ from a normal feed-forward perceptron network?

I'm assuming that even though in stateless LSTM you don't control the states as you would in a stateful model, that some state is still retained at passed on between sequences?  Is it passed on between batches?

I can understand why someone would use stateful LSTM, but why would someone use stateless LSTM instead of a feed-forward perceptron network?  Or vice-versa, why would someone use a regular FF instead of a stateless LSTM?


texts:
I've searched quite a bit and haven't landed on any useful results.

The problem statement is:
Given a set of vectors, I wish to find its approximate k-nearest neighbors.
The caveat here is that each of my dimensions resemble a different entity and hence we cannot use the same weight for each dimension while computing the distance.
Thus, solutions like kd-tree don't work as is.

Is there any data-structure or any alternate algorithm that I can use to find such approximate weighted k-nearest neighbors.

Note: Multiplying the initial input data with their weights so as to get a uniform weight is not an option.


texts: Validation loss much higher than training loss

texts:
I am training a CNN on some text data. The sentences are padded and embedded and fed to a CNN. The model architecture is:
model = Sequential()
model.add(Embedding(max_features, embedding_dims, input_length=maxlen))
model.add(Conv1D(128, 5, activation='relu'))
model.add(GlobalMaxPooling1D())

model.add(Dense(50, activation = 'relu'))
model.add(BatchNormalization())

model.add(Dense(50, activation = 'relu'))
model.add(BatchNormalization())

model.add(Dense(25, activation = 'relu'))
#model.add(Dropout(0.2))
model.add(BatchNormalization())

model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

Any help would be appreciated.


texts: Multiclass Classification that includes a Geospatial Element

texts:
I am attempting to train a classifier to predict different prices for an item in different suburbs. I have several features, two of which are a latitude and longitude for the centroid of the suburb.

I am attempting to train the model to classify the price of an item in a bin of $10 size. The geospatial element will definitely affect the price of the item, however the training data I have will have gaps in it (i.e. I don't have prices for all suburbs).

What is the best way to engineer a feature that will include this geospatial information and be able to fill in the gaps in the training/test data?

So far I have tried creating new features for the bearing and distance from the capital city which seemed to work okay, as well as binning the latitude and longitude which performs worse than the bearing/distance. I did consider using a geohash, however I think that this will be too complex a feature for a classifier to understand.