texts:
I have a number of lists, such as [1,2,3,4], [2,3,4], [1,2], [2,3,4,6,8,10], whose lengths are obviously unsame.

How can I use this as input of placeholder in Tensorflow?

As I have tried, the following setting will raise error.

tf.constant([[1,2],[1,2,3]...],dtype=tf.int32)

So I guess placeholder cannot be set by the upper input of lists.

Is there any solution?

Edit:

The following is my example. How to make it run without errors?


texts:
In a recent adult education class I took the prof shared a spreadsheet that was written in VBA code and takes ages to execute, I'm sure Python (SciPy, Pandas, Numpy) can handle this much faster.

The problem is a Direct Clustering Algorithm (DCA) one, all the searches I've done to solve this keep showing solutions trying to use K-mean which is not applicable.

The x-axis is different manufactured parts, and the y-axis are the different machines used in the manufacturing of the parts. 0 = not used, 1 = used.

The best description I can come up with is this:  It's trying to make as clean a diagonal across the matrix from the top left to the bottom right based on the 1s as possible.

Below shows the flow of how the VBA solution works through the problem.  In my final Python solution I'm not worried about the flow matching, just the final output.
Starting data set:

           Part
          a b c d e f
machine a 1 1 0 1 0 0
        b 0 0 1 0 1 0
        c 1 0 0 1 0 0
        d 0 0 1 0 0 1
        e 0 0 1 0 0 1

Step 1, Need columns decreasing and rows increasing number of 1s:

           Part
          c f d a e b
machine e 1 1 0 0 0 0
        d 1 1 0 0 0 0
        c 0 0 1 1 0 0
        b 1 0 0 0 1 0
        a 0 0 1 1 0 1

Step 2, Part d moved, move machine b up to obtain the following matrix:

           Part
          c f d a e b
machine e 1 1 0 0 0 0
        d 1 1 0 0 0 0
        b 0 0 0 1 0 0
        c 1 0 1 0 1 0
        a 0 0 1 1 0 1

Then part e was moved to obtain the following end matrix:

           Part
          c f e d a b
machine e 1 1 0 0 0 0
        d 1 1 0 0 0 0
        b 1 0 1 0 0 0
        c 0 0 0 1 1 0
        a 0 0 0 1 1 1

As you can see, the final product is a diagonal of the 1s across the matrix.

To put in in perspective, I've got thousands of part numbers to work with and the VBA solution shared in a spreadsheet can handle 350 at best but takes a LONG time (~30 min) to execute.

I have tried to search but it seems this kind of grouping which a mean/average isn't applicable doesn't seem to come up in the articles I could find.

I really appreciate the help in solving this.


texts:
I am in the process of learning what a convolutional layer is, and its functions. I have a couple of questions. One is what in_channels and out_channels are. In my code, I have the following:
nn.Conv2d(in_channels=3, out_channels=64, kernel_size=(4,4), padding=1)

I understand that the in_channels for this layer is 3 because RGB, but I'm confused on why out_channels is 64, and why I can change it to seemingly any other number without the program crashing.
Another question is what exactly a "neuron" is. For convolutional layers, is a neuron each filter? Also, if it is, what do the activation functions judge when evaluating each filter?


texts:
It is not clear to me how to calculate similarity between two products from the example.

How do they calculate that?


texts: Can you sum up gradients and apply in batch?

texts:
I'm following this Reinforcement Learning Tutorial. There, training data is collected during an episode. When the episode is done, the data is used to do backpropagation. However, instead of applying the calculated gradients to the model, they are stored in a buffer. The gradients of multiple episodes get summed up before they are applied.

I wonder if that is a reasonable thing to do. Usually, if the model is very good after training with the first episodes data, the gradients of the other episodes will be close to 0 (no change). If you used the tutorials code, the gradients of all episodes would be large, and the model would move away from the optimum. Am I missing something?

Edit: I tried buffering gradients with Q-Learning and compared it to directly updating the weights. It is working, but buffering is definitely worse.


texts:
I was self-teaching myself. I totally understand why depth of a neural network affects the learning and how it differs than its width. But I am looking for some theoretical justification about it. Papers I could come up with, e.g., Benefits of depth in neural networks or
The Power of Depth for Feedforward Neural Networks   are unfortunately is too deep and long. I am not also super good at Mathematics, however, I believe there must be a plain, short and compact math behind it. Can someone point me to some tutorials/articles/papers/reports, where I can easily understand it?


texts: How to force histogram plots to have same axes?

texts:
I am comparing my trained model with other benchmark models with the error histogram but the axis of histogram is different for each method as shown in figure.For instance to plot the error histogram of every method,I tried this code:

% Matlab code
Targets=Actual;
Outputs=Predicted_by_model;
errors=Targets-Outputs;
error_std=std(errors);
MAPE=mean(abs(Targets-Outputs)./Targets)*100;
histfit(errors);
legend('Proposed')
title(['MAPE = ' num2str(MAPE) '  ,    Error St.D. = ' num2str(error_std)]))

How to keep axis of every method to the same value.


texts:
I'm training a Scikit model but it seems that in all examples, they call the fit method on the entire training set. What I want to do however is call it per sample (i.e. looping through all samples). This has multiple reasons but most importantly are

MemoryError with my huge trainingset
Training with new data instead of recompiling entire model

Yet when I loop and call the fit model self.gnb = self.gnb.fit(sample.data,labels) and then debug, the gnb model only has one class (namely the last one). So how should I approach this?


texts: Are single input single output neural networks possible?

texts:
This might be a weird question but I'm trying to have a deep understanding of how neural networks work theoretically.

I was doing some tests with my perceptron and I decided to test it on a single input single output dataset. What I was looking for was 100% accuracy since what I was testing on was a trivial separable dataset with binary output. However, the accuracy was below 50% (it wasn't balanced). I realized that this is due to the fact that there is a single weight that when trained is either going to be =>0 or <0. So after going through the activation function (e.g. hardlim), unless the input is scaled to be [-1,1] range, it's always going to give the same output.

Does my explanation make sense? Are there any theoretical aspects I'm missing?

Also, are neural networks useful at all for single input problems? They seems useless to me since we're basically classifying by putting a point or points (for multiclass problems) on a single line to separate outputs, which is a pretty simple problem that doesn't need the intricacy of neural networks.


texts: Training a predictive model on multiple small multivariate time series

texts:
I'm looking to build a predictive model for hockey players individual statistics. My goal is to predict how many points a player can be expected to have at the end of the season.

To do so, I thought I could base myself on the progression of the player's statistics over the year. I currently have 10 years worth of data and, for example, a player could have played in 5 of those 10 years. In this case, I'm able to see if this particular player is on an increasing or decreasing curve regarding his statistics.

So, to resume, for some thousands of players the data goes like this:

  Player #1

  Season 2005-2006, Points : X, Game played : y, ...

  Season 2006-2007, Points : X, Game played: Y, ...

  Season 2007-2008, Points : X, Game played: Y, ...

  ...

  Player #2

  ...

My main problem is that I don't know what kind of data this is so I can't find more information on how to build my model. In my title I've written "multiple small multivariate time series" for a lack of better name. I've obviously looked into multivariate time series to fit them in an RNN architecture, but it doesn't seem to fit because of the multiple time series that aren't really related.

I've found this question which kind of relates but there were no answer and the related comments didn't help me.

All in all, I'm trying to find some guidance as to what kind of model I could use to work with my data. I'd also like to know if there's a name for this kind of data, so I can inform myself better.


texts:
I am doing something wrong with X in these time series plots, but what?

This code using plotly yields the following ridiculous time series line plot:

import plotly.graph_objects as go
fig = go.Figure([go.Scatter(x=df.index, y=df['Weekly_Sales'])])
plt.savefig('plot2.png', dpi=300, bbox_inches='tight')
fig.show()

Here is the basic data:

To show the error on my part (whatever it is) is ongoing, here is a second time series plot:

# Spec the chart size
plt.figure(figsize=(12,6))
x = df.index

#Plot the lines
plt.plot(x, weekly_roll)
plt.plot(x, monthly_roll)
plt.plot(x, quarterly_roll)
plt.plot(x, yearly_roll)

# Spec the x and y labels, chart title, and axis ticks
plt.yticks(fontsize=18)
plt.xticks(fontsize=18)
ax = plt.axes()
ax.set(xlabel='Time', ylabel='Revenue',
       title='Rolling Averages - Weekly Sales');
plt.savefig('plot4.png', dpi=300, bbox_inches='tight')
plt.show();

Here is the resulting image:

While both plots have a certain modern flair, it's not what I am going for in a time series plot. Help?

Here is the basic data from the head() of the df:

Date    Store Dept Weekly_Sales IsHoliday
2010-02-05  1   1   24924.50    False
2010-02-12  1   1   46039.49    True
2010-02-19  1   1   41595.55    False
2010-02-26  1   1   19403.54    False
2010-03-05  1   1   21827.90    False


texts: Reasons and prevention of trivial (and less trivial) misclassification errors?

texts:
I was not sure about posting this question with mentioning the name of the company, which I quite respect and admire. However, I've figured that a wider exposure might help the team to fix this and similar problems faster as well as increase the quality of the machine learning (ML) engine of their website.

The problem exposes itself by too many occurrences of a quite trivial misclassification error on Amazon's book categories classification (which I'm a frequent visitor of). In the following example, the underlying reason of such behavior is quite clear, but in other cases the reasons might be different. I am curious about what could be other potential reasons for misclassification and what are the strategies/approaches to avoiding such problems. Without much further ado, here's how the problem appears in real life.

I was reviewing some books, related to transitioning from graduate programs (Ph.D., in particular) to work environment in academia. Among several other books, I ran across the following one:

So far, so good. However, let's scroll down a bit further to see the the books ratings in relevant categories. We should expect Amazon to figure out categories, relevant to the book's discipline, topic and contents. How surprised was I (and that's an understatement!) to see the following result of Amazon.com's sophisticated ML engine and algorithms:

Clearly, the only fuzzy fact that connects this book with the subject "Audiology and Speech Pathology" (!) is IMHO the author's last name (Boice), which, is close to the word "voice". If my guess is correct, Amazon's ML engine, for some reason, decided to take into account the book's lexicographical attribute instead of the book's most important and most relevant attributes, such as title, topic and contents. I've seen multiple occurrences of similar absolutely incorrect ML-based decision making on Amazon.com and some other websites. So, hopefully my question makes sense as well as interesting and important enough to spark a discussion: What could be other potential reasons for misclassification and what are the strategies/approaches to avoiding such problems? (Any related thoughts will also be appreciated.)


texts:
I'm learning about neural networks and machine learning in general and wanted to perform a simple binary operator implementation in Tensorflow. I know implementing an operator is simple and is not a usual task for a neural network, but I want to know if I can set up a neural network and let it learn the operator for itself.

The main goal is to use two input nodes and one output node and train an AND operator. The problem is there are only four samples altogether:

0 && 0 == 0
0 && 1 == 0
1 && 0 == 0
1 && 1 == 1

When I train my network with this I always get extremely poor examples (probably because of underfitting?)

Now is this even doable? Should I even approach it this way? I can update the question with my code if it is relevant. Thanks for any insights!


texts: Are DBSCAN and dbscan from the sklearn.cluster package different?

texts:
I'm new to DBSCAN. I was looking at a few examples online and came across a few instances where the following lines were used while importing the dbscan module:

from sklearn.cluster.dbscan_ import DBSCAN
from sklearn.cluster.dbscan_ import dbscan

I would like to know if there is anything different between them? Or is it necessary for me to import both DBSCAN and dbscan?

Here's one link where both lines are used in the import:
https://gemfury.com/stream/python:scikit-learn/-/content/cluster/tests/test_dbscan.py


texts: Predict probability when model was trained in balanced dataset

texts:
I have a dataset of about 1M observation and I had to predict a response that occurs only about 10.000 times (1%).

I decided to train a random forest, but this takes a lot of time to train because the data is too large for my hardware. So I decided to take a sample, but an aleatory sample would be already too large to have a minimum quantity of response. (If  I take 10% aleatory, i would have only 1000 response)

Then I took a stratified sample. All responses and 10.000 aleatory non-responses, and trained my model in this dataset.

But now I need to rescale the probability so I have the real probability of the observation to be response.

I tried to simulate this problem with this code in R. Training a model in a balanced dataset and another one in the unbalanced data. But those models are not very correlated and I didn't find a good way to tranform the probabilities to the original unbalanced scale.

Found this is a good reference for future readers

I found that for logistic regression I can do this by just changing the intercept this way:

$$ \hat{\beta_0}  =  \hat{\beta_0^*} - log(\frac{\gamma_1}{\gamma_2})$$

Where $ \gamma_1 = Pr(Z=1|Y=1)$ and $ \gamma_2 = Pr(Z=1|Y=0)$. $Z$ is the an aleatory variable indicating if the observation is in the reduced dataset.

This was found in this book (in portuguese) page 216

simulate_data <- function(n){
  X <- data.frame(matrix(runif(n*20), ncol = 20))
  list(
    X = X,
    Y = rbinom(n, size = 1, prob = apply(X, 1, sum) %>% pnorm(mean = 13)) %>% as.factor()
  )
}

balance <- function(X, Y){
  X <- rbind(
    X[Y == "1",],
    X[Y == "0",] %>% sample_n(length(Y[Y == "1"]))
    )
  return(list(
    X = X,
    Y = as.factor(c(rep(c(1,0), each = length(Y[Y == "1"]))))
  ))
}

train <- simulate_data(100000)

library(randomForest)
m_desb <- mean(train$Y == "1")
modelo_desb <- randomForest(train$X,train$Y, ntree = 200, cutoff = c(1-m_desb, m_desb), nodesize = 30, mtry = 8)
bal <- balance(train$X, train$Y)
m_bal <- mean(bal$Y == "1")
modelo_bal <- randomForest(bal$X,bal$Y, ntree = 100, cutoff = c(1-m_bal, m_bal), nodesize = 50)

Code for plot

library(ggplot2)
data.frame(
  unbalanced = predict(modelo_desb, newdata = test$X, type = "prob")[,2],
  balanced = predict(modelo_bal, newdata = test$X, type = "prob")[,2]
) %>%
  ggplot(aes(x = balanced, y = unbalanced)) +
  geom_point(size = 0.3) +
  xlim(0,1) +
  geom_smooth() +
  geom_hline(yintercept = m_desb, linetype = "dashed") +
  geom_vline(xintercept = m_bal, linetype = "dashed")


texts: How to apply the gradient of softmax in backprop

texts:
I recently did a homework where I had to learn a model for the MNIST 10-digit classification. The HW had some scaffolding code and I was supposed to work in the context of this code.

My homework works / passes tests but now I'm trying to do it all from scratch (my own nn framework, no hw scaffolding code) and I'm stuck applying the grandient of softmax in the backprop step, and even think what the hw scaffolding code does might not be correct.

The hw has me use what they call 'a softmax loss' as the last node in the nn. Which means, for some reason they decided to join a softmax activation with the cross entropy loss all in one, instead of treating softmax as an activation function and cross entropy as a separate loss function.

The hw loss func then looks like this (minimally edited by me):

class SoftmaxLoss:
    """
    A batched softmax loss, used for classification problems.
    input[0] (the prediction) = np.array of dims batch_size x 10
    input[1] (the truth) = np.array of dims batch_size x 10
    """
    @staticmethod
    def softmax(input):
        exp = np.exp(input - np.max(input, axis=1, keepdims=True))
        return exp / np.sum(exp, axis=1, keepdims=True)

    @staticmethod
    def forward(inputs):
        softmax = SoftmaxLoss.softmax(inputs[0])
        labels = inputs[1]
        return np.mean(-np.sum(labels * np.log(softmax), axis=1))

    @staticmethod
    def backward(inputs, gradient):
        softmax = SoftmaxLoss.softmax(inputs[0])
        return [
            gradient * (softmax - inputs[1]) / inputs[0].shape[0],
            gradient * (-np.log(softmax)) / inputs[0].shape[0]
        ]

As you can see, on forward it does softmax(x) and then cross entropy loss.

But on backprop, it seems to only do the derivative of cross entropy and not of softmax. Softmax is left as such.

Shouldn't it also take the derivative of softmax with respect to the input to softmax?

Assuming that it should take the derivative of softmax, I'm not sure how this hw actually passes the tests...

Now, in my own implementation from scratch, I made softmax and cross entropy separate nodes, like so (p and t stand for predicted and truth):

class SoftMax(NetNode):
    def __init__(self, x):
        ex = np.exp(x.data - np.max(x.data, axis=1, keepdims=True))
        super().__init__(ex / np.sum(ex, axis=1, keepdims=True), x)

    def _back(self, x):
        g = self.data * (np.eye(self.data.shape[0]) - self.data)
        x.g += self.g * g
        super()._back()

class LCE(NetNode):
    def __init__(self, p, t):
        super().__init__(
            np.mean(-np.sum(t.data * np.log(p.data), axis=1)),
            p, t
        )

    def _back(self, p, t):
        p.g += self.g * (p.data - t.data) / t.data.shape[0]
        t.g += self.g * -np.log(p.data) / t.data.shape[0]
        super()._back()

As you can see, my cross entropy loss (LCE) has the same derivative as the one in the hw, because that is the derivative for the loss itself, without getting into the softmax yet.

But then, I would still have to do the derivative of softmax to chain it with the derivative of loss. This is where I get stuck.

For softmax defined as:

The derivative is usually defined as:

But I need a derivative that results in a tensor of the same size as the input to softmax, in this case, batch_size x 10. So I'm not sure how the above should be applied to only 10 components, since it implies that I would diferentiate for all inputs with respect to all outputs (all combinations) or  in matrix form.


texts:
I am looking to start learning openCV to perform a car detection program. Should I know data science and/or machine learning stuff before going for openCV?

If not, where can I start learning openCV with Python?


texts: Reinforcement Deep Learning for object detection

texts: After reading the state of the Art of object detection using the CNN's(R-CNN,Faster R-CNN,YOLO,YOLOv2,SSD) I was wondering if there is an efficient method that use deep learning with reinforcement learning for object detection ?


texts: 100 items 100 baskets divisor association analysis problem

texts:
I have the following exercise question:

Suppose there are 100 items, numbered 1 to 100, and also 100 baskets, also numbered 1 to 100. Item i is in basket b if and only if i divides b with no remainder. Thus, item 1 is in all the baskets, item 2 is in all fifty of the even-numbered baskets, etc. for example Basket 12 consists of items {1, 2, 3, 4, 6, 12}

Given this, I'm trying to solve the following 3 questions:

(a) If the support threshold is 5, which items are frequent?
(b) what is the confidence of the following association rules?
(1) {5, 7} → 2.
(2) {2, 3, 4}→ 5.

which way I should be approaching these types of questions?


texts: Does anybody know where this rule of thumb came from? Rule of them: embedding vector dimension should be the 4th root of the number of categories

texts:
I was taking an online ML course and the lecturer said that a rule of thumb for choosing the number of dimensions when embedding categorical data is the following
embedding vector dimension should be the 4th root of the number of categories
The lecturer worked for Google and when I looked on the internet for this I only found a Google blog which quickly mentioned it google blog link. I'm guessing it's something that they came up with at Google but was wondering if somebody else has maybe seen it in a research paper.


texts: How to determine the most active time of a user from active directory logs

texts:
I have user data login logs from Active directory data that lists when a user was logged in to a server. I am looking to find a deviation in the normal behavior of the user. For example if a user is usually active from say 9am to 5pm, if there is some activity outside this, it should be detected.
I am unsure if the min(first logon) and the max(last logon) should be considered.
Any input will be appreciated.


texts: How many models shall be used in ensemble modelling?

texts: I wish to make an ensemble of deep models to solve a classification problem. I want to know how many models shall be used to create that ensemble to ensure unbiased results. I have head 30+ models shall be used to determine the skill of the model but cannot find anything relating to ensemble modelling.


texts: Does logistic regression can only solve binary classification problem?

texts:
Does logistic regression only solve binary classification problems?
How can logistic regression solve multiple-class problems?


texts: why do we have to calculate the entropy of parent node in Information Gain?

texts:
Why do we need the entropy of parent node in the Information Gain.
Information Gain = entropy(parent) - w * entropy(children)

We can compare the entropy of the children without the need for the parent entropy.


texts: predicted value is more conentrated than actual with DNN

texts:
I have created an Deep Neural Network model with keras.
def nopitid_model(optimizer,loss='mean_squared_error',epochs=50):
model = Sequential()
model.add(Dense(24, input_dim=24, kernel_initializer='normal', activation='relu'))
model.add(Dense(15, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(5, activation='relu'))
model.add(Dense(1,))
# # Compile model
model.compile(loss=loss, optimizer=optimizer)
prediction = model.fit(x=X_norm,y=Y,batch_size=int(X_norm.shape[0]/5),
          verbose=0,
          epochs=epochs,
          validation_split=0.3)
return model,prediction

The MAE is about 5%, which is acceptable. However, after plot the predicted value vs actual value, I have noticed that the predicted value scattered between 1260 - 1290, whereas actual data range from 800-1600. More interesting, some predicted values also can be observed at 1230. I can't figure out what has caused this.
Looking for all suggestions.


texts:
I am trying to figure out an approach for calculating the probability of a renter making > 1 booking. I would prefer to use Python for this project. I have the following columns ready:
State, Category (fun, adventure, etc.), returning_renter? (TRUE if >= 1 booking).

What would be the most efficient approach for this project? Are there any particular Python libraries you would recommend?


texts: types of metrics to evaluate the performance of regression models?

texts:
I have a conceptual question related to types of performance analysis of regression models.

In general we have -
R Square, Root Mean Squared Error and Mean  squared Error to evaluate Regression models.

Then we have ROC AUC curves to evaluate the performance.

Is their anything else we can use to evaluate the model performance apart from these techniques. I have heard about statistical tests being used in performance evaluation but how do we use them once we have a model.


texts: Different testing and training accuracy values within a NN TensorFlow structure

texts:
In order to select the optimum number of my gradient descent algorithm, I had used a for loop of 1500 iterations and each 100 iterations training and testing accuracies are printed. Here everything is ok than I selected my iteration number that gave me 95% for training and 91% for testing and I had evaluated my trained model using the next code:

for i in range(0,half):
    x_train = np.reshape(Train_dat[i,:],[1,500])
    t_train = np.reshape(Train_target[i,:],[1,6])
    train= {x: x_train, t:  t_train}
    evaluation= pred.eval(feed_dict=train,session=session)
    est_classTrain[i] = np.argmax(evaluation)
    x_test=np.reshape(Test_dat[i,:],[1,500])
    t_test = np.reshape(Test_target[i,:],[1,6])
    test= {x: x_test, t: t_test}
    evalua= pred.eval(feed_dict=test,session=session)
    est_classTest[i] = np.argmax(evalua[0])

The problem is that after executing the above code performances have been reduced to 90% and 69% for training and testing, respectively.
My question is why I got different values?


texts:
I have a collection of 1000s of bottles of wine. I want to understand what could / likely to be driving the price either up or down. Below is an example of the data

Wine      Country       Area        Grape        Class    Year    Price
A         France        Burgundy    Pinot Noir   Grand    2014    +6%
B         France        Burgundy    Chardonnay   1er      2014    -1%
C         France        Burgundy    Pinot Noir   1er      2014    +4%
D         USA           California  Pinot Noir            2013    +1%
E         USA           California  Blend                 2014    +0.5%
F         USA           California  Chardonnay            2014    -4%
G         USA           California  Chardonnay            2013    -5%

From these fictitious numbers i would assume the reason bottle A is up is because pinot is up but more importantly France is up V USA wine. To the same extent Chardonnay is doing poorly but USA again is doing worse than France.

Here this is rather easy but if this was extended by 1000s of rows with more categories added, the exercise becomes less easy to see the trend.

My original thought would be to use the importance of a feature from a decision tree. But is there a better way?

Thanks


texts: How to make a neural network output a specific number out of a certain range?

texts:
I have a neural network with an Input layer, 2 hidden Dense layers and an Output layer.
I would like for each neuron in the Output layer to give me a number between 0 and 2 (either 0, 1 or 2), like so:

If I use a neuron for each possibility (a neuron for 0, a neuron for 1 and another for 2) and then pick the one with the best prediction, the output layer length would be far too much.
Is there a way to implement this ? (I am fairly new to neural networks and the like)


texts: Choosing the right parameters for SARSA and Q-Learning & Comparing Models

texts:
What is the correct way to fine-tune a model's (SARSA(0), SARSA(lambda), Q(0), Q(lambda)) parameters, and how can one compare the models?

I read that typically one compares the number of actions needed to achieve the goal, after X episodes. However, my model is slightly different than maze/grid worlds, so I'm not sure on how to do it.

Essentially its a time model, with X-days, and each day an action is selected, the reward on each day depends on the action selected, and the end-state reward depends on the actions selected in the other 249 days. It is an optimization model for a regulatory situation, which implies revising the whole reward every 250 days, based on the performance (which is tied to actions) of the other 249 days.

With that being said, I cannot measure the performance based on the number of actions needed to reach the terminal state, as it will always be reached with the same number of actions.

The models are being trained in a simulation, the optimal policy will be achieved once per model. The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state).

Ideally one would compare all possible models - as there are quite a few - with a smaller number of episodes, and the selected ones would then run with the full amount of episodes, for time-reasons.

Any suggestions on how to compare the models? Based on average rewards with the optimal policy's actions?


texts: Object detection is not improving although loss is decreasing

texts:
I am working on a project for detecting buildings from satellite images (using OpenStreetMap data as labels) using the Tensorflow Object Detection API. I recently upgraded to Tensorflow 2 and chose EfficientDet D4 from the Model ZOO.
I train (fine-tune) this pre-trained model on my data. Throughout the training, I keep making predictions using always the current state of the model. The point is to see how the performance of the model gets better with training.
Problem: The ability to predict buildings in validation dataset is not improving at all. In fact, it is getting worse! However, the loss during training is constantly decreasing.
Here is an example - in blue are predictions of the model after less than 1 epoch while in red are predictions by a model trained much more:

To me, it doesn't make any sense. I've spent loads of time trying different variations of this, using another dataset for evaluation, etc. But still I get worse results (in terms of precision, recall, f1) when I train the model more. And this is not due to overfitting, the model's performance starts to decrease almost immediately.
Earlier, I was using Tensorflow 1 and SSD Inception V2 as a model. That worked as expected, i.e. the model's ability to detect buildings was improving as the training progressed.
Question: What could be the cause of this and what can I try to solve this problem?I have run out of ideas.
I am new to machine learning so I am likely missing some important information here. Please let me know if more details are needed to answer this.


texts: When visualizing data that has <1 or <5 ppm how do you display this?

texts:
I have some data (parts per million) where some of it is 1 or greater than one (but has an actual number.) However, some of the data simply lists "<1" ppm.

What is a good way to visualize this in a graph?

Should I pick an arbitrary decimal less than one, so that it can be shown on the graph? I feel setting it to zero would not be right either, as they went to the extent of listing it, so it must be significant.


texts: sub-setting based on categorical variable in R

texts:
the below dataset is just top 8 rows of 100 rows dataset.

>a
  group marks    upd
2     C     3     up
3     C     4   down
6     C     7     up
1     T     2     up
4     T     5     up
5     T     6   down
7     T     1 middle
8     T     0   down

I used unique_groups=unique(a$group) to get a list of unique group names to be sent into a function.

Now, I need a function that gives a subset of this dataset where it takes each group from unique_groups and checks if 'upd' is in ('down','middle') and gives me a dataset having only those records having upd ('down','middle') of that respective group.

I tried to use do.call function to get the dataset that I want as-

do.call(rbind,sapply(unique_groups,FUN=function(y)
  {
  h=a[a$group==x & a$upd %in% c('down','middle'),]
  h
}
, simplify=F))

I wasnt getting the dataset that I want and it was giving this warning message:

In ==.default(a$group, x) :
  longer object length is not a multiple of shorter object length


texts: how many rows have values from the same columns pandas

texts:
I have a df with many columns that represent the market cap of companies that compose an index. The index of the dataframe is dates. Before the company enters the index or after it leaves it, the market cap of the company in the df is zero.

I want to know the mean number of days in which the index doesn't change.

for example:

[in]:
df = pd.DataFrame(np.array([[1, 1,np.nan], [np.nan,2, 10], [1,3, 100],[4,np.nan, 100]]),
                           columns=['a', 'b','c'])

df
[out]:
        a   b   c
    0   1.0 1.0 NaN
    1   NaN 2.0 10.0
    2   1.0 3.0 100.0
    3   4.0 NaN 100.0

what I want to know is how many rows have entries from the same column.

For example, row 0 has entries from columns a and b. row 1 from columns b and c. row 2 from columns a,b and c. an row 3 from columns a and c. therefore there are 4 rows with unique column combinations and 4 changes. The mean is then 1.


texts:
This is a quick question. If I compare neural network and random forest, the data size requirement is huge in neural network, but a decision tree or random forest can work with less number of records too.
Does any such problem occurs with XGBoost as well? Does it also need a lot of data so that it can go in multiple iterations to reduce the error term?


texts: When I deploy my web service in AZURE Machine learning, is it possible to have NULL or empty in my test data set? Or I have to fill all the information to get the results of my prediction?


texts: How to detect product name from the bill text?

texts:
I have to detect price and product name from given text.

Where should I look to if I am new to this topic ?

I don't know all product names. It can be anything.

Example 1

Hello Denys, thank you for purchasing beer. Enjoy!

Example 2

Hello Denys, here is your peanuts.

Products are - beer, peanuts.

P.S. I'm familiar with Ruby and found this library - https://github.com/jekyll/classifier-reborn. Is it applicable to this task ?


texts: How to implement HashingVectorizer in multinomial naive bayes algorithim

texts:
I had used TfidfVectorizer and passed it through MultinomialNB for document classification, It was working fine.

But now I need to pass a huge set of documents for ex above 1 Lakh and when I am trying to pass these document content to TfidfVectorizer my local computer hanged. It seems it has a performance issue. So I got a suggestion to use HashingVectorizer.

And I used below code for classification(Just replacing TfidfVectorizer by HashingVectorizer)

stop_words = open("english_stopwords").read().split("\n")
vect = HashingVectorizer(stop_words=stop_words, ngram_range=(1,5))
X_train_dtm = vect.fit_transform(training_content_list)
X_predict_dtm = vect.transform(predict_content_list)
nb = MultinomialNB()
nb.fit(X_train_dtm, training_label_list)
predicted_label_list = nb.predict(X_predict_dtm)

Got error:

  File "/home/rajesh/www/rajesh/docuchief2/project/web/env/lib/python3.6/site-packages/sklearn/naive_bayes.py", line 720, in _count
      raise ValueError("Input X must be non-negative")
  ValueError: Input X must be non-negative

So I got TfidfVectorizer is calculated as per occurrence of words so it works, but HashingVectorizer logic is different which I can not figure out how HashingVectorizer will implement in MultinomialNB.

Can someone please help me with how I can solve this performance issue.
Can I use TfidfVectorizer for a huge training dataset if yes then how?
If not then how can I use HashingVectorizer here?


texts: High probabilities of success for wrong predictions

texts:
I'm studying the behavior of machine failures in a production scenario. For this, I generated random data to form my imbalanced training set, consisting of categorical data, which indicate whether or not there was a failure in each subperiod. The failures were generated according to a exponential distribution. I have 24 features (Period_1 to Period_24), each containing information about the historical failures for 448 subperiods. Furthermore, I have three more features consisting of Temperature, Moisture, and Pressure (generated using the Normal distribution). My intention is to predict the behavior of the failures for the next period based on these features.
I used the ROC metric and considered several strategies to deal with unbalanced data, such as oversampling, undersampling, ROSE, and ADASYN. Furthermore, I tried to use ensemble to improve performance. I tested all of the following models: gradient boosting algorithm, random forest, Classification and Regression Trees, neural networks, Bagged CART, SVM, C5.0, eXtreme Gradient Boosting, and k-Nearest Neighbors. I also tried to use regularized models but none of these strategies worked. The best result obtained was using the model "SVMRadial" considering resampling with the ROSE package. In this case, ROC = 0.7614, Sensitivity = 0.7639, and Specificity = 0.6065 for the training set and Sensitivity = 0.75, and Specificity = 0.6914 for the test set (the latter obtained through the Confusion Matrix). However, when making predictions, the trained model is resulting in high probabilities for wrong predictions. So, I would like to know if this is a problem of the training model. Also, would anyone have any idea how to improve these results?
Any help will be appreciated.
A sample of the data:

The code for the chosen model:
set.seed(123);

partition <- createDataPartition(data_failures$Period_24, p = 0.8, list = F)
trainingSet <- data_failures[partition,]
testingSet <- data_failures[-partition,]

train.control <- trainControl(method = "repeatedcv", number = 10, repeats = 3, sampling = "rose", classProbs = TRUE, summaryFunction = twoClassSummary)

model_24 <- train(Period_24 ~., data = trainingSet, method = "svmRadial", preProc = "zv", metric = "ROC", trControl = train.control)

print(model_24)

predictions <- predict(model_24, newdata = testingSet)
print(confusionMatrix(predictions, testingSet$Period_24))

I'm using R considering the caret package.


texts: I am a high-school students who is learning about data science in his free time. I have gotten a neural network to work which is able to solve xor problems. My neural network uses sigmoid as the activation function for both the hidden and output layers. It also has only one hidden layer. I am wondering about what would be the best simple problem which I could solve with my neural net. I would like a database in which there is a probability output or something similar since I've had problems converting sigmoided output to normal values. I have looked on the UCI machine learning repository but have found nothing witch has caught my eye. I would appreciate any help! :)


texts:
I was watching this video  and Sentdex mentioned he had to switch around 1024 by 3 model. What does he mean by 1024 by 3 model and what did he change around?

Edited: Youtube link with the timestamp where he talks about this topic


texts: How to compute inner product between two networks'parameters

texts:
Consider a neural network with $f(x) = w^T_2 \sigma(w^T_1 x) $ where $\sigma(.)$ is a activation function such as ReLU. $w_2 \in R^{d \times k}, w_1 \in R^{k \times o}$ are two matrices. I would like to compute the inner product between two initialization of model's parameters $\theta =(w_2, w_1)$ and $\theta'=(w'_2, w'_1)$.
Should we stack all elements of networks parameter into a single vector, i.e $\theta, \theta'$ will be a big vector with the number of entries equal to $w\times k + k \times o$. Then we just compute the inner product between two vectors. Is that a good way to compute the inner product between two model' parameters to measure how similarity they are ?


texts: How Do I Plot a Nonlinear Decision Boundary?

texts: I finished training my Sci-Kit Learn Logistic Regression model and it is performing at 100% accuracy. However, when I went to plot the decision boundary, I got a bit confused. I am not running the typical Logistic Regression model. Instead, I have transformed the elements of the input feature matrix into nonlinear 2nd degree polynomial terms before training with the help of Sci-Kit Learn's sklearn.preprocessing.PolynomialFeatures(degree = 2) function. According to the documentation, this function transforms two features $[x,y]$ to six features $[1,x,y,x^2,xy,y^2]$. With this in mind, I wrote the general equation of the decision boundary like this: $\theta _0x^0+\theta _1x^1+\theta_2y^1+\theta_3x^2+\theta_4x^1y^1+\theta_5y^2+b=0$, while obtaining all the $\theta$ parameters from model.coef_[0][0] for $\theta_0$, model.coef_[0][1] for $\theta_1$, model.coef_[0][2] for $\theta_2$ (and so on), and the $b$ parameter from model.intercept_[0]. When I plotted this on Desmos, it came up with a weird plot nothing close to a rational decision boundary. My question: does the method model.coef_ return the parameters $\theta_0$ to $\theta_5$, IN THAT ORDER? For example, if it returns [a, b, c, d, e, f], does that mean $\theta_0$ is equal to a, $\theta_1$ is equal to b, $\theta_2$ is equal to c and so on?


texts:
If you are allowed to choose the features by hand and if you use
  enough features, you can do almost anything.For binary input vectors,
  we can have a separate feature unit for each of the exponentially many
  binary vectors and so we can make any possible discrimination on
  binary input vectors.This type of table look-up won’t generalize.But
  once the hand-coded features have been determined, there are very
  strong limitations on what a perceptron can learn.

This is what Hinton explains in his Neural Networks course but I don't get the binary input example and why it is a table look-up type problem and why it won't generalize? What does he mean by hand generated features? I understand that perceptrons cannot classify non-linear data but I cannot relate this to his slide (slide 26). It would be nice if anybody explains this with proper example.


texts: How to make a classification problem into a regression problem?

texts:
I have data describing genes which each get 1 of 4 labels, I use this to train models to predict/label other unlabelled genes. I have a huge class imbalance with 10k genes in 1 label and 50-100 genes in the other 3 labels. Due to this imbalance I'm trying to change my labels into numeric values for a model to predict a score rather than a label and reduce bias.

Currently from my 4 labels (of most likely, likely, possible, and least likely to affect a disease) I convert the 4 labels into scores between 0-1: most likely: 0.9, likely: 0.7, possible: 0.4, and least likely: 0.1 (decided based on how similar the previous label definitions were in their data). I've been using scatter plots with a linear model to try to understand which model would best fit my data and reducing overfitting, but not sure if there's more I can infer from this except that the data has homoskedasticity (I think? I have a biology background so learning as I go):

I'm not sure if there is a more official way I should be approaching this or if this regression conversion is problematic in ways I don't realise? Should I be trying to develop more scores in my training data or is there something else I should be doing?

Edit for more information: The current 4 labels I have I create based on drug knowledge of the genes and the drug data I currently have for each gene, I could incorporate other biological knowledge I have to make further labels I think.  For example, currently the 'most likely' labelled genes are labelled as such because they are drug targets for my disease, 'likely' label because they are genes which interact with drugs to cause a side effect which leads to the disease, and the other 2 labels go down in relevance until there are least likely genes with no drug related or statistical evidence to cause the disease.


texts: Is there a rule of thumb when designing neural network in deep reinforcement learning?

texts: In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used in deep reinforcement learning? We can assess reinforcement learning algorithm's performance itself with rewards and so on, but how can we be sure that neural network used in reinforcement learning algorithm is good or bad ?


texts: Softmax activation predictions not summing to 1

texts:
I am a beginner with rnns, consider this sample code

from tensorflow import keras
import numpy as np

if __name__ == '__main__':
    model = keras.Sequential((
        keras.layers.SimpleRNN(5, activation="softmax", input_shape=(1, 3)),
    ))
    X = [
        [1, 2, 3],
        [4, 5, 6]
    ]
    y = [
        [1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0]
    ]
    X = np.array(X)
    X = np.reshape(X, (2, 1, 3))
    y = np.array(y)
    # print(X)
    # print(y)
    print(model.summary())
    model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.01),
                 loss="categorical_crossentropy")
    model.fit(X, y, epochs=100)
    p = model.predict(X)
    print(p)
    p = list(np.squeeze(p))
    print(p)
    print(np.sum(p,axis=1))

I am using a simple rnn with batch size=2, 3 input features and 1 timestep,as the activation is softmax the last line prints [1,1] as the sum of predictions of a softmax is 1.
But when when I change the layer from a SimpleRNN to

keras.layers.LSTM(5, activation="softmax", input_shape=
                 (1,3),recurrent_activation="softmax")

The sum of predictions is no longer 1, why is that?


texts: perceptron implementation, preocedure for post-feature vecor: bag-of-words model

texts:
I have code to generate feature vectors in the style of the bag-of-words model, you can see it on my github page.

It renders output of the form:

/data/train/politics/p_0.txt, [0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]
/data/train/science/s_0.txt, [1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0]
/data/train/atheism/a_0.txt, [0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
/data/train/sports/s_1.txt, [0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1]

I want to move on to the next step of the perceptron but I'm confused about what to do next. I guess the next thing I need to do is generate a feature vector for the 'test' data, isn't it?

For instance, in this example what is 'teachingOutput' supposed to be, training data vectors? Such as the ones I have above?

I also tried poking around in this implementation in an attempt to figure out what to do but, alas, to no avail.


texts: Perform Person Re-Identification on custom image set

texts:
I've been recently using the YOLO system to detect people on images, which turned out really well.

My next step is to try to find images of the same person across the whole set of images I've retrieved earlier. I came across the AlignedReID: Surpassing Human-Level Performance in Person Re-Identification paper, read it through and found an implementation of it that offers pre-trained weights.

However, I'm kind of lost when trying to apply it to my (unlabelled) custom dataset. To me, the input of the system is an image and its outputs are k candidate images, ranked accordingly to the system's confidences. In other words, the system will try to match an image with the other images he works with.

Since the images I retrieved are unlabelled, I'd like to avoid re-training anything. But the more I think of it, the less it seems possible to me.

Can I load the pre-trained weights, define the images I retrieved as the test set and run it? Or should I re-train the last layers to adapt it to my dataset?

In other words, how generalisable are person re-identification systems?


texts: Training Objective of language model for GPT3

texts:
On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function:

Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important.

I am not sure if I understand this correctly. In my understanding, the objective function is to maximize the log-likelihood of the token to predict given the current context, i.e., $\max L \sim \sum_{i} \log P(x_{i} | x_{<i})$. Although we aim to predict every token that appears in the training sentence, the tokens have a certain distribution based on the appearance in human litterature, and therefore we do not actually assign equal weight to every token in loss optimization.
And what should be an example for a model to get the notion of "what is important and what is not". What is the importance refer to in here? For example, does it mean that "the" is less important compared to a less common noun, or does it mean that "the current task we are interested in is more important than the scenario we are not interested in ?"
Any idea how to understand the sentence by OpenAI?


texts: fit transform with mysterious special chracters

texts:
I tried to make the bag of words

pattern = re.compile('[^A-Za-z0-9]+' '' " $#! People Whitespace")
df["bag_of_words"]= df["bag_of_words"].apply(lambda s: ''.join([i for i in str(s) if not pattern.findall(i)]))

and apply fit transform and check the feature names

feature_names = count.get_feature_names()

but it includes some (Dr.)strange characters looks like following

['',
 '_',
 '____',
 '_____',
 '______',
 '_______',
 '__________',
 '_____________',
 '__________cat',
 '__________something',
 '_________soso,
 '_________dog']

what is this ______ thing? and how to remove it?


texts: doc2vec - How does the inference step work in PV-DBOW

texts:
I am quite confused about how we generate new paragraph vectors in PV-DBOW?

If I want to use the embeddings to classify some text how would I generate a vector for a new paragraph?

In the original paper the authors wrote:

  At prediction time, one needs to perform an inference step to compute
  the paragraph vector for a new paragraph. This is also obtained by
  gradient descent.

It is unclear to me how they would compute the paragraph vector for a new paragraph using gradient descent.


texts: I was reading this paper and my question is related to it. What I am stuck at is the intuition behind using two CNNs for feature extraction. Can just RNN not be used for feature extraction as well as for capturing temporal relationship?


texts: Adaptive Resampling in Caret with Pre-specified Validation Set

texts:
I was wondering if this is the correct way to get adaptive sampling in caret working with a pre-specified validation set using index. I can get this to work using the 'cv' method in caret like so
data(iris)
library(caret)
library(dplyr)
library(gbm)
index_train = sample(nrow(iris), floor(nrow(iris) * .8))
index_list <- rep(list(index_train), 10)

iris <- select(iris, -Species)

ctrl <- trainControl(method = 'cv',
                    number = 1,
                    index = list(index_train)
)

gbm_fit <- caret::train(Sepal.Length ~ ., data = iris,
                        method = 'gbm',
                        verbose = TRUE, distribution = 'gaussian',
                        trControl = ctrl)

However, when I switch to adaptive_cv, this throws an error
> ctrl <- trainControl(method = 'adaptive_cv',
+                      number = 1,
+                      index = list(index_train)
+ )
Error in trainControl(method = "adaptive_cv", number = 1, index = index_list) :
adaptive$min should be less than 1

Playing around with adaptive_min and number, it seems like adaptive_min has to be less than number but greater than 1. This means that I have to create artificial cross-folds even though I'm using the same train and validation holdout set. I do this by just repping my index vector 10x into a list (R will throw a subscript out of bounds error if you increase number and keep your index list the same).
index_list <- rep(list(index_train), 10)

ctrl <- trainControl(method = 'adaptive_cv',
                    number = 10,
                    adaptive = list(min = 5, alpha = 0.05, method = "gls", complete = TRUE),
                    index = index_list
)

This works but feels super forced, like I'm forcing caret to run 10 cv folds even though the folds are all the same train and validate set. Would this create 10x the amount of work for the computer? I really just want to run adaptive sampling on the hyperparameters of the model using my own train and validation set.
And just as a matter of curiosity, are there any other packages that would do bayesian optimization or adaptive resampling of hyperparameters better? Thanks!


texts: How can different classification algorithms expressed as neural networks?

texts: I have heard that each of the different classification algorithms can be expressed as neural network architecture. How can the different algorithms like Logistic Regression, SVM(Support Vector Machine), ELM(Extreme Learning Machine) be expressed as a neural network? Is there a way to convert the vector equation of a classification algorithm into neural network architecture?


texts: r - rpart in text mining for classification of documents

texts:
I am using rpart to classify the documents whether it belongs to class a or class b. I want to prove or provide reasons  on what basis the document was classified correctly or wrongly. How can i do that?

I displayed the summary of the model. But it was not so convincing.

Please help


texts: Should I scale high ranging ordinal fields?

texts:
In the left column, I have an ordinal integer field.
In the right column, I have a scaled float feature.
Should I scale the ordinal field since it is getting so much bigger than the other feature?


texts: How to extract name of objects from technical description (NLP)

texts:
I have a list of technical descriptions of mechanical parts such as

Relay Compressor A4u8skk

Relay-Start-Compressor sdfvsh2

Relay-Start-Compressor-Copelan 05-569-21

Compressor-J4KS0AK-5gss-fj2

and i would like that my model output for the each sample of that list above will be "Compressor" , i have 700+ unique samples like the one above, not only Compressor but large list of items .

What is the best approach / models that i should look for that kind of task?
is topic modeling related ?


texts: Handling NA Values in the Chicago Crime Rate data set

texts:
I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields.
I feel that even though there are about  6 million rows (data from 2001 - present) that is a lot of data to drop (especially since the rows contain all other data like crime type, ward, date, location description, etc.)

Here are the columns and the number of NA's found in each column:

COMMUNITY_AREA ID CASE_NUMBER DATE BLOCK IUCR PRIMARY_TYPE DESCRIPTION
        616029  0           0    0     0    0            0           0
LOCATION_DESCRIPTION ARREST DOMESTIC BEAT DISTRICT   WARD FBI_CODE X_COORDINATE
                   0      0        0    0       47 614854        0        60921
Y_COORDINATE YEAR UPDATED_ON LATITUDE LONGITUDE LOCATION CRIME_TYPE COMMUNITY_NAME
       60921    0          0    60921     60921        0          0         616120

When I look up RPubs for this project a lot of people are either dropping all rows with NA's or not even bothering to talk about the missing data, both solutions in my opinion are Not good solutions.

Part of why I don't want to drop all those rows is because each row is a valid crime, when I drop them I am now missing crimes, this contributes to my counts, and categories. And since the data does say what type of crime was committed I can include it in these counts.

Has anyone worked with this data before? Do you have a suggestion for handling the missing data? Or can I leave it in there, is there an issue with that?

I plan on doing a time series analysis on Crime Rate, however, there is no missing data in the crime field so I do not think it will effect it.


texts: Keras load pre-trained weights. Shape mismatch

texts:
I have some trouble loading pre-trained weights with Keras.
Let's say I have a keras model model and that my weights are stored at my_weights.h5.

I try to load my weights as follow :

model.load_weights("my_weights.h5", by_name=True)

But this give me the following error :

Layer #1 (named "conv2d_1"), weight <tf.Variable 'conv2d_1/kernel:0' shape=(3, 3, 32, 64) dtype=float32> has shape (3, 3, 32, 64), but the saved weight has shape (32, 3, 3, 3).

So I tried to see what was the shape of my weights and my model structure :

for layer in model_body.layers :
    print(layer.name+" : input ("+str(layer.input_shape)+") output ("+str(layer.output_shape)+")")

print("__")
with h5py.File(weights_filepath, 'r') as f:
    for k in f.keys():
        for l in f[k].keys():
            for m in f[k][l].keys():
                print(k+ " : " + m + " : " + str(f[k][l][m].shape))

conv2d_1 : input ((None, None, None, 32)) output ((None, None, None, 64))
__
conv2d_1 : kernel:0 : (3, 3, 3, 32)

(I kept only the layer that appear in the error)

By seeing this, I don't understand why the shapes mismatch, and where the shape (3, 3, 32, 64) in the error come from). Am I missing something ?


texts: Best way to suggest answers given historical question-answer pairs

texts:
Many question-answering implementations focus on extracting information from large documents/corpora of text such as Wikipedia.
I have access to a full chat log from the customer service of a large electronics company and I'm wondering if I can use the large amount of historical question-answer pairs in order to come up with useful answer suggestions for the customer care agents. The goal is thus to provide the customer care agent with useful answer suggestions. These could either be the result of some matching- or classification endeavor over the historical answers, but could also be the result of generative methods.
The question is basically the same as this one:
Question and Answer Chatbot for Customer Support. Given that a lot has changed and evolved in NLP-land, I figured I'd ask this question again.
What I've experimented with thus far is to use state-of-the-art language models like BERT to vectorize the incoming questions and use cosine-similarity to find the most similar question and return the answer that was given to that question. This seems to work pretty well for generic questions that are answered with generic answers, but is horrible for more specific questions. Unfortunately, the generic questions make up only a small portion of the dataset.
I'm looking for suggestions to implement an approach that takes full advantage of the available historical question-answer pairs.


texts: I'm trying to add Gower distance for my KNN implementation but I can't find out the formula for Gower distance. What is the formula for Gower distance?


texts: How to think about - and sometimes impute - geographic distances

texts:
I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain attributes (proximity being one of them, and the typical age, sex, education, etc. being the others) will engage in an activity in NYC (e.g., visiting MOMA, taking a Broadway show, moving to the city altogether, enrolling in an area school, things like that).

My basic question is - missing values aside - is it correct to just take distances into account "as is" or should they somehow be divided between "driving/train distances" and "flying distances" (essentially, converting them into "the number of hours it takes someone to get to NYC by the most efficient means")?

If we take Los Angeles and Richmond, VA as examples - the distance to NYC from LA is about 10 times that of Richmond; flight times are only 4 times longer, but flight time from LA and drive time from Richmond are approximately the same. So what's the right way to think about that?

And once the right approach to distances is determined, how does one go about imputing distances for missing values?


texts: Refactoring Tensorflow 2 to Distributed Tensorflow for CNN

texts:
I have been trying to reproduced this paper that is training a CNN with Tensorflow.
Unfortunately, I only have access to limited computing resources. I have a few raspberry pi that I was thinking of networking with my machine to make the computation less burdensome. But, I am having trouble figuring out how to refactor the code to work in a distributed way. I reviewed the documentation on Tensorflow's site and some of the more simple tutorials, but I am still getting stuck. I was considering
using the ParameterServerStrategy.I was wondering if anyone else could share examples of refactoring similar models. I think I am mostly having trouble getting the input to work with this step function from Tensorflow tutorial.
The input is of images in two different directories to perform classification. I am not too certain what the step function is doing other to iterate through data. I would appreciate any advice on input.

def dataset_fn(_):

    data_dir = "~/Fire-Detection-UAV-Aerial-Image-Classification-Segmentation-UnmannedAerialVehicle-main//frames//Training//Training"

    data_dir = pathlib.Path(data_dir)
    image_count = len(list(data_dir.glob('*/*.jpg')))
    print(image_count)

    train_ds = tf.data.Dataset.list_files(str(data_dir / '*/*'), shuffle=False)
    return train_ds

@tf.function
def step_fn(iterator):

  def replica_fn(batch_data, labels):
    labels = ['fire','no_fire']
    with tf.Range as tape:
      pred = tf.keras.Model(batch_data)
      per_example_loss = keras.losses.BinaryCrossentropy(
              reduction=tf.keras.losses.Reduction.NONE)(labels, pred)
      loss = tf.nn.compute_average_loss(per_example_loss)
      gradients = tape.gradient(loss, keras.Model.trainable_variables)

    keras.optimizers.apply_gradients(zip(gradients, keras.Model.trainable_variables))

    actual_pred = tf.cast(tf.greater(pred, 0.5), tf.int64)
    accuracy.update_state(labels, actual_pred)
    return loss

  batch_data, labels = next(iterator)
  losses = strategy.run(replica_fn, args=(batch_data, labels))
  return strategy.reduce(tf.distribute.ReduceOp.SUM, losses, axis=None)

```


texts: Manually labeled data quality check methods?

texts:
In this article, the author says that manual data labeling is a "not-as-bad-as-it-sounds" method, but to err is human and in this article by Shopify, I learned that high-quality annotation is necessary, so we should assess the quality of the manually labeled data. It seems very expensive to have someone else check all the annotated data to assess the quality.
Then the question arises: how to assess the quality of the annotation?
From my experience, I learned that scholars may employ some tricks to filter out some low-quality surveys as described in this article: data quality checks. For instance, we can design a data set to be labeled(or corrected) with some amount of duplicated cases(or with some tiny changes that don't affect the labels) distributed randomly, and after the annotation, we can check if those cases are labeled consistently. Another example is that we can sample a small subset of the labeled cases and have some experts relabel them and estimate the quality of the whole dataset.
I wonder if there are some systematic methods to do such a data quality management task?


texts:
How should one structure an input data matrix (containing EEG data) for an RNN?

Normally, RNNs are presented as language models where you have a one hot vector indicating the presence of a word. So if you input was the sentence "hello how are you", you would have 4 one hot vectors (I think):

[1, 0, 0, 0]
[0, 1, 0, 0]
[0, 0, 1, 0]
[0, 0, 0, 1]

How do these individual vectors get condensed into a single data matrix?

In the case of single channel EEG (with only 1 electrode), 256 samples per second, 1 second long samples, how should this data be structured? Would it be 256 vectors? If so, what does each vector represent/contain? Or should it be 1 vector that is 256 elements long?

Furthermore, how does this extend to multi channel EEG with, say, 64 electrodes over 256 time samples?

I would prefer to use the raw EEG data, rather than trying some dimensionality reduction (calculating means, spectrograms etc)


texts: From where should I start Machine Learning?

texts: I want to know how to start from scratch for Machine Learning. Also which language is best for implementing its algorithms or developing future applications based on it. Thanks!


texts: Significant difference in testing vs cross validation accuracy

texts: What could be the possible reasons for a significant difference in cross validation and testing f1_scores? I am performing 3 fold Stratified cross validation and the testing f1_score is almost 0.15 less than cross validation score. How can I come up with a more effective cross validation strategy so that the two scores are closer?


texts: What is NLP technique to generalize manually created rules in text?

texts:
Let's say we have a free text containing key-value entities.

Example: "... patient's tumour has width 6 cm and height 5 cm"

Then an expert comes, marks it as important, thus we do have the rule for finding the same entity in new, different texts.

What is the name of NLP technique that produces generalized rule, so that we are able to recognize also synonyms/variations with the same meaning?

Example: "... patient with cancerous growth, 60 x 50 mm"

I need to to learn terminology to be able to search for papers. If you could add also some references to state-of-art approaches, would be great.


texts: I have read a lot of documentation surrounding NP chunking but what about Verb Phrases? Has there been a fixed set of rules for VP chunking?


texts: What type of optimization problem is this?

texts:
We have a deformable mirror controlled by 40 actuators that have input voltages from -1V to 1V.
Prior to hitting the mirrors, a pulse of light enters a diffraction grating and then the light is spread over the mirror by wavelength. Each actuator applies a difference phase shift to each wavelength. In the end, the light gets reconverged and read by a CCD.

Our goal is to maximize the integral of the spectrum read by the CCD with respect to the 40 variable voltages. This optimization will be done in matlab. I mostly just need to know what type of problem this is...

The program will do something like:

Measure a spectrum and integrate
Input the new spectrum to the optimization toolbox.
Output a new set of voltages to try
Go back to 1

There is likely multiple local maxima and the positions will change from day to day as our setup is always changing. Any help on where to look for more information would be much appreciated!


texts: python sample code for hyperparameter optimization using Population Based Training

texts:
I am doing an experiment on word2vec hyperparameter optimization using Grid Search, Bayesian Optimization and Population Based Training.
For this experiment, the code for Grid Search and Bayesian method has been written and working. Therefore I need help with writing the code for the Population Based Training Method. Please i need help with sample python code for hyperparameter optimization using Population Based training.

Thanks


texts: Implementation of the LSTM using Keras in R with multiple outputs

texts:
I'm implementing the LSTM based on this tutorial (https://blogs.rstudio.com/ai/posts/2017-12-20-time-series-forecasting-with-recurrent-neural-networks/), but the example consider multiple inputs with a single output. How the code will be modified if I have a single input and multiple outputs?
dir.create("~/Downloads/jena_climate", recursive = TRUE)
download.file(
    "https://s3.amazonaws.com/keras-datasets/jena_climate_2009_2016.csv.zip",
      "~/Downloads/jena_climate/jena_climate_2009_2016.csv.zip"
    )
unzip(
  "~/Downloads/jena_climate/jena_climate_2009_2016.csv.zip",
  exdir = "~/Downloads/jena_climate"
)

library(readr)
data_dir <- "~/Downloads/jena_climate"
fname <- file.path(data_dir, "jena_climate_2009_2016.csv")
data <- read_csv(fname)

data <- data.matrix(data[,-1])

train_data <- data[1:200000,]
mean <- apply(train_data, 2, mean)
std <- apply(train_data, 2, sd)
data <- scale(data, center = mean, scale = std)

generator <- function(data, lookback, delay, min_index, max_index,
                      shuffle = FALSE, batch_size = 128, step = 6) {
  if (is.null(max_index))
    max_index <- nrow(data) - delay - 1
  i <- min_index + lookback
  function() {
    if (shuffle) {
      rows <- sample(c((min_index+lookback):max_index), size = batch_size)
    } else {
      if (i + batch_size >= max_index)
        i <<- min_index + lookback
      rows <- c(i:min(i+batch_size, max_index))
      i <<- i + length(rows)
    }

    samples <- array(0, dim = c(length(rows),
                                lookback / step,
                                dim(data)[[-1]]))
    targets <- array(0, dim = c(length(rows)))

    for (j in 1:length(rows)) {
      indices <- seq(rows[[j]] - lookback, rows[[j]],
                     length.out = dim(samples)[[2]])
      samples[j,,] <- data[indices,]
      targets[[j]] <- data[rows[[j]] + delay,2]
    }

    list(samples, targets)
  }
}

lookback <- 1440
step <- 6
delay <- 144
batch_size <- 128

train_gen <- generator(
  data,
  lookback = lookback,
  delay = delay,
  min_index = 1,
  max_index = 200000,
  shuffle = TRUE,
  step = step,
  batch_size = batch_size
)

val_gen = generator(
  data,
  lookback = lookback,
  delay = delay,
  min_index = 200001,
  max_index = 300000,
  step = step,
  batch_size = batch_size
)

test_gen <- generator(
  data,
  lookback = lookback,
  delay = delay,
  min_index = 300001,
  max_index = NULL,
  step = step,
  batch_size = batch_size
)

# How many steps to draw from val_gen in order to see the entire validation set
val_steps <- (300000 - 200001 - lookback) / batch_size

# How many steps to draw from test_gen in order to see the entire test set
test_steps <- (nrow(data) - 300001 - lookback) / batch_size

library(keras)

model <- keras_model_sequential() %>%
  layer_flatten(input_shape = c(lookback / step, dim(data)[-1])) %>%
  layer_dense(units = 32, activation = "relu") %>%
  layer_dense(units = 1)

model %>% compile(
  optimizer = optimizer_rmsprop(),
  loss = "mae"
)

history <- model %>% fit_generator(
  train_gen,
  steps_per_epoch = 500,
  epochs = 20,
  validation_data = val_gen,
  validation_steps = val_steps
)
predictions <- model %>% predict_generator(
  generator = test_gen,
  steps = test_steps
)
```


texts:
I have data that I plotted with a normal distribution and a QQ plot. I was wondering especially in QQ plot it seems that 95.4% of the data is normally distributed.

My question is what does numbers meanings that above 2 sigmas in QQ plot? Should remove them or I need to transform this variable and make it more normally distributed?


texts:
I have N models, each of them are used to predict on a set of data.
I am currently combining their predictions by averaging across rows.
Need suggestions on combining their predictions. End goal is to classify the samples.


texts: Best file format for transfer of EHR data

texts:
I am working on a clinical trial where we have several sites sending us EHR data.  The sites are currently sending the data in excel files.  I have a feeling someone's opening the files because 3 of the files have 64,999 rows exactly, and excel 2007 cuts off at 65,000.

I am working in python, but I am trying to prevent the people at the local sites from opening the files in excel.

What's the best format for the files to be sent to me such that they can't be opened and cut off in excel?  csv unfortunately can also be opened in excel.


texts: Neural Net that memorizes what it sees in order?

texts:
I'm sorry for this weird question, I know ML is about generalization but I have a specific use case where I'd like to build a neural network or really just a matrix, that memorizes everything it sees in order. Can this be done?
I have not been able to find any examples of this kind of thing online, is it possible? I understand it's a little unconventional (it is technically overfitting, after all).
It doesn't have to be a neural net, but I thought that would be ideal. Maybe I just need an algorithm to evolve a Matrix. Perhaps Neural net algorithms are not the best at doing this since they tend to generalize. I thought they might be a good fit because I know that neurons are proven to be able to encode any function.
Anyway, if what I'm trying to do is still unclear let me give you an example: let's say I show the algorithm unique observations A, B, C in that order during training. After training, if I show it A I want it to produce B. If I show it B I want it to produce C and if I show it C I don't care what it produces. (A, B, C could be any kind of data, for instance, images).
Can anyone point me in the right direction? I haven't used machine learning algorithms all that much, but I am familiar with scikit-learn, maybe there's something in there? Maybe I could actually use a NN for this, with a lot of epocs of backprop?
Thanks for your advice!
PS I know you might be tempted to say something like, "why would you want this, it's essentially just a copy of your original dataset," or "just write an explicit program that maps the observations," But please, I am searching for a way to encode these relationships in a matrix-type memory structure.


texts: machine learning application in ticket pricing

texts:
I want to sell a day-pass and month-pass ticket, the price is segmented based on customers' residential community or other factor. I would like to know the pricing strategy by which I can get profit or how much loss I would get.

Can someone share some ideas or reference?


texts: Regression: How to deal with positive skewness in continuous target variable

texts:
I'm working on a regression problem. My aim is to "learn" the distribution of a continuous target $y$ as good as possible to make predictions. My model looks like:

$$y_i=\beta X_i + u_i.$$

$y$ is right skewed (positive skewness) and consists of positive and negative integer values.
$X$ is a matrix containing columns with float and integer values. There also is a large(er) number of indicators (dummy variables)

These are my results. Here I compare the distribution of the actual $y_i$ (truth, blue) and the predicted $\hat{y_i}$ (predicted, yellow):

As you can see from the figure, I tend to underestimate values in the lower range (around zero) and I overestimate values somewhere between 200 and 500.

I would like to improve my estimation especially in the range somewhere around zero and 600, where most observations are found. I wonder what my options are?

Please note that the whole range of data is relevant to me, so that I cannot simply prune the data/distribution.

So far I did no feature engineering, scaling etc. What I tried was a number of different estimation methods, such as linear regression (OLS), generalised models (GAM) with local regression and regression splines, lasso/ridge, Keras neural net, boosting with LightGBM and Catboost. All of them produced very similar results (as shown in the figure above).

Edit [2019-12-28]:

A brief follow-up to my original question: I have applied a logistic transformation to my $y$ and estimated a GAM model. Unfortunately the results are not much better than with non-transformaed data. MAE decreases slightly, but the general fit is still not too good (see figure below, the "flat" curve are actual values and the pointed curve are predicted values after re-transformation).

I prefer to stick to linear models (OLS, GAM, Lasso) since boosting or neural nets did not deliver much better results than linear models.


texts: Confusion about neural network architecture for the actor critic reinforcement learning algorithm

texts:
I am trying to understand the implementation of the actor critic reinforcement learning algorithm. According to this, there should be just one neural network with two heads for the action probabilities and the state values. It is also true in their tensorflow implementation here.

However, if I refer to the tensorflow implementations such as this and this, they use two nn and update them separately.

So, my question is what is the correct way to go about the neural network for the actor critic algorithm?


texts: How much information is produced in our Society?

texts:
I was reading an interesting article about the growth of data in our society:
https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.1740-9713.2012.00584.x

Does any of you know if there exists a more recent statistic that compares the growth of data from the 70s till now?


texts: Advantages and disadvantages of using classification tree

texts: I was working on a project and was trying to validate my decisions. I wondered why would I want to use a decision tree over more powerful algorithms like random forest or Gradient boosting machine which uses similar tree based architecture.


texts: Predicting outcome of games - how to make team 1/2 interchangable

texts:
Currently I'm writing some models to predict outcomes of team games, but I'm having a hard time featurizing the teams so that the order of teams doesn't matter.

For example, currently I duplicate all the data and reverse it to reduce the effect of ordering on the outcome, but that's not proving perfect. If I have "Ravens vs Panthers = Panthers win", I also put in "Panthers vs Ravens = Panthers wins".

When I then go to my model and ask for the output of Panthers vs Ravens and Ravens vs Panthers, I get different (but similar) results. For example the probabilities of Panther's winning may be 75% in the first ane 72% in the second.

How else might I make it 100% ambiguous as to which team is team 1 and which is team 2?


texts: One hot encoding as input to recurrent neural networks

texts:
I'm trying to predict next label in a pattern based on previous labels using recurrent neural network. In total I have 100 labels

Example of input pattern:

1) orange, apple, banana, lemon -> grape
2) apple, banana, pineapple, mango -> orange
3) lychee, orange, grapefruit, apple -> lemon

Although this is a bogus example but it explains the problem pretty well. My target variable is a member of the input sequence set.

What I want to do now is as there is no ordinal relation between my input pattern, I dont want to simply label encode the data as the model might implicitly learn from the ordinal nature of label encoding, so I want to go for one-hot encoding.

But I'm having a hard time understanding how to create input feature space for input of one hot encoding. Should I have pattern input as:[1,0,0,0..,0],[0,1,0,0..,0],[0,0,1,0..0],[0,0,0,1..,0] or should it be just one matrix with 1 in place of all the labels present in the data and 0 where they aren't present something like [1,1,1,1,0,0..,0]?


texts: Multiple keras models parallel - time efficient

texts:
I am trying to load two different keras models in parallel. I tried to use the functional API model:

input1 = Input(inputShapeOfModel1)
input2 = Input(inputShapeOfModel2)

output1 = model1(input1)
output2 = model2(input2)

parallelModel = Model([input1,input2], [output1,output2])

This works but it does not run in parallel actually. Inference time is just the sum of each model's individual inference time.

My question is should this run concurrently?
I also tried to load them in different py files with gpu memory options. Still I haven't got parallelism (inference time is x1.5 for each model)

Is there any way to get inference time of both models as close to a single's model inference time?
Is the only solution to add a second gpu?

UPDATE: in different scripts they seem to be able to run in parallel, so there must be a way to efficiently run in python/keras as well.


texts: Creating a RNN dataset and model for understanding context of a sports event

texts:
I'm working on a project which requires my code to understand what is happening at any instant in a video feed of a sports match. For example, take a football match. At a given moment, my code focuses around the area where the ball is currently being played. Now, if a player A of team X passes the ball to player B of the same team X, the code must know that the ball has been passed succesfully to a teammate. However, if a player C from the opposite team Y intercepts this pass, the code must know that too. Similarly for other significant events of a football match such as a goal being scored, foul being committed etc.

I was thinking I would need to implement a RNN model whose training dataset consists of individual frames of previously recorded matches of the same sport, and for predictions, pass about 5 seconds of video (split into frames) and get the output as the event that just occured (like ball passed succesfully, ball tackled, goal scored etc)

But I am not very clear on how this dataset should look like. Like, when converted to a csv file, what should be the columns. I am also not clear how to pass the inputs for predictions (which will ofcourse depend on the format of the dataset).

Can someone help me understand howto build a model for this? And there is some other logic which could work better than mine, or uses a model other that RNN, can you share that ?


texts: Will one hot encoding / unbalanced columns cause bias to Clustering Analysis?

texts:
I'm wondering if having too many columns about one certain feature is gonna cause bias to the clustering analysis.

For example, if my dataset has columns = ['incoming calls', 'outgoing calls', 'missing calls', 'age'], and if I run clustering algorithms such as K-means or Mixture Model, will the clustering results be biased since it splits datasets mainly based on calls?

Another example is if I have two categorical columns: color ('red','blue','green'), and shape ('circle','square'), after one hot encoding, color will expand into three columns and shape will expand into two. If I cluster on the one-hot encoded dataset, will color have a larger weight than shape in terms of splitting the data?


texts: Python - Calculate Cost profitability and benefit of the model

texts:
I've this code in Python in order to calculate the precision of my model and to print confusion matrix using Decision Trees Classifier:

coef_gini = DecisionTreeClassifier(criterion = "gini", random_state = 100, max_depth = 3, min_samples_leaf = 5)
coef_gini.fit(training_features, training_target)

y_pred = coef_gini.predict(test_features)
y_pred

for name, importance in zip(training_features.columns, coef_gini.feature_importances_):
    print(name, importance)

print ( "Train Accuracy using Decision Trees Classifier is : ", accuracy_score(training_target, coef_gini.predict(training_features)))
print ( "Test Accuracy using Decision Trees Classifier is : ", accuracy_score(test_target, y_pred))
print ( "Confusion matrix using Decision Trees Classifier is ", confusion_matrix(test_target, y_pred))

What is the cost matrix? Is this the money that company will lost for each wrong predictive target value? Anyone have an example?

Thanks!


texts: Text classification: how to assign 1 or more classes to each document?

texts: I'm very new to text mining. I am trying to build a classifier which assigns zero, one or more topics to each document of the Reuters 21578 dataset. However, I'm not sure how to make it work using standard R packages. I've looked at a lot of simple online examples using packages such as RTextTools and RWeka, etc. which creates classifier to assign one topic to each document and I'm stuck as I can't directly follow the examples to achieve what I desire. Thanks in advance!


texts: Creating trial number ascending list, for separate subjects in long format data frame, R?

texts:
I have a very long data frame (over 300 000 rows), consisting of all subjects trials of a task, in long format. So about 300 rows is successive trials for one subject, followed by the other subjects underneath. What I want to do is create a new column with trial numbers for every subject.

Such as:

subject   trial_number
    101           1
    101           2
    101           3
    102           1
    102           2
    102           3

I am thinking I should somehow make R choose a subject number from column subject, and then create an ascending list and then loop this over all the subject numbers. But I have not been able to figure out how to loop over subject numbers, while also creating an ascending list in one and the same column over these subjects?

I don't have example code for failed attempts or so, as I haven't been able to figure out how to structure this. I'm thinking some kind of combination of  subset and within? Or are there better solutions my googling skills haven't helped me find yet?


texts: word/sentence alignment for English document

texts:
I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below:

I don't want to go there. My e-mail address ok.

should be aligned with

i do n't want to go there my e mail address ok

Or the tokens from the first sentence is aligned correctly with the tokens in the 2nd sentence. I have tried some sequence alignment methods, which do not perform well. Considering there are alignment tools for machine translation, automatic alignment for English should be easier right?


texts: Problems with decision tree labeling of nodes

texts: Decision trees as we know assigns label to the node based on majority class voting. I am curious to find that what could be the problems with such labeling schemes? Does it lead to overfitting the data?


texts: I intend to do classification modelling, but my target variable has only one value

texts:
Currently I have a dataset and I am trying to predict whether someone will default on their bank loan.

The dataset is quite tricky. It covers those who have defaulted in the past, but is also tracking those with current loans. So we don’t know whether they will default in the future.

This means that the target variable (default, not default) has one unique value which is 1. There aren’t any observations that can describe the 0 class.

Assuming I can’t use external datasets, what’s the solution to this problem? How can I best understand those who have defaulted if I don’t have something to compare them to?


texts: Effect of Stop-Word Removal on Transformers for Text Classification

texts:
The domain here is essentially topic classification, so not necessarily a problem where stop-words have an impact on the analysis (as opposed to, say, sentiment analysis where structure can affect meaning).
With respect to the positional encoding mechanism in transformer language models, when using a pretrained LM is stop-word removal as a preprocessing step actively harmful if the LM was trained on a corpus where they were left in? I'm still working on fully understanding the mechanism but I feel like removing stop-words would affect which wavelength is used to construct the context between any given pair of words with stop words between them, which in turn would impact the encoding.
Or, would this not matter because the regression when trained figures it out from consistently processed input? I feel like it should matter but haven't been able to find anything on the topic.


texts: Can distribution values of a target variable be used as features in cross-validation?

texts:
I came across an SVM predictive model where the author used the probabilistic distribution value of the target variable as a feature in the feature set. For example:

The author built a model for each gesture of each player to guess which gesture would be played next. Calculating over 1000 games played the distribution may look like (20%, 10%, 70%). These numbers were then used as feature variables to predict the target variable for cross-fold validation.

Is that legitimate? That seems like cheating. I would think you would have to exclude the target variables from your test set when calculating features in order to not "cheat".


texts: Variable length unordered inputs for a deep neural network

texts:
I have a function $f(x_1, x_2) = Y$ and the goal would be a user gives me a variable list of Y's where each Y was generated from the same constant $x_2$ parameter but $x_1$ varies.

I want my program to give my best guess of $x_2$. I'd like to try and train a deep neural network to do this.

A few problems is that the order of Y that the user can give me will be random and the number of Y's will vary as well. I'd hope the more Y's they gave me the more accurate my prediction of what $x_2$ would be is.

I'm not quite sure how to setup a model where it's unordered with variable length inputs.

Example

$f(x_1, x_2) = x_1 \cdot x_2 + noise$ where $x_1$ and $x_2$ are $\in \mathbb{R}^3$

 $x_2$ is unknown and I'm giving lots of examples where $x_2$ is fixed but $x_1$ varies
 I don't know what $x_1$ is but I observe the output $f(x_1, x_2)$
 I can train the neural network with synthesized data
 I'd imagine if the user gave more observations I'd get a more accurate response


texts: Collinearity between continuous and categorical variable

texts: I have a medical dataset with features age, bmi, sex, gender, # of children, region, charges, smoker. Here smoker, gender, sex and region are categorical variables and others are numerical features. How do I check for collinearity between these in my dataset?


texts: Deep Learning model for classifying skin diseases

texts: I have planned to create a deep learning model that classifies skin diseases(around 5 to 7 diseases). Please suggest me a good deep learning model to go with. I am planning to integrate this model as a mobile application. It goes like this. The user uploads a skin disease image and the machine is going to classify it accordingly. I need a good degree of accuracy.


texts: How to identify outliers from a small list of numbers?

texts:
I want to identify outliers from a very small group of numbers. How shall I do that?

For example, for the group of numbers: -0.4, 0.4, 52.1, actually 52.1 is an outlier.

I've tried using interquartile range to identify the outliers, but it won't identify 52.1 as the outlier.


texts:
I'm dealing with an imbalanced dataset and as usual it's very easy to obtain a high accuracy, but the recall on the less frequent class is very low. I would like to improve on the false negative of the less frequent class.

Focusing for concreteness on the sklearn Random Forest, one possible strategy is to set a class_weight penalizing the errors on the less frequent class and scoring with a sklearn scoring function as ROC.

grid = GridSearchCV(pipe, cv=number_of_cross_validations, n_jobs=1, param_grid=param_grid , scoring = scorer)
classifier = grid.fit(X_train, y_train)

where pipe is pipeline containing the RandomForestClassifier and param_grid contains different class_weight to try.

My question is probably related to this question, indeed class_weight alone seems to not be enough to lower significantly the false negative. As an extreme example, if I set:

 class_weight = {0: 0.0000001, 1: 0.9999999}

(where 1 is the class with less instances, with a ratio 1:50), I would expect a final classifier predicting nearly always 1, since every time it makes an error the cost to pay is very high. But that's not what happen, the fitted classifier continue to make a significant number of false negative. Why is it so?


texts: file name pattern matching and identifying

texts:
I have a automation system which manages files sent by multiple client inside S3. It so happens client sends file on a regular basis. Let's not consider the content of the file for time being as it can be something of new data or append on old data.

Typically my files are arranged on a pattern:

i) client_a/data_type1/20180610/file_name1_{some_random_text}.csv
ii) client_a/data_type1/20180611/file_name1_{some_random_text}.csv
iii) client_a/data_type1/20180610/file_name2_{some_random_text}.csv
iv) client_a/data_type1/20180611/file_name2_{some_random_text}.csv
v) client_b/data_type2/20180610/file_name3_{some_random_text}.csv
vi) client_b/data_type2/20180611/file_name3_{some_random_text}.csv
vii) client_b/data_type1/20180610/file_name4_{some_random_text}.csv
viii) client_b/data_type1/20180611/file_name4_{some_random_text}.csv

This is a sample of the file management. Now I want to be able to be able to identify unique files. For example case i and ii are same files received on different date. But may be separated by some_random_text (usually date or something).
The primary assumptions here are :

File name length varies as per client and datatype
Random text maybe at the first or end of file name for different files but   for not same file. i.e file_name1 will always have random text after the name
File name extension will vary on different files.

So I want to build a system which takes realtime input of the files received and do an analysis of the name and path to determine if newer version of he same file is received and place it accordingly (same in the sense data may be added to the file).

I have very limited understanding of machine learning algorithms as of whole and not sure which set of algorithms this requires. I am thinking of looking at clustering algorithms. I just want to sure i start in the right direction. Suggestions will be highly appreciated.


texts: How to reshape data for LSTM training in multivariate sequence prediction

texts:
I want to build an LSTM model for customer behaviour. It's the first time for me working on a timeseries, so some concepts are not clear to me at all.

My prediction problem is multidimensional, meaning that I also want to predict many informations associated to an action for each customer.

The dataset is currently shaped as a list of 2d padded arrays of one-hot encoded features (customer actions + other informations), for example:

   customer_id               encoded_features
0   25464205      [[0,1,0],..,[1,1,1],[1,0,1],..,[1,0,1]]
1   56456574      [[0,1,1],..,[1,0,1],[1,0,1],..,[1,1,1]]

where each element in the encoded_features entries represents a specific timestep.

My idea here is to use keras input shape

(n. customers, n. timesteps, length of features encoding)

In the example above it would be (2,#timesteps,3).

I have two main questions:

Is this whole setting rigth for the prediction of next single customer action? I would like to simply give a new sequence of features for a certain customer and predict all features in the next timestep.
I am thinking about splitting the data (according to a certain ratio) into sequential training and test sets, in order to test the trained model on unseen feature vectors. In the example above it would be:

  customer_id       X_train          y_train
0   25464205       [[0,1,0],..]      [1,1,1]
1   56456574       [[0,1,1],..]      [1,0,1]

  customer_id       X_test           y_test
0   25464205       [[1,0,1],..]      [1,0,1]
1   56456574       [[1,0,1],..]      [1,1,1]

Notice that X_train and X_test will generally contain all Train/Test events, except for the last one which has to be predicted.
Is this a correct interpretation?


texts: r glm - Error in names(coef) <- xnames only for 2 columns in data

texts:
I am getting the below error when i run the R code for glm():

> fmla = 'status_id~ratings'
> logisticmodel <- glm(fmla, data = playdata, family=binomial(link="logit"))

Could you please correct me where I have gone wrong, I googled a lot on it and the only answer is generally naming convention, however, this is not case for me since I tried with changing the data by placing all the values as 2's and it worked -  so it is not a naming issue with variables.

Below is the data description:

> str(playdata)

'data.frame':   24160 obs. of  13 variables:
 $ idd         : int  57251659 63385939 51939145 64339389 33725679 47000250 62738883 33725679 53589441 36670488 ...
 $ status_id       : int  1 1 1 1 1 1 1 1 1 1 ...
 $ id              : int  22820543 22953283 22919397 22699949 22658030 22720403 22581860 22915483 22621108 22651736 ...
 $ group_id        : int  2 2 2 2 2 2 2 2 2 2 ...
 $ created_date    : POSIXct, format: "2017-03-31 12:45:10" "2017-04-04 10:50:11" "2017-04-03 16:40:04" "2017-03-28 14:40:48" ...
 $ question_id     : int  20073221 20219031 20185301 19948471 19906458 18651404 19816152 20175471 19865888 19897662 ...
 $ surftoanswertime: int  533 484 98 476 388 0 741 757 2222 381 ...
 $ skiprate        : num  0.981 0.95 0.993 0.875 0.966 ...
 $ ratings         :integer64 0 0 1 2 0 0 1 1 ...
 $ pos             :integer64 0 0 1 2 0 0 1 0 ...
 $ negratings      :integer64 0 0 0 0 0 0 0 1 ...
 $ cf              : num  NA NA 1 1 NA NA 1 0 NA 1 ...
 $ cf1             : num  -1 -1 1 1 -1 -1 1 0 -1 1 ...

Actual formula is as below -  only the ratings and negratings are throwing the above error.

fmla = 'status_id~skiprate+ratings+negratings+surftoanswertime+cf1'

Could you please help me in understanding and correcting the error.

> logisticmodel <- glm(fmla, data = playdata, family=binomial(link="logit"))
Error in names(coef) <- xnames : names() applied to a non-vector
In addition: Warning messages:
1: In glm.fit(x = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  :
  non-finite coefficients at iteration 1
2: glm.fit: algorithm did not converge

If I use more columns I am getting the below error:

Error in `*tmp*`[fit$pivot] : object of type 'closure' is not subsettable
In addition: Warning messages:
1: In glm.fit(x = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  :
  non-finite coefficients at iteration 1
2: glm.fit: algorithm did not converge