texts: Neural Network fails to train when a particular batch normalization layer is removed?

texts:
Background info:
I built a baseline CNN model for the cifar10 dataset using batch normalization and then ReLu activation after each convolution layer. There are a couple of max pooling layers in between some of the convolution -- batchnorm -- relu layers, and then two fully connected layers at the end. The second to last fully connected layer also is followed by batchnorm and ReLu activation, whereas the last has a softmax activation.

I am providing my jupyter notebooks code that links to google colab that you can just run so you can see what I'm talking about. If you try to run the code, make sure to hit the Edit tab, click notebook settings, and select TPU for hardware accelerator

Here is the code for the baseline: https://github.com/valentinocc/Keras_cifar10/blob/master/cifar10_cnn_tpu.ipynb

The problem:
I want to replace all batchnorm and ReLu blocks with the ELu activation, which is supposed to reduce bias shifting and thus make batch normalization unnecessary. However, if I remove the batchnorm after the second to last fully connected layer (in the function "dense_elu_block" in the latter two programs linked to), the neural network does not train.

Here is the code for the model with layers using ELu activations except for the fully connected layers, so the model still works: https://github.com/valentinocc/Keras_cifar10/blob/master/cifar10_ELUcnn_tpu.ipynb

Here is the code for the model that doesn't seem to train itself at all: https://github.com/valentinocc/Keras_cifar10/blob/master/cifar10_ELUcnn_tpu_BROKEN.ipynb

The only difference between the models in the above two links is that the first one uses batchnorm-ReLu instead of ELu for the second to last fully connected layer, which is in the function "dense_elu_block".

I appreciate any advice, help, and input. Thank you for your time. If I did anything wrong in my post let me know so I can make a better one next time.


texts: How does Neural Network denoise an image?

texts: I understand the mathematical formalism behind how neural networks work as a classifier or perform regression analysis. But I face difficulty to realize how they are such a great denoising instrument. Even a simple fully connected single hidden layer network can denoise appreciable amount of noise from a relatively simple enough signal (with proper training of course). Do neural networks mimic one or more mathematical operations of filter (such as convolution) and that's why they are so effective in denoising application?


texts: Where do I find the .csv file for the submission after creating it?

texts:
I am making a submission of a classificaition problem with CNN on Google Colab. So I have arrived at doing this:

subm.to_csv('submission.csv', index=False)

so in theory I should have finished. The only problem is that I don't know where to find the newly created csv file.

Can somebody please help me? Thanks in advance.


texts: Weights initialization in Neural Network

texts:
I was viewing code for custom neural network for sentiment analysis. It had 3 layers (1 hidden layer). I am more concerned with weight initialization for the layers

 self.weights_0_1 = np.zeros((self.input_nodes,self.hidden_nodes))
 self.weights_1_2 = np.random.normal(0.0, self.output_nodes**-0.5,(self.hidden_nodes,self.output_nodes))

What is the idea behind initializing zero's weight matrix. I have learned that initializing weights to zero might lead to linearity.

This might be a very vague question, I will be happy to provide any specifics you want.
https://github.com/udacity/deep-learning/blob/master/sentiment-network/Sentiment_Classification_Solutions.ipynb


texts: Evaluating the performance of a machine learned recommendation system

texts:
I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes from the set of possible employers $E = \{{e_1,...,e_m\}}$.

From this, I have trained a machine learning model. This model then takes some $r_i$ from the test set, and assigns a probability to each member of $E$. The results are then ranked, from highest probability to lowest probability.

E.g. $P(e_2|r_i)=0.56, P(e_{52}|r_i)=0.29, P(e_{29}|r_i)=0.14,...etc.$

The resume, $r_i$ belongs to some individual, so this ranking is used to inform the individual as to what companies the model believes are most likely to hire them, given the details of what their resume contains (their skills, past employers, education, personal summary). In this case, company $e_2$ is most likely, followed by $e_{52}$ and so on.

My question is, how do you evaluate the performance of this recommendation system? Where the information need of the user is to learn what companies their resume matches to the best.

My own ideas

My understanding from information retrieval is that we need to determine some measure of relevance. From this, it's possible to use some measure like mean average precision to measure performance. Determining relevance seems like the tricky part. For instance $e_2$ has a high probability, but is it actually relevant? Maybe $r_i$ is based on aeronautical engineering, but $e_2$ is a food store, which is clearly not relevant. My current idea is to take each $r_i$ in the training set belonging to the same label $e_j$, and then compute a single TF-IDF vector which is the average of the TF-IDF vectors belonging to each $r_i$ labelled as $e_j$.

E.g. (an unrealistic example) Suppose $r_2$ and $r_9$ are labelled as $e_4$. Now suppose $r_2$ has TF-IDF vector $[0.2, 0.1, 0.5, 0.2]$ and $r_9$ has TF-IDF vector $[0.22, 0.12, 0.44, 0.22]$. Then the average of these is $[0.21, 0.11, 0.47, 0.21]$. Repeating this process for all $e_j\in E$ results in $m$ of these vectors. From this, it's possible to compute the cosine similarity between some $e_i$ and $e_j$.

Returning to the first example, we can take the true label of $r_i$, and then find the cosine similarity between this label and each member of $E$. Then we set some threshold and evaluate whether $\text{cosineSim}(\text{true label}, e_j) < \text{some threshold}$. If the cosine similarity is above the threshold, then $e_j$ is relevant, otherwise, $e_j$ is not relevant.

I'm not sure if this is a sensible/valid approach (I wonder if it defeats the point of the machine learning, since I may as well just use the cosine similarity? That said, I cannot forgo the machine learning component in this project).

Maybe this is an over complication, and something like top k accuracy would be fine. I.e. is the true label in the top k suggestions?

I'm not sure, I'm interested to have some more informed perspective.


texts:
I have time-series data of a few metrics. I know which metric is the response variable and the independent ones. I need to fit a model between them. The relationship could be linear, quadratic, logarithmic, piecewise linear, multiple linear, etc. Basically, it could be anything.

Is there any technique / properties I could use to find the relationship between the metrics and fit a model?

Right now, I have written a brute-force script in R.
For example, I have response variable A which depends on X1, X2 and X2.

  A = C1*f1(X1)+C2*f2(X2)+C3*f3(X3) is my model.

My script tries all possible combinations of f1, f2 and f3.
By combinations, I mean that I initialy start with all linear, then one of them is quadratic, then cubic, then logarithmic, etc.

I am using lm().

I then choose the model which has the least AIC as my final model.

This obviously takes too long.

I definitely need an automated way of finding this model.

Could you suggest a better way to do this?


texts:
I'm using keras in R to predict financial time series. I guess that ANN converges.

But when I do predictions I can see that probabilities don't sum up to 1

I'm using relu as final activation and loss = "binary_crossentropy".
Does that mean the net doesn't converge or do I miss something else?


texts:
T=tensor([101,123,414,463][234,903,313,341]...)
train=TensorDataset(T)
train_dataloader=Dataloader(train)

Now I would like to update tensor T[0] i.e tensor
T[0]=tensor([101,123,567,463])

for this i have tried as follows:
train_dataloader.Dataset[0].index_copy_(0,tensor([2]),tensor([567])

is it possible to modify this way or not?
Any kind of reference is helpful


texts: how to make sklearn pipeline using custom model?

texts:
I want to make a sklearn pipeline using the custom Artificial Neural Network I already have. I want to make pipeline in which input goes to ANN and its output goes to the sklearn.svm.SVC model and final prediction is made.
So, how can I do this using sklearn pipeline?


texts:
I am trying to merge >2 tables together, but using dual axis only allows merging 2 (per row/ col).
I have 3 tables, each displaying the profits, lose and gains respectively of different companies, how do I merge them into a single table? (In all tables and the intended resulting merged table, the y axis would be the monetary amt, and the x axis the company names)


texts: How to get maintenance interval from maintenance outcomes?

texts:
I have a machine, which needs maintenance.

Every time the technician visits the machine, there are four possible outcomes:
a) The machine is broken, b) The machine is still running, but it's high time that maintenance is performed, c) It's now the right time for maintenance: Something should be done, but it's not too late d) The technician came too early, there's nothing to do at this moment.

The technician rates every visit to the machine, and the goal is that the rating is c) every time. I want to write a software which helps determine the perfect maintenance interval I to achieve that.

I would model it like p(a)+p(b)+p(c)+p(d) = 1; with the probability distribution of p depending on I. I'd expect that p(a) increases with I, while p(d) decreases. If I had a good approximation to p(I), I could determine the next maintenance time when p(c) has the maximal likelihood.

However, I am new to Machine Learning and don't really know how I could get a reasonable function for p(I).

How can I determine p(I) from a number of observations? Any hints where I could starting reading into the topic?


texts: I am having issues with the output dimensions of my neural network

texts:
I am trying to make an image denoiser using a neural network in keras. I have 2 sets of images, un-noised images, and the same images with noise. All the images have the dimensions of (200,200,3). I cant use one-hot encoding, as I have memory issues with it; So I am using the sparse_categorical_crossentropy loss. This is the model:

x_train, x_test, y_train, y_test = train_test_split(x,y, test_size=0.1,
random_state=1)
y_train = y_train.reshape(180, 120000)
y_test = y_test.reshape(20, 120000)

model = Sequential()

model.add(Conv2D(32, (3,3), activation='sigmoid', input_shape=(200,200,3)))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(120000, activation='relu'))
model.add(Dense(120000, activation='tanh'))
model.compile(loss='sparse_categorical_crossentropy', optimizer='SGD')

print(model.summary())
model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=2)

I get this error when I try to train it: Error when checking target: expected dense_2 to have shape (1,) but got array with shape (120000,). How can I fix this?


texts:
In class we have been presented with a Geometric classification model such that the goal is to construct a linear decision boundary
$\bf{w} \cdot \bf{x} = t$; where $\bf{w}$ is the vector from the center of the negative class to the center of the positive class (see figure below), and $\bf{x}$ is an instance to be classfied.
For the above model we have: $t = \frac{(\bf{p} - \bf{n}) \cdot (\bf{p} + \bf{n})}{2}$

Can someone please explain how we get the decision boundary, i.e. t.
Why is it the dot product of the vector $\bf{w}$ and $\bf{n} + \frac{\bf{w}}{2}$. I have been trying to understand this through linear projections, but I am stuck.


texts: Default value of mtry for random forests

texts: It is argued that the default value of mtry for random forests is square root of total number of features (for classification) and number of features divided by 3 for regression. Can someone tell me the literature where it is specifically mentioned?


texts: Support vector machine margin term, norm or norm squared?

texts:
$$ \lambda||\hat w||^2 +(1/n)\sum   max(0,1-y_i(\hat w \hat x_i -b))   $$

we know that $2/||\hat w||$ is the width of the margin.

The second term penalizes a misclassified point for how far away it is from the margin relative to the width of the margin. E.g., suppose there is a misclassified point $x_0$:
$$ 1-y_0(\hat w \hat x_0 -b)=3 $$
That means $x_0$ is $3/||\hat w||$ away from $1-y_i(\hat w \hat x_i -b)=0$ and is penalized for $3$.

The first term penalizes for the inverse of the width of the margin squared. I find it hard to reconcile with the second term - they seem to be of different scales. Is there any reason (intuitively) why $||\hat w||^2$ is used instead of just $||\hat w||$?

PS: Perhaps one reason is that  $||\hat w||^2 $ is easier computation-wise (quadratic programming)? Or perhaps norm squared assumes sample noise to be Gaussian? I am not sure. Has anyone seen the use of $||\hat w||$ instead of $||\hat w||^2$?


texts: Best way to deploy and Schedule Deep Learning Model

texts:
I am looking for best way to deploy my Deep Learning application which does the following:

Fetches data from BigQuery
Finds anomaly (based on trained model)
Throws the output in a mysql database.

I want this program to run everyday on a specific time like a cronjob.

Any lead on how can I achieve this deployment, other than using a cronjob because it will be unnecessary to pay for the time and resources when it will not be in use.

Thanks in Advance.


texts: Least Squares Regression $Ax=b$ when $A$ is fixed and $b$ is varied

texts:
The typical setting for least squares regression (or over-determined linear system) for $Ax=b$ is to solve $x$ given $A$ and $b$.
In other words, $A$ and $b$ are fixed when we solve the problem.

My question is that is there any application that $b$ is changed by users (i.e., $b$ is interpreted as a query) while $A$ is fixed?


texts:
I have recently observed in many cases that taking mean of the cost function vs sum of cost function results in very different results (specifically in thenano)?

Many times sum of cost gives nan while taking mean resolves it. Why is this the case? Mean requires sum too, then what is the possible reason for this behaviour?


texts:
I have an embedding layer that returns a tensor of type: batch x max_len x embeddings_dim (8 x 200 x 300)

I want to input this to a Conv2d layer with kernel 3x3.

I tried

conv =  Conv2d(300, 100, (3,3))
conv(embedding) # embedding.shape = (8 x 200 x 300 x 1)

I get an error:

Torch: invalid memory size -- maybe an overflow?

What should be the input of the Conv2d ?


texts: Undestanding Bayesian network with OpenMarkov

texts:
I downloaded OpenMarkov software for  probabilistic graphical models and tried it on mtcars dataset.

The mtcars.csv data looks like this:

In OpenMarkov GUI, I went to Tools > Learning and loaded mtcars.csv dataset. I then adjusted preprocessing settings to have Discretize with Equal width intervals for all variables.

I then chose Hill Climbing algorithm (default) and Automatic learning options. On learning, the result was as follows:

My question is what exactly does this figure represent? Does it represent a Bayesian network or some other type of probabilistic graphical models? Also, do arrows mean that hp affects cyl and carb; and cyl in turn affects disp and carb and so on?


texts: Dwell Time computation from Real-Time user location

texts: We have a collection of real time user locations from our application. Each record contains longitude, latitude and the time at which the geo location was captured. How can I compute the dwell time at an user's stationary place from this data?


texts:
Having n vectors of percentile ranks for a list of common users between group #1 and groups #2:n

e.g. vec1 = {0.25, 0.1, 0.8, 0.75, 0.5, 0.6}
     vec2 = {0.35, 0.2, 0.6, 0.45, 0.2, 0.9}

The percentile ranks represent activity frequency within the group for instance, opening times.
The goal is to find similarity between group #1 and groups #2..n in terms of these common users's rank.

The direction taken so far is to use dot product in order to take the magnitude into account (due to rank). The problem is that the scalar answer can take any value and so a threshold cannot be drawn.

Do I need to draw a threshold with respect to the dot product of vec #1 (group 1) with itself? or is there another way to set a threshold (maybe dynamic) ?


texts: Does the SimpleRNN in Keras have a hidden state, or does it just use the output value as the hidden state?

texts:
When using tf.keras.layers.SimpleRNN,does this SimpleRNN have a hidden state, or does it just use the output value as the hidden state. That is, does it follow the formulas $h_t = \tanh(w_h\cdot h_{t-1} + w_x\cdot  x_{t-1}+b_h)$, $y_t = w_o\cdot h_t + b_o$? Or is $h_t=y_t$?
If the input is of length 5, and I make keras.layers.SimpleRNN(1), why are there only 6 parameters according to the summary. There should be 5 for $w_x$, 1 for $b_h$, 1 for $w_h$, and possibly 2 for output, for a total to 7 or 9.


texts: How do you see the element of a csv table with many columns (>30) which the names of its columns is more than 10 character in pandas?

texts:
How do you see in pandas the element of a csv table with many columns (>25) which the names of its columns is more than 10 character?
I have 5000 rows and 32 columns and the label of some columns are more than 10 characters.
How I ca see them and work with different columns?
Excel does not work! All of the items are sloppy
Access is OK but could not detect the long labels of items!
What is your solution for it?


texts:
I am training a neural network (arbitrary architecture) and I have a label space that is not one-hot encoded, but continuous. The reason is that for the given problem, it is not possible to assign a single class only, it is more of a probability mapping. So in the end, my targets sum up to 1 again, but they are not 1-hot.

I wonder if I am misunderstanding Keras documentation, but for what I read, there is no Crossentropy implementation for this. There is categorical and sparse_categorical (which seem to do exactly the same, but only expect a different label format). My idea was to wrap every single target index into a binary crossentropy, but this feels wrong and I think there is a better solution. Can you please help me find an appropriate CE loss for my task?


texts: Importing .ipnyb file from Kaggle into local Jupyter

texts:
Total beginner question here, please let me know if it would be more appropriate somewhere else.

I just created my first iPython notebook in Kaggle and I downloaded the ipnyb file. Now I have installed Jupyter locally and want to try working on the same notebook that way.

It seems like I got Jupyter working because I am able to create a new local notebook and it looks like what I would expect:

But when I open the ipnyb file that I downloaded from Kaggle, I just see what looks like raw JSON instead of a live notebook:

I also noticed that the icons of these two notebook files look different:

Any suggestions about what I might be doing wrong and how I might get my properly import my Kaggle notebook into Jupyter locally?


texts:
I have the below sets of data per application, you can call them as software metrics. These metrics vary depending on the size of an application.

Bugs
CodeSmells
Vulnerability

The size of the application is determined by LOC (Lines of code), how can i showcase the complexity of each app relative to the lines of code if i visualize each of these parameters.

  Example

           Bugs   LOC
SweetApp   10     10000
SourApp    120    5660000
SaltyApp   55     1500

How do i visualize Bugs per app in relation to the LOC, because LOC determines their complexity a higher number in bugs doesn't necessarily be bad considering a higher number in LOC.


texts: How to find optimized x values (input features) after training in deep learning?

texts:
I did deep learning training by Keras.

I have done the training part by model.fit If I do model.predict, it only gives me y value. But I want to know x (input features) that gives the best y value, which means the optimized x values after training.

Is there any way to fetch the optimized x values (input features) in Keras or else?

I really appreciate your help.

Thank you.


texts:
When training an XGboost model some of the information printed regards "extra nodes". I can't find an explanation of these anywhere in the documentation. What exactly are extra nodes?

[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 54 extra nodes, 0 pruned nodes, max_depth=5
[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 58 extra nodes, 0 pruned nodes, max_depth=5
[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 48 extra nodes, 0 pruned nodes, max_depth=5
[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 46 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 48 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 50 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 60 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 50 extra nodes, 0 pruned nodes, max_depth=5
[14:13:11] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 46 extra nodes, 0 pruned nodes, max_depth=5


texts: How to determine the convergence of Stochastic Gradient Descent?

texts:
While coding the batch gradient descent, it is easy to code the convergence as after each iterations the cost moves towards minimum and when the change in cost tends to approach a pre-defined number, we stop the iterations and conclude our gradient descent has converged.
But in stochastic GD, the cost tend to wander off the local mimina at one point of time. Deciding a threshold of change does'nt work here as SGD does not always move to converge, rather fluctuates a lot.
While coding in python, how do I know the number of iterations when the cost tends to be at minimum?


texts: Balancing data-sets for regression problems

texts:
Unbalanced data-sets are a well described problem for classification-problems.

However, for regression similar problems can arise. An example is the data-set where target variable has a very irregular histogram.

When fitting a model on such data, the main focus will be on the peak around the value 120. However, I want to create a model which works over the entire interval from 40 to 160.

How would you typically approach these problems?
Are you aware of any literature regarding this topic?
Is balancing the data-set a good approach and how can this be implemented?


texts: The proper way to codify Na in a list in R

texts:
I am trying to impute missing timeseries present in different dimentions, row by row, on the whole date set. I showed the type of return of na.kalman() and it happens to be a tribble, I am not so keen on R, so I thought converting it to a plain vector will do it, something like array(as.data.frame(imputed_slice))

library("imputeTS")
library(readr)

slice <- read_csv("slice.csv")
little_slice <- slice[c(750: 850)]
for (row in 1:nrow(little_slice)) {
  ts <- little_slice[row,]
  if(any(is.na(ts)))
     {
       imputed_ts <- na.kalman(ts, model = "auto.arima")
       little_slice[row,] <- array(imputed_ts)
  }
}

#529, 811, 812 missings

Looking for warning, I see something like:

  imputeTS: No imputation performed for column 64 because of this Error
  in na.kalman(data[, i], model, smooth, nit, ...): Input data needs at
  least 3 non-NA data point for applying na.kalman

For every row, so I tried to force other columns to be NA

Like this:

slice$`811` <- NA
slice$`812` <- NA
slice$`822` <- NA
slice$`832` <- NA

To sum up, why na.kalman() not recognizing my NAs?


texts: What's the difference between feature importance from Random Forest and Pearson correlation coefficient

texts:
I have following business domain. I have a product with three outputs/labels. The outputs are impacted by 1000 procedures, each procedure is digitized and measured. The customer wants to know what is the most influential procedures on the outputs.

1.
From Pearson correlation coefficient we could learn how two variables' relationship, say 1 is proportional, -1 is negative proportional and 0 is no relation. So I could find the biggest value of Pearson correlation coefficient to find more influential procedures.

2.
From Random Forest algorithm, I could know the top feature importance. So I could identify also the most influential procedures.

Which one is better?


texts: I am using one file for training (e.g train.arff) and another for testing (e.g test.atff) with the 70-30 ratio in Weka. I want to ask how can I use the repeated training/testing in Weka when I have separate train and test data files and the second part of the question is what is the advantage if we use repeated and what if we dont use it? Thanks in advance


texts: How does dropout work during testing in neural network?

texts:
The below paragraph is picked from the textbook Hands-On Machine Learning with sci-kit learn & Tensorflow. I couldn't understand what the author is trying to convey. It would be really grateful if someone can provide an explanation for below paragraph with an example. I have referred these links Dropout in Deep Neural Networks, https://www.coursera.org/lecture/deep-neural-network/dropout-regularization-eM33A, that somehow aligns with the same objective. However, i am seeking good explanations. Please help me.

  Suppose p = 50%, in which case during testing a neuron will be connected to twice as many input neurons as it was (on average) during training. To compensate for this fact, we need to multiply each neuron's input connection weights by 0.5 after training. if we don't, each neuron will get a total input signal roughly twice as large as what the network was trained on, and it unlikely to perform well. More generally, we need to multiply each input connection weight by the keep probability(1-p) after training.
  Alternatively, we can divide each neuron's output by keep probability during training (these alternatives are not perfectly equivalent, but they work equally well)


texts: Does ktrain's performance the same as in using Pytorch?

texts:
I am new to natural language processing (NLP) that TensorFlow and Pytorch are too hard for me. Thanks to the technology, there's another choice to build the model: ktrain. But, when I went through the illustration, the BERT model manually tokenizes the text, adds special tokens, adds padding, adds attention mask. Finally, BERT extracts the feature from the last hidden state, ready to be customized for the downstream task.
When I look inside the ktrain's source code, it seems it extracts the feature using sklearn's DictVectorizer. Does the performance degrade using it, or the performance would be similar? Thanks.


texts: Which cloud platform to maximize my impact as a data scientist?

texts:
I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer.  By this I mean handling the following on my own:

preprocess big data at low latency
design & train deep learning models on massive data
deploy models to serve predictions at massive scale
stream/preprocess incoming data to update models in real time

Which cloud platform would you choose to do this?

GCP:  Allows me to do the above with the minimum effort (serverless model hosting, model versioning etc), however, it ties me to tensorflow (I'm an MXNet fan).  Looks like I need to pick up apache beam for distributed data preprocessing...
AWS: Maximum flexibility but seems far less clean.  Appears to be more suited to a team of 5 experts wanting to achieve the above.

What software would you choose?

I'm essentially looking to pickup the minimum amount of things to have the maximum impact.
Currently spend most of my time using python + MXNet + EC2 and am comfortable with (2)


texts: keras model predicts exact same output for every input

texts:
I am working on an image captioning project, where a simplified version of my model is the following:
inputs1 = Input(shape=(2048,), name='img_input')  # InceptionV3
x = Dropout(0.5, name='dropout1')(inputs1)
img_features = Dense(256, activation='relu', name='img_dense')(x)

inputs2 = Input(shape=(768,), name='caption_input')  # BERT
x = Dropout(0.5, name='dropout2')(inputs2)
captions_features = Dense(256, activation='relu', name='captions_dense')(x)

outputs = subtract([img_features, captions_features], name='features')

model = Model(inputs=[inputs1, inputs2], outputs=outputs)
model.compile(loss='mean_squared_error', optimizer='adam')

My data generator yields the dense image embedding and the dense caption embedding as X and a vector of zeros as y, so finally I have this:
for i in range(n_epochs):
    print(f"Epoch {i+1}")
    generator = data_generator(encoded_train_images, encoded_train_sentences, rows_per_batch)
    model.fit(generator, epochs=1, verbose=1)

The basic logic is that the dense layers will learn how to "compare" between the image representation and the caption representation to yield a vector that should be as close as possible to zeros.
Now, put aside the simplicity of the model itself. My problem is that for any image and caption I put as inputs I always get a 256-vector of zeroes.
model.predict([embedded_image, embedded_caption])

Any thoughts?


texts: Replace value of a column if the value of another column is a duplicate

texts:
I have a Pandas dataframe that contains three columns: ID, name and date. The name column is not unique and may contain duplicates. I want to change the date of the duplicated name to the earliest date. For example:
Input:
id  name    date
1   Rachel  3/10/2020
2   Dave    1/5/2020
3   Rachel  1/10/2020
4   Rachel  1/1/2020
5   Jason   4/15/2020
6   Dave    7/1/2020

Output:
id  name    date
1   Rachel  1/1/2020
2   Dave    1/5/2020
3   Rachel  1/1/2020
4   Rachel  1/1/2020
5   Jason   4/15/2020
6   Dave    1/5/2020

In the sample data above, there are three dates associated to Rachel, the earliest date is 1/1/2020, therefore, in the output, all the dates associated to Rachel are changed to 1/1/2020.


texts:
I am working on a binary classification model.  Currently, when I use scikit logistic regression, it outputs binary values like 0s and 1s. However, I understand, from online reading, that it outputs probability, and based on threshold of 0.5, converts them into two classes.

1) Does building a risk prediction model mean just stopping our project as soon as we get the probability output and not apply this threshold? Is that what is called as Risk prediction model?  If yes,  how do I do that using scikit logistic regression?

2) Does scikit logistic allow us modify the threshold?

3) Can all classification algorithms like SVM, RF, XGBOOST, etc. be used to build a risk prediction model without going for the threshold cutoff?


texts: Research in high-dimensional statistics vs. machine learning?

texts:
(I've posted this question on CV, but I feel it would also be great to hear from experts in DS community.)

As a PhD student starting to think about dissertation topics, I am particularly interested in high-dimensional statistical learning. I wish to find some research review/survey/papers (or webpages, blogs, whatever...) about the state-of-the-art research in this research area, but there seems limited resources I can obtain. My first question is then,

Could you describe some current interesting research topics in
high-dimensional statistics? If you can list any relevant resources
(papers, webpages, etc.), that would be really helpful.

In addition, I've noticed that high-dimensional statistical learning is closed related with machine learning research. For example, the idea of penalized regularization in high-dimensional statistics was used in machine learning domain, like support vector machine, boosting tree, (sparse) additive models, etc. My question is,

what are good research papers about the interplay of
high-dimensional statistics and machine learning?

Last, since high-dimensional statistics was really motivated by genetic research (like gene-expression analysis, or genome-wide association studies), most of applications in high-dimensional research are devoted in that area.

Are there any successful applications of high-dimensional statistics
in areas other than genetics, particularly say, image/text mining,
recommendation, etc, areas where machine learning techniques have
long been used?
A new question to machine learning researchers/practitioners: I might be wrong, but as I understand, most machine learning algorithms are designed for low-dimensional problems (or at least the number of features is smaller that number of observations). Are there any successful applications of machine learning techniques for modeling high-dimensional data?

Any resources/comments are highly appreciated. Thanks.


texts:
I work in a professional services company and would like to get some analytics on how discipline the fee earners post their times to the system may have an impact to the revenue.

One area that I am thinking of is to see if there is a correlation between how late the time entry is posted after the work was done and whether that time entry is being written off (i.e. does not make it to the bill to the client).

Is this a reasonable thing to analyse?
If so, what technique should I be looking at to analyse this, e.g. correlation, regression, clustering, etc?

If there are other things that I can analyse as well, please let me know.

I am quite new to this data analytics/mining/science so please guide me. I have some understanding of R language which will be my primary language in analysing the data (and SQL to get the data).

Thank you.


texts: Neural Network input layer with many zeros

texts:
I want to design a neural network, an LSTM actually, which can analyze the function call log of an application. My goal is to get my system able to understand if any given functions call sequence is malicious or not. The sequence should also include the arguments for each function call.

My design involves an input layer where neurons groups concern a single call function with its arguments.

This design also implies that if I ever get this network trained, my input will always be a sequence of zeros, a few numbers different from zero, and other zeros.

Can this neural network design ever work?
Is there any best practice I'm not aware of that states I should avoid a design like that?

Updates

My intention is to define the input as a sequence of numbers.

Function arguments which are not numbers, such strings, are converted into numbers by some heuristics.

Assume the following as a possible input of the NN:

{[0,0,0,0],[0,0],[0,0,0,0],[1.2, 3.45, 5.33, 9.10],[0,0,0,0,0]}

Brackets are there just for convenience because I plan to make my input as a sequence of numbers without any structure.

The following is the representation of this scenario:

you have a space of 5 functions with the following signatures:
(for convenience I made all arguments float)

f1(float a,b,c,d)
f2(float a,b)
f3(float a,b,c,d)
f4(float a,b,c,d)
f5(float a,b,c,d,e)

and the input I included above is the case where system is evaluating a the safety of a call

f4(1.2, 3.45, 5.33, 9.10)


texts: implement Sliding Window to detect text in an image

texts: I am working on a text detection in the images. How can I implement a sliding window in a python to detect text in an image?


texts: Computing confidence interval for average from individual predictions

texts:
In the attached image, I am plotting actual vs predicted along with confidence intervals for each of the predicted points. The black star represents the average of all the predicted points. What formula can I use to combine the confidence intervals of all the predicted data points to get a confidence interval for the average? Also, please assume that the average could be a weighted average.

I am using python, numpy.


texts: Input data of variable length - two scenarios

texts:
I'm trying to figure out how I could train a neural network with inputs that have variable length. This issue comes up in the following 2 scenarios I'm trying to solve.

Scenario 1:
I have a long list of running distances for various runners which looks something like has 3 columns: runner, date, distance.
Obviously some runners have a lot of entries and others don't. I'm trying to make predictions on the number of miles a given runner will run next. So I'm guessing i need to transform my data to have one line per runner, which gives me variable length features. How can I deal with this in a ML application?

Scenario 2:
I'd like to take various strings ("teststring", "P@ssword", "NotAPassword123", etc...) and classify it as a password or not. I guess i'm trying to figure out how to a) convert strings to numbers to train on and b) how to deal with the fact that they have variable length.

Thanks for reading this far...


texts: A few questions to understand a random forest blog

texts:
I'm trying to understand a nice blog on the trade-off between sensitivity versus specificity with the random forest and logistic regression models. I have a few questions:

1) The blog used a 10 fold cross-validation in the ranger package in R (see the model mod_rf) and set the metric as ROC. So, is the final output (confusion matrix) we get there is for the one with the best ROC (AUC value) among the 10 validation sets?

2) When I try to see the variable importance by varImp(mod_rf), it says the importance values are not available. Why is that? How can I get it?

3) The caret package in R allows upsampling to adjust for an imbalance in the data. They tried the logistic regression (see model sim_glm) to do upsampling and specified repeats = 2 to repeat the 10-fold cross-validation 2 times. How does it work? I'm not clear. Does it upsample females to create a 50-50 ratio of males and females "before" each fold of cross-validation? How would the process work for upsampling, repeats = 2 and 10-fold cross-validation in the case of a random forest?

4) If the AUC (from ROC curve) in my training data is about 10 percentage points less than the AUC from the test data, how should I explain that (this happened to my data)? I thought the training data would always show higher AUC than the test data because we used training data to build our model.

I appreciate your responses.


texts: Performance Evaluation Metrics used in Training, Validation and Testing

texts:
Which specific performance evaluation metrics are used in training, validation and testing and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide variety of metrics? I don't think performance is evaluated in training, but not 100% sure.

Specifically I am actually after deciding when to use (i.e. in training, validation or testing) correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)

Sorry about this being broad - I have actually been asked to define it generally (i.e. not for a specific dataset). But datasets I have been using have all numeric continuous values with supervised learning situations.

Generally I am using performance evaluation for environmental data where I am using ANN to model. I have continuous features and am predicting a continuous variable.


texts: How to find average lag time with variance & confidence of two time series

texts:
I have two variables as time series, one a consequent of the other, I would like to find the average time delay it takes the dependent variable to act on the independent variable. Additionally, I would like to find the range of variance that is associated with the lag time and its respective confidence level. I am unsure how to go about this in a statistically valid way, but I am using Python.

Currently I have used np.diff(np.sign(np.diff(df))) to isolate the relative max & mins of the time series to then try to find the time gap between the subsequent pairs of mins & max but that doesn't seem too valid to me -- thoughts? The out put of the mins & maxes return an array like [0, -2,  0,  2,  0,  0, -2] where -2 is the relative min & 2 is relative max.

Methodological pointers would be greatly appreciated.

Thank you for your time & stay safe!

All the best,
RS


texts: how to modify the 2 dimantion array in to 4 dimension to feed into CNN

texts: i am new to python I have an array train_x(887,20) how to convert it in to 4 dimension to feed into CNN


texts: What is the difference between multi-layer perceptron and generalized feed forward neural network?

texts:
I'm reading this paper:An artificial neural network model for rainfall forecasting in Bangkok, Thailand. The author created 6 models, 2 of which have the following architecture:

model B: Simple multilayer perceptron with Sigmoid activation function and 4 layers in which the number of nodes are: 5-10-10-1, respectively.

model C: Generalized feedforward with Sigmoid activation function and 4 layers in which the number of nodes are: 5-10-10-1, respectively.

In the Results and discussion section of the paper, the author concludes that :

Model C enhanced the performance compared to Model A and B. This suggests that the generalized feedforward network performed better than the simple multilayer perceptron network in this study

Is there a difference between these 2 architectures?


texts: how to prepare data for cross validation in mnist dataset?

texts:
How to use k-fold cross validation for MNIST dataset?
I read article documentation on sci-kit learn ,in that example they used the whole iris dataset for cross validation.

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores

for example while importing mnist dataset in keras

from keras.datasets import mnist
(Xtrain,Ytrain),(Xtest,Ytest)=mnist_load()

in this dataset is already divided in test and train , so to apply cross validation on the entire dataset do we need to make Xtrain and Xtest as one entity to exploit the whole data.


texts: Data\Feature Leakage - feature too close to target?

texts:
In General:
The target itself built from very correlated features, because there is no ground truth - only rule based one.
I have a problem in the following method:
Output: binary. built from multiple binary scenarios.
Input: multiple features + features very close to the scenario.
For Example:
Output: predict whether the customer is money laundering. The target is not a ground truth (the future trial decision is not available) -  scenario for example: Has the number of returned checks >= 10?
Input: multiple features + feature -  how many checks returned?
I think there is some flaw here, maybe it's not data leakage (because input is in t(n-1) and target in t(n)).
'The flaw' - I'll recognize the target very close to the target itself (t(n-k); k is small).  If I will not using that feature - I'll 'close my eyes to reality'; even if the model I use is very good.

Does my intuition make sense? What will the downside of using those features?
Thanks.


texts: Replace entire columns in pandas dataframe

texts:
I would like to replace entire columns of a pandas dataframe with other columns, for example:

and I would like to replace the columns A and B. What I did is the following:

df['A']=dataFrame['opcodes'].values
df['B']=dataFrame['opt'].values

or also

df['A']=dataFrame['opcodes']
df['B']=dataFrame['opt']

but it does not work. In particular I get the following error:

KeyError: 'opcodes'

in both cases. Can anyone help me? Thank's in advance.

[EDIT] The original dataframe is the following:

on which I did the following modification:

dataFrame['opcodes'] = dataFrame['instructions'].apply(lambda x:[i.split()  [0] for i in x])

now I would like to define an other dataframe on which only the 'opt' column and a column with the values 'opcodes' appear.


texts:
Frequency encoding is a widely used technique in Kaggle competitions, and many times proves to be a very reasonable way of dealing with categorical features with high cardinality. I really don't understand why it works.

Does it work in very specific cases where frequencies are correlated with the target or is it more general? What's the rationale behind it?


texts: Can I have a neural network output values >100?

texts:
All the samples and articles i have seen are all having outputs of 1 or less. Is there a hidden reason why no one is using NN to produce higher value integers?? My situation is that I want NN to predict the exact outcome value rather than a -1

If I have a NN with say 4 inputs and 1 output with
Round 1 Input: 51,22,35,43 Output: 847
Round 2 Input: 25,31,46,29 Output: 761
I want
Round n Input: 51,33,19,27 Output: ??? (expected value)

I have read that GA is better in training NN but few examples have been provided to guide beginners like me. Am yet to see a detailed article on how to get this done. Using same example above, if i get the (near) best weights from first evolution using Round 1 error value, do i add the weight as parents in the next generation for Round 2?? How do i ensure that the best chromosome from previous rounds don't die after that generation.

Links to existing articles and detailed explanation will be very much appreciated. Thanks.


texts: Predicting probability of occurrence using Historical Time Series Data

texts:
If you are given 1000 two digits numbers in a row.
You have to find 5 most probable numbers that would appear at 1001th place by using historical pattern.
(If we can find probability of occurrence also, it would be really great)


texts: Split npz Dataset into Train/Test using Sklearn

texts:
I have a dataset of faces stored in an NPZ file that I would like to train it on Siamese Network. To do that, the dataset must be split into train / test using Sklearn. However, when I run the code to do the split I face this error message:
ValueError: Found input variables with inconsistent numbers of samples: [2, 199139]

How can I solve this issue given that my dataset consist of 199139 labeled faces, so I have a face and a label that needs to be split to be ready for training. Initially this is the code I use for split:
from sklearn.model_selection import train_test_split

# Split into training and testing data
(train_images, train_labels), (test_images, test_labels) = train_test_split(data, labels2, test_size=0.20, random_state=42)

Thank you in advance !!


texts: Understanding the training of R-CNN architecture, why is SVM necessary?

texts:
In the R-CNN paper, when referring to the training of CNN extraction module, the authors say:
"Aside from replacing the CNN’s ImageNetspeciﬁc 1000-way classiﬁcation layer with a randomly initialized (N + 1)-way classiﬁcation layer (where N is the number of object classes, plus 1 for background), the CNN architecture is unchanged."
To me, this implies that the CNN is capable of scoring the images resulting of region proposals. This is reinforced by my understanding that the CNN is trained separately (without using the SVM). If this is correct, and the CNN handles scoring, how can the SVM even be used?
In my mind, this means that either the classification part of the CNN is removed, or my understanding is all wrong, and we iteratively train the CNN feature extraction module and SVM classification module…
Edit: The ConvNet uses log-loss to fine-tune feature extraction on object proposals. Am I correct in the understaning that the last layer of the ConvNet is flattened and densely connected to a a softmax output layer, which is then stripped after fine-tuning is completed?


texts: Is this an implementation of Reinforcement learning?

texts:
I've written code in Python to replicate the results of a brief and simple paper about reinforcement learning.

A brief description of the problem: we have a generated time series of returns (that becomes the succession of states), a set of discrete actions (1,0,-1), and a linear approximation of the action-value function. So there is an offline learning algorithm that computes the action maximizing the action-value function approximator then computes the reward and receive the new state. Then it updates the parameters based on TD(0).

My question is here: is this enough to classify this as RL?

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import random
import math
from math import atan
from numpy import array
import statistics

#preliminary functions: price to returns and an approximation of the gradient
def PtoR(x) :
    x_1=x[1:]
    x_0=x[:(len(x)-1)]
    xx=(x_1-x_0)/x_0
    return(xx)

def der(f, x, der_index=[]):
    # der_index: variable w.r.t. get gradient
    epsilon = 2.34E-11
    grads = []
    for idx in der_index:
       x_ = x.copy()
       x_[idx]+=epsilon

       grads.append((f(x_) - f(x))/epsilon)

    return grads

#generation of datas (time series)
beta,z=np.random.random(2).reshape(-1,1)

for i in range(1, 5001):
    a, b=np.random.randn(2)
    beta=np.append(beta,0.9*(beta[i-1])+b)
    z=np.append(z, z[i-1]+beta[i-1]+3*a)

data=np.exp(z/(max(z)-min(z)))
plt.plot(data)
ret=PtoR(data)

#possible actions
a=(1,0,-1)

#Action-value function approximator
def qfun(par):
   return(par[0]+atan(par[7])*par[1]+atan(par[8])*par[2]+atan(par[9])*par[3]+atan(par[10])*par[4]+atan(par[11])*par[5]+atan(par[12])*par[6])

#Inizialization of parameters
pars=np.random.random(7)
gamma=0.7
alpha=0.08

azione=(ret[0:5]>0)*1-1*(ret[0:5]<=0) #vector of initial actions arbitrary calculated, in order to star the process
returns=np.cumsum(azione*ret[0:5])   #vector of returns

i=5
for i in range(5, len(ret)):
    #State at t-1
    st=np.array((ret[i-5],ret[i-4],ret[i-3],ret[i-2], ret[i-1]))

    #decide what's the best action according to the qfun
    if(qfun(np.append(pars,np.append(1,st)))>qfun(np.append(pars,np.append(-1,st)))):
        if(qfun(np.append(pars,np.append(1,st)))>qfun(np.append(pars,np.append(0,st)))):
            az=array(1)
        else: az=array(0)
    elif(qfun(np.append(pars,np.append(-1,st)))>qfun(np.append(pars,np.append(0,st)))):
            az=array(-1)
    else: az=array(0)

    #epsilon-greedy exploration/exploitation
    azione=np.append(azione,np.random.choice(np.append(az,a),1,p=(0.94,0.02,0.02,0.02)))

    #receive state and reward at t
    st1=np.array((ret[i-4],ret[i-3],ret[i-2],ret[i-1], ret[i]))
    reward=np.mean(azione[(i-4):(i+1)]*st1)/statistics.stdev(azione[(i-4):(i+1)]*st1)

    #verify what would have been the best action
    if(qfun(np.append(pars,np.append(1,st1)))>qfun(np.append(pars,np.append(-1,st1)))):
        if(qfun(np.append(pars,np.append(1,st1)))>qfun(np.append(pars,np.append(0,st1)))):
            azione_g=array(1)
        else: azione_g=array(0)
    elif(qfun(np.append(pars,np.append(-1,st1)))>qfun(np.append(pars,np.append(0,st1)))):
            azione_g=array(-1)
    else: azione_g=array(0)

    #update the parameters (not sure if the cicle here is useful or correct)
    for u in range(0,10):
     delta=reward+gamma*qfun(np.append(pars,np.append(azione_g,st1)))-qfun(np.append(pars,np.append(azione[i],st1)))
     gradiente=der(qfun, np.append(pars,np.append(azione[i],st)), der_index=range(0,7))

     for t in range(0,7):
           pars[t]=pars[t]+alpha*delta*gradiente[t]

    returns=np.append(returns,returns[i-1]+azione[i]*ret[i])


texts: Predict the date an item will be sold using machine learning

texts:
I would like to predict the date a item will be sold using features such as:

      Product ID          Price        Days_since_first_post       Last_repost     Type       First_Post_Date         Sold_Date

How would machine learning principles be used in such a way to do this task? If so, what is the proper means by which to design such an algorithm? What would be the different steps which would be involved.

More generally, what are the logical steps that should be taken when tackling a machine learning problem.

Thank you


texts: Can data analytics be a basis for artificial intelligence?

texts:
I'm very passionate about how computers can be made able to think intelligently and independently (in our favour, of course!). I'm currently studying Bachelors science of information technology at UTS (University of Technology:Sydney). I have two months before I start my second year, and have not yet been able to decide on which major should I select that can lead myself towards dedicated study of Artificial Intelligence (which I love with my life).

I have the following majors available:

Internetworking and Applications
Data Analytics
(there are other two as well, but business oriented).

Here is the link to my subjects. I believe that being able to play with data is a sign of intelligence (I may be wrong too!). Will one of these subjects form me a good foundation for my further study in A.I.? Or should I jump into Engineering? Or Pure Science?


texts: POC - Get an idea to create a Predictive Model

texts:
I'm trying to look for an idea to create a predictive model having the following data:

Customer_ID - Integer
Catalog_ID - Integer
Country_Code - Integet
Year - Integet
Month - Integer
Day - Integer
Quantity_Purchased - Integer
Product_Purchased - Double

I'm trying to look for a use case that creates a predictive model than can give me the ability to propose a product for a customer the next time that he comes to my website.

Is this a collaborative filtering use case? If yes, I only can use the last two fields of my dataset, right?

Thanks!


texts:
In theory we would always like to predict qualitative responses using
  the Bayes classifier. But for real data, we do not know the
  conditional distribution of Y given X, and so computing the Bayes
  classifier is impossible. Therefore, the Bayes classifier serves as an
  unattainable gold standard against which to compare other methods.

In a real life situation why we are not able to calculate the Bayes Classifier ?
Why can't we calculate the conditional distribution of Y given X ?
Can some one explanation with an example?

Any help is appreciated.


texts: I'm studying theoretical machine learning at university, and I would like to know how to solve this VCdim inequality ? I managed to understand the VCdim exercises but this seems too complex for my level of understanding.Can anyone give me a hand of help ?


texts: predict an array like [1,2,3] increasing

texts:
I want to generate a simple model and classify it with decision tree.

The idea is if numbers in an array are increasing then what I need is that.

eg.

[1,2,3]
[2,4.2,5.6]

these paterns like increasing will be predicted as true

and other then those will be predicted as false

How can I achieve this with decision trees?


texts: ML model to predict values from text (a lot of training training data)

texts:
I have around 1M entries of the type:
id | big5_openness | big5_conscientiousness | big5_extraversion | big5_agreeableness | big5_neuroticism | input_text

Where id is just the identifier of some user (it is not important), the big5 values are numbers between 0 and 100 (showing how much of each of them a person is), and the input_text is the text that is used to predict this values.
These are actual predictions, but I want to develop an algorithm that uses these predictions as training data to predict these values for an arbitrary text.
Here is some information on how the original predictions were made by the software that created the data:

(https://cloud.ibm.com/docs/personality-insights?topic=personality-insights-science)
Does anyone know what kind of ML algorithm I could use to train on this type of data (where the input is text, and the predictions are 5 discrete classes? For me, this immediately looks like a problem that can be solved by neural network (encode the texts somehow).
Thank you in advance, have a great day!


texts:
I'm wondering if there is a web framework well suited for placing recommendations on content.

In most cases, a data scientist goes through after the fact and builds (or uses) a completely different tool to create recommendations. This involves analyzing traffic logs, a history of shopping cart data, ratings, and so forth. It usually comes from multiples sources (the web server, the application's database, Google Analytics, etc) and then has to be cleaned up and processed, THEN delivered back to the application in way it understands.

Is there a web framework on the market which handles collecting this data up front, as to minimize the retrospective data wrangling?


texts: Looking for help calculating a probability formula

texts: How do I put this into a calculator or a excel spreadsheet formula.  I have never done this math before, but I want to figure it out.


texts: Train and predict on a varying number of inputs - time based events

texts:
I have the problem where I am trying to build a model which takes in n events for a single user as input for prediction, the problem is that the number of events is not set, so for example:

A user may have performed a single event:
time - eventType - eventData

Yet another user for whom I need a prediction may have performed many of these time based events.

What architectures / models can I explore?


texts: How gradients are learned for pooling layers in Convolution Neural Network?

texts:
Assuming we could compute a layerwise Hessian of the error function when training a
neural network, the error sub-surface of pooling layers will be flat.??

Is that correct?

There is no weights to be learnt for pooling layer but for eg. max pool can have different values at different iterations? Will that effect the error surface?


texts: How can I use COCOAPI/PyCOCOTools to evaluate the performance of Tensorflow Lite models

texts:
I have used the Tensorflow Object Detection API to train models on a custom dataset. The tensorflow object detection API also allows evaluating the trained models on a test set and gives results in the COCO eval format.  An example output from the evaluation can be seen here:
Evaluation output from Tensorflow Object Detection API reporting the MSCOCO metrics
In this image, there are results such as mAP @ 0.5 IoU and mAP @ 0.75 IoU. It also reports the AR (Average Recall). It also reports an overall mAP.
After training my model, I have converted the models to Tensorflow Lite and I have applied post-training quantization on the model. Now I want to calculate similar metrics for my tflite model on my test set.
Can someone guide me on how I can go about doing this? I am trying to understand how I can use the COCOAPI to do this but so far I am not sure how.
A related question is, can someone please tell where I can find documentation for the COCOAPI? I have searched everywhere and I cannot seem to find it.
Thanks


texts: What is the level of measurement / name of the scale of list-features?

texts:
If you look at publications, you can have a dataset

title of publication
list of authors
number of pages
year of publication

The Level of measurement of "number of pages" is  interval scale, the year of publication is interval scale as well, the title is nominal. But what is the list of authors? Simply saying that it is nominal seems not to capture a major part of this attribute.


texts: Using tensorflow to test a variable amount of correct labels

texts:
I'm using a neural network to analyze item choices made by players in a computer game. In the game players can choose between 0 and 7 items. Right now I'm struggling with how I can evaluate my data.
Tensorflow provides a nice method for getting the k highest values.
tf.nn.top_k(input, k=1, sorted=True, name=None)

The input here would be the prediction by my neural network. The output I get by applying top_k to the prediction would then be compared to applying top_k to the correct_output (the one the neural net should have had) and by doing this multiple times and averaging I get the accuracy that I want to have. The problem I'm running into is that k should depend on the amount of 1's in the correct_output. I am lost as to how I can achieve this.
edit: correct output (if I understand how this works correctly) should already be a tensor. It is loaded from a .pickle file and at the beginning of the code it is prepared as follows:
correct_output = tf.placeholder('float')
As to what it looks like: it is simply a list of given length of 1's and 0's


texts: How to get out of local minimums on stochastic gradient descent?

texts:
I'm not programming a neural network but I'm looking at it from a non-hands-on, theoretical point of view and I'm currently wondering how to escape a local minimum and how to get to a global minimum.

If you start at a point, for instance: (red)

When you compute the gradient of the error function and step in the direction of greatest descent, you'd end up in that immediate local minimum. AFAIK, you'd get stuck there. How do neural network trainers go about this? Do they start from a new random configuration of weights at each batch and see if the cost is smaller, or is there some way of immediately getting to the global minimum?

I've heard of a method resetting the learning rate to 'pop' out of local minimums but I'm not sure how that works when the gradient is 0. I've also heard that stochastic gradient descent is more reliable than gradient descent at finding global minimums, but I don't know how using the training data in batches rather than all at once allows it to steer around local minimum in the example, which is clearly steeper than the path to the global minimum behind it.


texts: How to make LightGBM to suppress output?

texts:
I'm trying for a while to figure out how to "shut up" LightGBM. Especially, I want to suppress the output of LightGBM during training (the feedback on the boosting steps).

My model:

params = {
            'objective': 'regression',
            'learning_rate' :0.9,
            'max_depth' : 1,
            'metric': 'mean_squared_error',
            'seed': 7,
            'boosting_type' : 'gbdt'
        }

gbm = lgb.train(params,
                lgb_train,
                num_boost_round=100000,
                valid_sets=lgb_eval,
                early_stopping_rounds=100)

I tried to add verbose=0 as suggested in the docs. But this does not work.
https://github.com/microsoft/LightGBM/blob/master/docs/Parameters.rst

Does anyone know how to suppress LightGBM output during training?


texts: Why imbalanced data-set will bias the prediction model towards the more common class?

texts: As we know, an imbalanced data-set has a disadvantage of training a model for deep learning. However, I don't know how to explain it with mathematics?


texts: Does batch normalization mean that sigmoids work better than ReLUs?

texts:
Batch normalization and ReLUs are both solutions to the vanishing gradient problem.  If we're using batch normalization, should we then use sigmoids?  Or are there features of ReLUs that make them worthwhile even when using batchnorm?

I suppose that the normalization done in batchnorm will send zero activations negative.  Does that mean that batchnorm solves the "dead ReLU" problem?

But the continuous nature of tanh and logistic remain appealing.  If I'm using batchnorm, will tanh work better than ReLU?

I'm sure that the answer depends.  So, what has worked in your experience, and what are the salient features of your application?


texts: How to deduct ARIMA's order's parameters?

texts:
According to their official site, the ARIMA function has 3 parameters for the order parameters. It says:

"The (p,d,q) order of the model for the number of AR parameters, differences, and MA parameters to use."

I have this code that tries all the possible combinations for the order's parameters:
import warnings
from pandas import read_csv
from pandas import datetime
from statsmodels.tsa.arima_model import ARIMA
from sklearn.metrics import mean_squared_error

# evaluate an ARIMA model for a given order (p,d,q)
def evaluate_arima_model(X, arima_order):
    # prepare training dataset
    train_size = int(len(X) * 0.90)
    train, test = X[0:train_size], X[train_size:]
    history = [x for x in train]
    # make predictions
    predictions = list()
    for t in range(len(test)):
        model = ARIMA(history, order=arima_order)
        model_fit = model.fit(disp=0)
        yhat = model_fit.forecast()[0]
        predictions.append(yhat)
        history.append(test[t])
    # calculate out of sample error
    error = mean_squared_error(test, predictions)
    return error

# evaluate combinations of p, d and q values for an ARIMA model
def evaluate_models(dataset, p_values, d_values, q_values):
    dataset = dataset.astype('float32')
    best_score, best_cfg = float("inf"), None
    for p in p_values:
        for d in d_values:
            for q in q_values:
                order = (p,d,q)
                try:
                    mse = evaluate_arima_model(dataset, order)
                    if mse < best_score:
                        best_score, best_cfg = mse, order
                    print('ARIMA%s MSE=%.3f' % (order,mse))
                except:
                    continue
    print('Best ARIMA%s MSE=%.3f' % (best_cfg, best_score))

set= ps.read_csv(path+ "set.csv")
X = set.fpio_1.values
# evaluate parameters
p_values = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
d_values = range(0, 3)
q_values = range(0, 3)
warnings.filterwarnings("ignore")
evaluate_models(X, p_values, d_values, q_values)

But I was wondering whether it is possible to know how to deduct the best possibility before trying all of them ? How do you exactly think them ?


texts: One LSTM for two currencies or two LSTM one for each currency?

texts:
Suppose I am building an LSTM model for currency forecasting. Assume that I am working on two rates: USD vs GBP and USD vs EUR.
Should I build one LSTM model with input size of two features (GBP and EUR)? Or should I build two LSTM models, one for GBP prediction and the other for EUR?
And how to decide?


texts: Wikipedia corpus for NLP - Cleaned sentences

texts:
I can see many wikipedia dumps out there.
I am looking for a wikipedia-made corpus, in which every line is one sentence, without any wikipedia meta tags.


texts: np.c_ converts data type to object. Can I prevent that?

texts:
Was trying my hand at the Titanic dataset, when I wanted to One Hot Encode a categorical feature, after which I wanted to combine the original data with the new one hot vectors. The datatypes are as such:

data : Pandas Dataframe

Titles_ohe : Numpy sparse matrix (float64)

I tried to merge them into a dataframe using np.c_ :

columns = (list(data))+list(Titles.values)

data = pd.DataFrame(np.c_[data.values, Titles_ohe.toarray()], columns=columns)

However on checking the data type of the resulting Dataframe, all the attributes have been changed to the object datatype. Is there any way I can prevent this while using np.c_, or is there an alternative solution? Thanks in advance for any help!


texts: Is there any standard or normal range for the amount of LSTM loss function?

texts:
I am working on a LSTM network that I get loss amounts around 4.7 e-4 . It seems adding more layers and increasing epochs don't help to decreasing it. I also using a Dropout = 0.2 for each of my layers and implemented all the jobs with Keras library.

I like to know about this loss amount? is this large or is OK> Are there any rule of thumb for loss?

And why I can't decrease my loss amount? Is there any problem here?


texts: Plotting relationship between 2 data points where one data point is a boolean

texts:
I am working with the titanic survivors data set.

I have the data as a DataFrame and I can create 1D visualizations such as histograms, and also see the correlations by calling data.corr().

I would like to create a scatter plot to represent the correlation between 'age' and 'survived'.  I can't figure out how to plot this data because 'survived' is effectively an integer of 0 or 1 (died or lived, respectively)

If I do something like:

titanic_data.plot(x='Age', y='Survived', style='o')

I get a plot that looks like this:

What I would like is a plot that somehow takes the average survival rate by age and created something more like this:


texts: Is there any repository of scientific workflow task execution history?

texts:
I've my final semester project on scientific workflow task execution prediction. Now, the earlier research works, which I've studied, created histories of workflow task executions by actually running those in clouds or grids, which is not feasible for me. I also need the data to test my model. So, is there any repository for scientific workflow task execution history? Or any other way to get some data?
Any suggestions will be helpful, thank you!


texts: Input layer is incompatible even when dimensions (apparently) match

texts:
I am making a sequential neural network for classification, with 3 dense layers, which will be trained on a simple synthetic dataset. The description of dataset is as follows:

Data and class labels are integers. They are 2000 each.
There is only a single feature column (populated by np.arange(2000) * 3)
There is only a single label which indicates last digit of number (populated by np.arange(2000) *3 % 10).

After making the model, I am encountering the following error when calling model.fit():

ValueError: Input 0 of layer sequential is incompatible with the
layer: expected axis -1 of input shape to have value 1500 but received
input with shape (100, 1)

I have uploaded the commented Jupyter Notebook for this code on Google Collab: https://colab.research.google.com/drive/14v92NTBxIEIFJh2BhybfqhawHYIBvKnm?usp=sharing
Any suggestion about how to fix this error and get reasonable accuracy on training set?


texts:
I have a $(418,2)$ matrix and I want to convert it to csv. so I write:

np.savetxt('titanic1.csv', Sol, fmt='%.2f', delimiter=",",header="PassengerId,Survived")

But the header I get is "# PassengerId" instead of "PassengerId" which is what I want. How can I solve this?


texts: What is the Space & Time Complexity of Mini-Batch K-Means clustering algorithm?

texts:
For vanilla K-Means clustering algorithm I know that the time complexity is:

  Time complexity: O(tknm),

where n is the number of data points, k is the number of clusters, and
t is the number of iterations, m is the dimensionality of the vectors.

So, when I studied about Mini-batch K-Means to make the algorithm converge faster, I wanted to find out what is the Space & Time complexity of it?

Essentially so that I understand well, how much we are optimizing over vanilla K-Means.


texts: How can RNN handle variable sized inputs?

texts:
I came across this answer which is specific to Keras. But my question is at concept level. I am getting confused, How can RNN handle variable size inputs?
here

Let us suppose we want to do a  sentiment classification

Training batches of different length : Which I think means all training examples in a batch must be of same length but different batches can have variable length training examples.

Now, suppose length of training examples in

batch-1 = 10,
batch-2 = 15.

Now while training batch-1 , the RNN unrolls and there would be 10 columns in the below figure to handle input of sequence 10. The shared weights W, U,V dimensions must be in sync with sequence length. I mean the shared weights would be w1,w2,..w10 .  I understand till this part.

Now, in While processing batch-2 which is of sequence length 15, The unrolled RNN must take sequence of length 15. That means there are 15 columns in the below figure .

The number of weights changed from 10(w1,w2,...w10) to 15(w1,w2,..w15).
Things don't add up here for me.

What happens to shared weights? How are they learnt?


texts: What is auxiliary loss in Character-level Transformer model?

texts:
I am reading Character-Level Language Modeling with Deeper Self-Attention from Rami Al-Rfou. In the second page, they had mentioned about Auxiliary Losses which can speed-up the model convergence and as an additional regularizer. They said they had 3 kinds of auxiliary losses:

Auxiliary losses at intermediate sequence positions
Auxiliary losses from intermediate hidden representations
Auxiliary losses at target positions multiple steps.

However, I cannot find any information or reference explaining auxiliary losses. I would like to know:

What is auxiliary losses? Is it an unsupervised prediction model using part of the data?
How can I calculate the auxiliary losses?


texts: Where was gradient accumulation originally proposed?

texts: Gradient accumulation is a common technique when training neural network with large batch size that cannot fit into GPU memory. Yet I couldn't find where gradient accumulation was originally proposed. Any hints?


texts: what is the difference between strong and weak clustering?

texts:
What is the difference between strong and weak clustering? and what algorithm is considered as strong and weak clustering? Is fuzzy c-means and bisecting k-means considered as strong clustering?
I haven't found any good explanation of the difference between strong and weak clustering, and the algorithm type of each strong and weak clustering


texts: Would a decision tree classifier be applicable here?

texts:
File from a situation where it is required to predict today’s stockprice from the stock prices of the previous three days:

Could you use a decision tree classifier for this task? Why or why not?


texts:
I am trying to find a way to calculate all possible combinations of a sequence that have a certain length of long run.

When answering questions regarding sequences of heads and tails, sometimes participants will consider a sample space of longest run.
For a sequence of five coin flips, this is easy enough to calculate using a brute-force method - writing out all 32 possible sequences and then categorising them based on their longest run.
So, for example, a sequence with the longest run of 5 has a probability of 1/32 as there is only one way to have a longest run of 5 with a sequence of length 5.

However, I now have a sequence of length 10. I want to find out exactly how many sequences out of the 1024 possible sequences that have a longest run of 2 or 3.
I am assuming that it doesn't matter if the longest run is of heads or tails.

Is there code that could be used to calculate this?

So far I have:

x <- c(0, 1) # heads = 0, tails = 1

p = permutations(n = 2, r = 10, v = x, repeats.allowed = T)

p.df = as.data.frame(p)

To create a data frame of all possible combinations. Now I need to find the longest run of consecutive zeros or ones in each row and count how many rows have the same longest run.


texts: How to approach for predicting semantic similarity between two phrases

texts:
I need pointers on the latest research, tools, and techniques for predicting semantic similarity between two phrases.
Problem Statement: Given two propositions A and B with A know to be true, predict if B agrees, contradicts, or is neutral.
Examples
A = X is greater than Y | B = Y is greater than X | Result = contradict
A = X is greater than Y | B = Y is less than X | Result = agree
A = X is greater than Y | B = P is less than X | Result = neutral
A = X increases with Y | B = X is directly proportional to Y | Result = agree
A = X is joining Y and Z | B = Y and Z are connected with X | Result = agree
What I have tried
I am looking into Google's Universal Sentence Encoder and alternatives for evaluating semantic similarities with mixed results. Most of the solutions available will give a result of exactly similar
for the first example. I am also going through some research papers on grammar-based similarity techniques in natural language.
Any help or direction towards research papers, tools, or libraries is greatly appreciated.


texts: How to find slope of curve at certain points

texts:
how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results .
How to automatically find the points where the slope changes drastically in curve  like around at point 5,6 in below graph
x=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, ]

y=[512, 256, 128, 64 , 32 , 16 , 8  , 7  , 6  , 5  , 4  , 3  , 2  , 1  ]

Numpy gradient give below result

np.gradient(y)
[-256. , -192. ,  -96. ,  -48. ,  -24. ,  -12. ,   -4.5,   -1. ,-1. ,   -1. ,   -1. ,   -1. ,   -1. ,   -1. ]

can we use numpy.gradient to find the slope of curve ? since finding slope of line and curve is bit different Shown in this link
2.Using custom slope function
def slope(x1, y1, x2, y2):
    m = (y2-y1)/(x2-x1)
    return m

slope_value=[]
for i in range(len(y)):
    i += 1
    v=slope(y[i], x[i], y[i-1], x[i-1])
    print(i,v)
    slope_value.append(v)

result: [-0.00390625,  -0.0078125,  -0.015625,  -0.03125,  -0.0625,  -0.125,  -1.0,  -1.0,  -1.0,  -1.0,  -1.0,  -1.0,  -1.0]


texts: Change the way spacy works - Custom properties for training and prediction

texts:
Spacy detects the entities using its predefined algorithm. It parses tokens in text considering position of tokens with respect to tokens surrounding it. It also takes into consideration the POS tagging for these tokens.

However, I believe it misses the position of tokens above and below (For example in a tabular data) or it also misses few properties of text like if it is underline etc.  This statement is based on my understanding. Please correct me if I am wrong in these.

Now the question is, can such properties be taken into consideration while doing training and prediction of entities? I have seen Extension Attributes, but these do not play role during training and prediction but work as meta data.


texts: Predict the corresponding value in one column using a list of values found in another column

texts:
Please have a look at this link. This was a question I asked few months back and after some suggestions and exploring I was able to successfully use TFIDF along with MultinomialNB classifier to pretty accurately predict the 'Item' based on the Composition column. I wrote the answer myself to tell how I solve it. But this time the same logic isn't helping really well for a similar dataset.

Old data:(referencing my previous example in the link above)

UID    Item               Composition
1      Water              Hydrogen,Oxygen
2      Sulfuric acid      Hydrogen,Sulfur,Oxygen
3      Alcohol            Spirit
4      Hydrochloric acid  Hydrogen,Chloride
5      Citric Acid        Hydrogen,Carbon, Oxygen

New data example:

UID    Item                 Composition
1      [Sweater]            [Wool, knitting, handmade, knitting needle]
2      [Jeans]              [Denim, cotton, orange thread, stonewash, blue dye]
3      [CottonTrouser]      [Cotton, littlepolyster, weaving, handstitch, vcut]
4      [SilkShirt]          [wormsilk, artificialsilk, weaving, hand looming, color dying, coating]
5      [Carpet]             [Wool, cotton, organic cotton, knitting, sewing]

This time I have a lot of such data. There around 4200 such items in the Item column. I am trying to use TFIDF with bigrams and trigrams and using MultinomialNB to make the algorithm learn all the items in the Composition and predict the Item accurately.

1) I would like TF-IDF to so the n-grams at a word level so it can vectorise the words.
2) I would like the classifier to classify the Items based on the different compositions so it can learn what compositions put together.

For some items in Item column, they have around 10 values that comprise in the Composition column. So I used min_df=8 and ngram_range=(1,8), hoping it can try to build a vocabulary of around 8 words per composition.

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=8, norm='l2', encoding='latin-1', ngram_range=(1,8),     analyzer='word',lowercase=True,stop_words='english')
features = tfidf.fit_transform(dftest.Composition.values.astype('U')).toarray()
labels = dftest.Indexer
features.shape

No matter what combinations of this I try the predictions are going wrong and I notice that the ngrams aren't really look at the word level in the Composition column.

I need expert advice on how to better solve this problem and mistakes in my approach. Consider me a novice in this area.


texts: Multi Output Regression to Compare the Difference of the Network Outputs

texts:
I would like to build a network with Keras / Tensorflow, where I read in several features and receive two outputs (actual value/setpoint).
I would like to compare these to determine whether they exceed a specified threshold.
Does anyone have experiences/tips with such a regression model?


texts: Are vectors generated by doc2vec and similar models uniformly distributed?

texts:
I have read that vectors in a word2vec model are very much not uniformly distributed and are thought to follow Zipf's law; is this the same for the associated models like paragraph2vec, doc2vec, etc?

My first thought is that they should be similar - if you hand the proverbial monkey a dictionary and allow it to string random words together, almost all of the things it writes won't make sense; the things that do make sense certainly follow some reasonable, very restrictive grammatical rules, etc. I know doc2vec and all the related models seem to pick up on a lot of the grammatical structure (or almost structure), so shouldn't that mean that they'll group grammatical things together?

On the other hand, it's true that there are so very many ways to put words together and so many things to say that they could easily fill out a huge space and maybe could end up being uniformly distributed.

Thanks!


texts: Stuck on building a customer support chatbot from scratch using reddit dataset

texts:
I've a trained model who can mimic day to day conversation occurring on reddit. But, here my problem is that I want it to reply to a specific use cases based on the vocabulary it had learned.

Summary:-
I am building a college chatbot, and there are many use cases. I've trained a model on reddit dataset, and now I've a model who can mimic reddit conversation. But, I want it to map it's vocabulary to one of my use cases. How should I tackle this scenario?

Any idea? Please help, I've tried searching the internet for days, but no answer.


texts: How to decrease the gap between training and validation error?

texts:
I have been executing an open-source Text-to-speech system Ossian. It uses feed forward DNNs for it's acoustic modeling. The error graph I've got after running the acoustic model looks like this:

Here are some relevant information:

Size of Data: 7 hours of speech data (4000 sentences)
Some hyper-parameters:

batch_size       : 128
training_epochs  : 15
L2_regularization: 0.003

Can anyone point me to the directions to improve this model? I'm assuming it is suffering from over-fitting problem? What should I do to avoid this? Increasing data? Or changing batch-size/epochs/regularization parameters? Thanks in advance.


texts: How to Combine tfidf with LSTM in keras?

texts:
I am classifying emails as spam or ham using LSTM and some of its modified form(by adding constitutional layer at the end). For converting documents into vectors I am using keras.text_to_sequences function.
But now I want to use TfIdf with the LSTM can anyone tell me or share the code how to do it. Please also guide me if it is possible and good approach or not.
If you are wondering why I would like to do this there are two reasons:

I want to see if this improves the results.
Second, my professor asked me to perform Latent Dirichlet Allocation, and use same features for both of the tasks.


texts: why use a perceptron when it seems multiple IF staments do the same thing

texts:
If we are to stick with an X and Y axis, the X axis being Time and the Y axis being test scores. Where more time equates to High test scores. You can use a binary classifcation algorithm to predict success. Wouldn't 2 if statements do the same thing

If (time > someValue)
   User will probably pass

Another Scenario
Is I have papayas, and they have two labels, squishness and color.

If the squishness is greater then 7 //on a scale of 1-10
   AND the color is green
     Then it is a good papaya

Else
   It isn't ripe.

I dont understand the value of a perceptron in these scenarios.


texts: What I need to write in the line level of torch.vison for 21 classes?

texts:
In this code I found , line labels = torch.ones((records.shape[0],), dtype=torch.int64) ,that there is only one class and 0 in the case of  Faster RCNN is reserved for the Background. What would be for 21 classes?
def __getitem__(self, index: int):

file_name = self.file_names[index]
records = self.data[self.data['file_name'] == file_name]

image = np.array(Image.open(file_name), dtype=np.float32)
image /= 255.0

if self.transform:
    image = self.transform(image)

if self.mode != "test":
    boxes = records[['xmin', 'ymin', 'xmax', 'ymax']].values

    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    area = torch.as_tensor(area, dtype=torch.float32)

    labels = torch.ones((records.shape[0],), dtype=torch.int64)

    iscrowd = torch.zeros((records.shape[0],), dtype=torch.int64)

    target = {}

    target['boxes'] = boxes
    target['labels'] = labels
    target['image_id'] = torch.tensor([index])
    target['area'] = area
    target['iscrowd'] = iscrowd
    target['boxes'] = torch.stack(list((map(torch.tensor, target['boxes'])))).type(torch.float32)

    return image, target, file_name
else:
    return image, file_name


texts:
My team is looking for a user review dataset which we can compare to one of these Amazon datasets.

The dataset needs to include text reviews and a review helpfulness rating. I know this is a long-shot, but if anyone knows of a dataset we could use then that would be great!

Thanks in advance.


texts:
My x_train shape is (798,3) and y_train input shape is (798, 1). I am creating a RNN like this

def create_rnn_model():
    model = Sequential()
    model.add(SimpleRNN(20,return_sequences=False,stateful=stateful,activation='relu',batch_input_shape=(1,3,1)))
    model.add(Activation('relu'))
    adam = optimizers.Adam(lr=0.001)
    model.compile(loss='mean_squared_error', optimizer=adam, metrics=[root_mean_squared_error])

    return model

But this returns the error

  ValueError: Error when checking input: expected simple_rnn_1_input to
  have 3 dimensions, but got array with shape (798, 3)

My batch size =1 and my timestep is 3 and dat_dim=1 .Then where am I doing it wrong?
Any help is appreciated.

  EDIT

I changed my x_train to shape (798,3,1) and y_train shape to (798,) and ran the model but it threw an error

ValueError: Error when checking target: expected activation_1 to have shape (20,) but got array with shape (1,)

But I can run the model with 1 unit. How do I specify the model to run with 20 units instead of one.


texts: Expectations of the skill set of a Data Scientist

texts: Although there are more powerful tools than Excel which a data scientist should know how to use, would it be expected that someone indicating they are a Data Scientist know how to use Excel?


texts: Why does ada (adaboost) in R return different training error graphs and variable importance plots when running the same function multiple times?

texts: Question says most of it. I created a matrix of descriptors, set the vectors of responses, and input a set number of iterations. Each time I run the function with the same exact inputs, I get the same confusion table, but I get a different training error graph and different variable importance plots.


texts:
I'm doing some data prep on a dataset provided by a telecommunication company. There is a continuous variable that indicates how many months have passed since a customer renewed her contract. However, 20% of observations have missing values. I figured out that there are two reasons why a value might be missing:

1- a customer did not renew the contract

2- a customer is still on her first contract and therefore never had the chance to renew it.

What I want to do is to fill the missing values with respectively "Not renewed" and "Not applicable", and then discretize the rest of the numerical values through a supervised algorithm (chi2, MDLP) so that the variable would turn into a categorical one.

In the end, for this variable I'd have some observations categorized through simple labeling, while others through a supervised discretization algorithm.

Question: is it correct to do so? If not, how should I handle the problem?

Thanks!

Please see my comment below for additional information.


texts: Will historical data lead to target leakage?

texts:
I'm bulding a employee churn model. I've employee data from 2016 to 2019 (of people who stayed/left the company), my goal is to train using data from 2016 to 2018 and predict on 2019.

Since there's people who did not leave the company between 2016-2019 there's alot of repetead employees, so my training set is: data from 2018 about employees who did not leave the company injected with data of employees who left the company in 2016/2017/2018 in order to only have one person once in the training dataset.

My question

Does having only the persons who left in 2016/2017 lead to target leakage?
I'm not using time dummies but can my model overfit thinking employees are more likely to leave the company in 2016/2017 because it's what it sees more often?
And if so how can I avoid this problem?

Thanks :)!


texts: ValueError: Input 0 of layer sequential is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, 25, 25, 1]

texts:
I am trying to use conv1D but getting that error.
My dataset's is batched and has a shape of [None, 25, 25, 1]
I am using input_shape=(25,25)
I am not able to figure out what should I change so I can get it to work.
My model:
model = Sequential()
model.add(Conv1D(32, kernel_size=3, activation='relu', input_shape=(25,25))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))


texts: Amount of multiplications in a neural network model

texts:
I'm currently reading this book and want someone to tell me if what currently I'm assuming about neural network is right or not.   https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/

If a layer has 30 neurons: In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer. That would be 30 neurons of the first layer multiplied by the 30 neurons of the second layer that would give a total of 900 (30*30 = 900) multiplications (is this is correct?)

But those 900 multiplications is only for the first layer and the second layer of the neural network(nn). So if I have a feedforward nn that has 3 layers I would have to do 900 more multiplications because they are the multiplications of the output of the first layer (that they are the input of the second layer) with the weight of the third layer.

So to recap what I said so far I have a feedforward nn with 3 layer with 30 neurons in the first two layers multiply each neuron with all the neurons of the second layer it would be 30 * 30 = 900 multiplications for each additional layer that I add, it adds 900 additional multiplications.

Then for a model with three (3) fully connected layer would be 900 multiplications in the first two layer because for what I said earlier and 900 multiplications in the second and third layer for a total of 1,800 (900 + 900)  multiplications excluding an activation function and this is only for a forward pass, is my understanding correct?

And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass. Am I correct assuming this for a Neural network?. That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.


texts: getting error Error in x - y : non-conformable arrays using nuralnet

texts:
I have a data frame

> str(dataset)
'data.frame':   2354 obs. of  11 variables:
 $ SessionID      : int  11 12 21 21 22 26 33 33 33 46 ...
 $ ItemID         : int  214821371 214717867 214548744 214838503 214837487 214714790 214706441 214820225 214834865 214716932 ...
 $ n              : int  2 1 1 1 1 1 2 2 2 4 ...
 $ diffinscnd     : num  6.68 0 0.053 0.053 0 ...
 $ total_clicks   : int  2 1 2 2 1 1 6 6 6 4 ...
 $ unique_Category: int  1 1 1 1 1 1 1 1 1 1 ...
 $ unique_items   : int  1 1 2 2 1 1 3 3 3 1 ...
 $ Class          : int  0 0 0 0 0 0 0 0 0 0 ...
 $ mean.Class.    : num  0 0 0 0 0 0 0 0 0 0 ...
 $ avg            : num  0 0 0 0 0 0 0 0 0 0 ...
 $ Category       : int  0 0 0 0 0 0 0 0 0 0 ...

when i am trying to apply neural network in using the code below

> set.seed(123)
>  smp_size <- floor(0.80 * nrow(dataset))
>   train_ind <- sample(seq_len(nrow(dataset)), size = smp_size)
>   trainset <- dataset[train_ind, ]
>  testset <- dataset[-train_ind, ]
> maxs <- apply(dataset, 2, max)
>  mins <- apply(dataset, 2, min)
> scaled <- as.data.frame(scale(dataset, center = mins, scale = maxs - mins))
>   trainNN = scaled[train_ind , ]
>  testNN = scaled[-train_ind , ]
> creditnet <- neuralnet(Category ~ SessionID + ItemID + n + diffinscnd + total_clicks + unique_Category + unique_items + Class + avg, trainNN,hidden = 3, lifesign = "minimal",linear.output = FALSE, threshold = 0.1)

then i got the following error

hidden: 3    thresh: 0.1    rep: 1/1    steps: Error in x - y : non-conformable arrays
In addition: Warning message:
In cbind(1, act.temp) :
  number of rows of result is not a multiple of vector length (arg 1)
how to solve this error??


texts: Input data for this dataset to be feed into keras for training

texts:
Suppose I have 3 csv files which forms the dataset for training a machine learning model in Keras.

file1.csv

Name, X1,       X2,         X3
Joe,  1.16,                 1.00,                   1.11
Joe,  1.19,                 1.11,                   1.17
Joe,  1.17,                 1.13,                   1.16

file2.csv

Name, X1,       X2,         X3
Jack,   1.81,               1.23,                   1.15
Jack,   1.34,               1.53,                   1.87
Jack,   1.35,               1.64,                   1.75

file3.csv

Name, X1,       X2,         X3
Bo,     1.42,               1.64,                   1.43
Bo,     1.35,               1.53,                   1.32
Bo,     1.46,               1.64,                   1.53

Based on the data, I will classify whether the person has good or bad performance. For the data above, Joe has good performance while the rest have bad performance.

In keras, the above dataset will be transformed into numpy ndarrays X_train and Y_train to be fed into model.fit() like below;

model.fit(X_train,
          Y_train,
          nb_epoch=5,
          batch_size = 128,
          verbose=1,
          validation_split=0.1)

I am confused over how X_train and Y_train should look like. What should be the shape of X_train and Y_train?

Suppose I have the following dataframes read from the csv files.

df1 = pd.read_csv(file1.csv)
df2 = pd.read_csv(file2.csv)
df3 = pd.read_csv(file3.csv)

How should I use these dataframes to get X_train and Y_train?

I am using python v3, keras with tensorflow.


texts: Remove Outliers - Market Basket Analysis

texts:
I'm having some thoughts on whether I should remove the outliers. I'm trying to find the tags that are commonly used together. Imagine that I have the following dataset. The first column is the Tag_ID and the second column is the Number of People that used that Tag.

1   3472034
2   1277918
3   1249839
4   1010770
5   915099
6   898292
7   636792
8   604352
9   555673
10  298495
11  291511
12  211074
13  200868
...

(This was copied from my actual dataset).

My question is: Should I remove a Tag instance when it is much more frequent than the other? Is that regarded as a good practice?

Many thanks!