texts: Build a tool for manually classifying training data images

texts:
I have a large number of images that I need to classify for training a clustering algorithm, and I would like to do so offline (the data is proprietary). Basically, I'd like to build a desktop survey tool that enables me to rapidly place each image into one or two categories. Ideally, the tool would:

Search in a pre-specified desktop folder for an image;
Display the image and a static list of categories, allowing me to click on one;
Upon clicking, record the category associated with the image;
Store the image filename and associated category in a dataset somewhere;
Display the next untagged image in the folder and repeat the process.

Is there an easy way to build this kind of tool in Python, or some other pre-built utility that I could use for free offline?


texts:
I am currently trying to import the apyori module in PyCharm. However, I am getting the ModuleNotFound Error. i have installed the module by typing pip install apyori on my terminal but I am still unable to import the module on my project. Why is this happening and how can I solve this problem?
Thanks!


texts:
Is there a good toolbox for handling and analyzing missing values in Python 2.7?

There is a good toolbox for doing this in Python 3.6 here (missingno): https://github.com/ResidentMario/missingno

I need to work in Python 2.7. so, this is why I ask.


texts:
I am hand-coding a transformer (https://arxiv.org/pdf/1706.03762.pdf) based primarily on the instructions I found at this blog: http://jalammar.github.io/illustrated-transformer/.
The first attention block takes matrix input of the shape [words, input dimension] and multiplies by the attention weight matrices of shape [input dimension, model dimension]. The model dimension is chosen to be less than the input dimension and is the dimension used as output in all subsequent steps.
There is a residual connection around the attention block and the input is meant to be added to the output of the attention block. However the output of the attention block is shape [words, model dimension] and the input is form [words, input dimension]. Should I interpolate the input down to the model dimension as is done in ResNet? Or maybe add another weight matrix to transform the input?


texts: Extracting Features for Graph transformation

texts:
Suppose I have a directed graph G (V,E) whose transformation is defined by a library of patterns. Each vertex is of particular type.

The library of patterns contain subgraphs (g1,g2,g3 etc)and it's associated transformations(g'1,g'2,g'3 etc). I search the  graph for each of these patterns and replace them with the associated transformed graph. After all detected patterns are substituted I get the resulting graph G'. Basically this is a search and substitute, of subgraphs.

The problem is this. I wish to use machine learning predict an estimate of the number of types of vertex in transformed graph. What could be possible features to extract ?


texts: Good lectures (books, articles, blogs) about the art of modeling a CNN network

texts:
I have been using Deep Learning with Tensorflow for a pet project only for educational purpose, and have been able to obtain quite good results with a CNN of my own.

However, the process I used to get there is not satisfying, because it looks like trial/error based on basic metrics: performance on training set and evaluation sets. I do not know if this is efficient, or if some layers are useless or over/under sized. Same for the kernel sizes.

Do you have any book or article or blog to recommend on the subject, for learning to be a better practitioner ?
I have read "The Bengio" ("Deep Learning") and another book in French about practicing Tensorflow ("Comprendre le Deep Learning: Une introduction aux réseaux de neurones"), but none gave a practical method. This looks like craftmanship only.


texts: Asynchronous Hyperparameter Optimization - Dependency between iterations

texts:
When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), should I expect:

Dependency between sequential iterations where the loss value improves sequentially (e.g., the optimized hyperparameters in iteration number 10 are better than the optimized hyperparameters generated in iteration number 9, etc.). In this case I should always select with the hyperparameters generated in the last iteration.

or

Expect independency between iterations where after all 100 iterations are completed I should select the iteration with the smallest loss value.

If option a) is the right answer, that what does it mean if the best Hyperparameter are associated with iteration 50, does it mean that the data is not stable, or the loss function is ill-specified, and therefore, the hyperparameter optimization process outcome should not be trusted?


texts: How to make use of POS tags as useful features for a NaiveBayesClassifier for sentiment analysis?

texts:
I'm doing sentiment analysis on a twitter dataset (problem link). I have extracted the POS tags from the tweets and created tfidf vectors from the POS tags and used them as a feature (got accuracy of 65%). But I think, we can achieve a lot more with POS tags since they help to distinguish how a word is being used within the scope of a phrase. The model I'm training is MultnomialNB().
The problem I'm trying to solve is to find the sentiments of tweets like positive, negative or neutral.
Structure of datset:
Created pos tags:
I created tfidf vectors from the tweet and gave the inputs to my model:
tfidf_vectorizer1 = TfidfVectorizer(
    max_features=5000, min_df=2, max_df=0.9, ngram_range=(1,2))
train_pos = tfidf_vectorizer1.fit_transform(train_data['pos'])
test_pos = tfidf_vectorizer1.transform(test_data['pos'])

clf = MultinomialNB(alpha=0.1).fit(train_pos, train_labels)
predicted = clf.predict(test_pos)

With the above code I got 65% accuracy. Rather than creating TF-IDF vectors of POS and using them as modal inputs. I'm wondering is there any other way that we can use POS tags to increase the accuracy of the model?


texts: Computing SVD through Eigendecomposition of correlation matrix

texts:
I am following the excellent series on SVD by Steve Brunton from the University of Washington, on YouTube, but I have trouble interpreting his 4th video on the subject.
If I understand correctly, he mentions that one can compute the economy SVD decomposition $X = \hat{U}\hat{\Sigma}V^T$ with the following :
$$X^TX= V\hat{\Sigma}\hat{U}^T\hat{U}\hat{\Sigma}V^T = V\hat{\Sigma}^2V^T \implies X^TXV = V\hat{\Sigma}^2 $$
$$XX^T= \hat{U}\hat{\Sigma}\hat{V}^T\hat{V}\hat{\Sigma}\hat{U}^T = \hat{U}\hat{\Sigma}^2\hat{U}^T \implies XX^T\hat{U} = \hat{U}\hat{\Sigma}^2 $$
Instead of using U,S,VT = svd(X) in Python, I want to try decomposing the image in its SVD, and reconstructing the original using only $r$ values with this technique. I am trying to apply this on a picture of The Starry Night, loaded in a grayscale numpy array X, which I do this way :
r = 10
XT = X.transpose()
C_1 = XT @ X
C_2 = X @ XT
[Lambda_1, V_hat] = np.linalg.eig(C_1)
[Lambda_2, U_hat] = np.linalg.eig(C_2)
V_hat_T = V_hat.transpose()
X_tilde = U_hat[:,:r] @ np.diag(Lambda_1)[0:r,:r] @ V_hat_T[:r,:]

But I can't reproduce the image, whatever the value I attribute to $r$, whereas the product of U @ S @ VT , computed with the svd(X) function, perfectly reconstructs the original picture.
What am I doing wrong? It is certainly superfluous to mention that I am a beginner, and I probably made some big mistake.


texts: How do I use LSTM Networks for time-series classification problems?

texts: I have 2 binary outputs (1 and 0) with time series data. The dataset order is shown in the image..Can anyone suggest me how to handle this problem with LSTM? Particularly in MATLAB or Python. Thanks in advance.


texts: How to generate multiple vectors given their distances

texts:
Well, the title says it all. I'm looking for an algorithm that would generate multiple random vectors with given distances between them.
Any suggestions?


texts: How to convert categorical data to numerical data in Pyspark

texts:
I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to perform a classification algorithm taking all the inputs to determine the income range. I need to build a dictionary of variables to mapped variables and use a map function to map the variables to numbers for processing. Essentially, I would my dataset to be in a numerical format so that I can work on implementing the models.

In the data set, there are categorical columns like education, marital status, working class etc. Can someone tell me how to convert them into numerical columns in pyspark?

workclass = {'?':0,'Federal-gov':1,'Local-gov':2,'Never-  worked':3,'Private':4,'Self-emp-inc':5,'Self-emp-not-inc':6,'State-gov':7,'Without-pay':8}

I created a sample dictionary with key value pairs for work class. But, I don't know how to use this in a map function and replace the categorical data in the CSV file with the corresponding value.

wc = pd.read_csv('PATH', usecols = ['Workclass'])

df = pd.DataFrame(wc)
wcdict = {' ?':0,' Federal-gov':1,' Local-gov':2,' Never-worked':3,' Private':4,' Self-emp-inc':5,' Self-emp-n-inc':6,' State-gov':7,' Without-pay':8}
df_new = df.applymap(lambda s: wcdict.get(s) if s in wcdict else s)
print(df_new)

This is the code I have written in normal python to convert the categorical data into numerical data. It works fine. I want to do the conversion in spark context. And, there are 9 categorical columns in the data source. Is there a way to automate the dictionary update process to have a KV pair for all 9 columns?


texts: Tuning hyper parameters for different models with caretList

texts:
I'm trying to train an ensemble using the caretList function in the caret package.
I'm using these models:
model_list <- caretList(Ticket_ContrattoTerminato ~ .,
                        train[,-1],
                        trControl = my_control,
                        methodList = c("glm", "svmRadial", "rf",
                                        "xgbTree"),
                        metric="ROC",
                        tuneList = NULL,
                        continue_on_fail = FALSE,
                        preProcess = c("center","scale"))

I'd like to tune the hyper parameters for every single model but I don't know how to do it.
Do you have any suggestion?
Thank you!


texts: Dropping less frequently used categorical data?

texts:
I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite some codes which occur with random frequency (the most frequent appearing in 64K records, the lest in one record, quite a large number ) The number of categories having only 1 record in the dataset is quite large (35), and about 110 categories have 10 or less occurences;

>>> df.groupby('category')['target'].agg(['count','mean']).sort_values('count')
                      count      mean
category
code__1                   1  0.000000
//readabity
code_36                   1  0.000000
code_37                   2  0.000000
code_38                   2  0.500000
//readability
code_103                 10  0.000000
code_151                 73  0.000000
code_175                706  0.247875
code_188              23145  0.032059
code_189              64414  0.074006

For the total dataset we have:

>>> df['target'].agg(['mean','count'])
mean          0.072536
count    159880.000000

I'm a bit concerned that my dataset will grow out of hand when I'm using all categories and convert them to 189 binary columns with pd.get_dummies. I'm thinking of throwing out all categories with less than 10 occurences and replacing them for a dummy category. Now my 3 questions:

is this wise? will I introduce data leakage or mess up future datasets?
is there a way to do this in sklearn/pandas? I'm thinking of a python approach by storing code103..189 into a list cats, iterate over my category column and substitute all categories not in the cats list to dummy, this is reproducible over a future set or test set.
Another approach would be to replace all categories to dummie where the mean is close to the total mean of target column? then I would ditch code_189 (the most frequent) because its varies barely from 0.072?


texts: Handling Categorical Features on NGBoost

texts:
Recently I have been doing some research on NGBoost, but I could not see any parameter for categorical features. Is there any parameter that I missed?

__init__(self, Dist=<class 'ngboost.distns.normal.Normal'>, Score=<class 'ngboost.scores.MLE'>, Base=DecisionTreeRegressor(ccp_alpha=0.0, criterion='friedman_mse', max_depth=3,
 |                        max_features=None, max_leaf_nodes=None,
 |                        min_impurity_decrease=0.0, min_impurity_split=None,
 |                        min_samples_leaf=1, min_samples_split=2,
 |                        min_weight_fraction_leaf=0.0, presort='deprecated',
 |                        random_state=None, splitter='best'), natural_gradient=True, n_estimators=500, learning_rate=0.01, minibatch_frac=1.0, verbose=True, verbose_eval=100, tol=0.0001)

https://github.com/stanfordmlgroup/ngboost


texts: What should I do to test the confidence of my deep learning model?

texts:
I've recently fine-tuned a deep learning framework/model BERT for a sentiment classification task. I had a 80/10/10 train/validation and test set. After running several experiments, I've gotten a decent model that I'd like to eventually productionize. However, I'm looking to create an experiment to test the robustness/reliability/confidence of the model before putting it into production. What are some ways/experiments that can be conducted to test the robustness/reliability/confidence of this model or its predictions?
For example, are there statistically sound principles behind calculating the standard error for binary predictions on a new datapoint?


texts: How to interpret coefficients from logistic regression?

texts:
I ran a logistic regression (statsmodel) on my data with 60 features using the below code

import statsmodels.api as sm
logit_model=sm.Logit(y_train,X_train_std)
result=logit_model.fit()
print(result.summary())

I was able to see that few variables had negative coefficient and few had positive coefficients.

Am I right to understand that irrespective of sign of coefficients, all the below variables are significant predictors that influence the outcome?

Or does negative coefficient mean they don't have any influence on the model outcome? But p-value is significant. Am a bit confused. Can you help in simple terms please

The below output shows the records whose p-values were less than 0.05


texts: ML in R (caret-package) missing hyperparameters

texts:
I have a pretty specific question regarding the caret package however I still hope to finde help here.
I recently worked with the caret package and trained a multilayer perceptron with method = 'mlp'. I looked up the github page of Max Kuhn (developer of caret), and it says that you only need to tune one hyperparameter: the size (number of neurons in the hidden layer). Which is really convinient.

However it further states that caret for the training builds on the RSNNS Package (by Bergmeier). The mlp model implemented in this RSNNS package has additional tunable parameters over just the size hyperparameter (i.e. learnFunc,hiddenActFunc,Std_Backpropagation, maxit).

So I asked myself what values caret uses for those parameters? Default values or are those optmizied?


texts: Recommendation System using on-device ML or Cloud ML?

texts:
I am developing a Flutter/Dart app which would need a recommender system to provide personalized recommendations to users for new photos, videos, ads, etc. relevant to their interests.
Can this be implemented using TFLite because I am hesitant to use Cloud ML services as I want the users’ data to be private. If this can’t be done using TFLite, What service should I learn: AWS ML, Azure ML, etc?


texts: Discrepancy between training set and real-world data set: domain adaptation?

texts:
I have read in literature that in some cases the training set is not representative for a real-world dataset. However, I cannot seem to find a proper term describing this phenomenon; what is the proper term to address this problem?

Edit:

So far I have settled for the term domain adaptation, shortly described as a field in machine learning which aims to learn from a certain data distribution in order to predict data coming from a different (but related) target distribution.


texts: Why do recurrent layers work better than simple feed-forward networks?

texts: On a time series problem that we try to solve using RNNs, the input usually has the shape $input features \times timesteps \times batchsize$ and we then feed this input into recurrent layers. An alternative would be to flatten the data so that the shape is $(input features \times timesteps) \times batchsize$ and use a fully connected layer for our time series task. This would clearly work and our dense network would be able to find dependencies between the data at different timesteps as well. So what is it that makes recurrent layers more powerful? I would be very thankful for an intuitive explanation.


texts: Correct learning algorithm for grouped examples

texts:
I need to learn which item will be (most probably) picked from a group. The data I have consists of groups by 10 items and which one was picked(or if neither was picked).

My current approach is to ignore the groups and to use flat list with all items classified as picked or not picked and use some classification algorithm, but I wanted to know if there is a way to use the knowledge that the item was picked from that specific group.


texts: Extract density estimator from discriminator (MLE method) in adversarial training

texts:
I'm trying to understand if the latest ML methods associated with GANs can estimate the pdf of complex joint distributions from data (as opposed to the generator model).

So far I've found from Ian Goodfellow's paper, pg 29:

"For GANs, the generator is $P_{model}$, while
for NCE and MLE, $P_{model}$ is part of the discriminator. Beyond this, the differences
between the methods lie in the update strategy. GANs learn both players with gradient
descent. MLE learns the discriminator using gradient descent, but has a heuristic
update rule for the generator. Specifically, after each discriminator update step, MLE
copies the density model learned inside the discriminator and converts it into a sampler
to be used as the generator. NCE never updates the generator; it is just a fixed
source of noise."

Does anyone know further details of this process he refers to?  The closest thing I've found is this, which uses the well-known discriminator relationship

$D(x)=\frac{P_{data}}{P_{data} + P_{model}}$ and inverts to get $P'_{data}=P_{model}\frac{D(x)}{1-D(x)}$ where $P'_{data}$ is the updated  $P_{model}$ for the next iteration.  Then the generator on the next iteration is $P'_{data}$ via rejection sampling.  By starting with a known pdf this has reliably converged in my experience.

The problem here is that the model continues to grow each iteration.  It's not especially scale-able.

Does anyone know if Ian refers to a method that is trained more similarly to the GANs, with a "static" sized model in which the pdf can be extracted from the discriminator at the end of training, and the generator can be more like what we see in GANs (as opposed to rejection sampling)?


texts: Can metadata be used to adapt parsing for an unescaped in field use of the delimiter?

texts:
I have data coming from a source system that is pipe delimited. Pipe was selected over comma since it was believed no pipes appeared in field, while it was known that commas do occur. After ingesting this data into Hive however it has been discovered that rarely a field does in fact contain a pipe character.

Due to a constraint we are unable to regenerate from source to escape the delimiter or change delimiters in the usual way. However we have the metadata used to create the Hive table. Could we use knowledge of the fields around the problem field to reprocess the file on our side to escape it or to change the file delimiter prior to reloading the data into Hive?


texts: Dealing with pre-trained model for grayscale images

texts:
I would like to do Transfer Learning using one of the novel networks such as VGG, ResNet, Inception, etc.
The problem is that my images are grayscale (1 channel) since all the above mentioned models were trained on ImageNet dataset (which consists of RGB images).
One of the solutions is to repeat the image array 3 times to make it 3 channel.
Is this really the only solution for that? Is it a good solution? Are there any other solutions?


texts: Concatenating embedding and hand-designed features for logistic regression

texts:
An interviewer told me that we cannot concatenate an embedding from a neural network (such as a  pre-trained image representation) and hand designed features (such as image metadata) for use in a linear model such as logistic regression. He says they must only be used with neural networks.
I understand the resulting data may not be linearly separable but that to me seems like a case-by-case empirical question.
Is he right? Can someone explain why?


texts: How to render environment in Tensorforce?

texts:
How can one render the environment using the Tensorforce library?

I've tried calling environment.render, but it says that the function does not exist. This is my code:

from tensorforce.agents import Agent
from tensorforce.environments import Environment
from tensorforce.execution import Runner

# Create an OpenAI-Gym environment
environment = Environment.create(environment='gym', level='MountainCarContinuous-v0')

agent = Agent.create(agent='random', environment=environment)

# runner = Runner(agent=agent, environment=environment) # Initialize the runner
# runner.run(num_episodes=NUM_EPISODES) # Start the runner
# runner.close()

# Train
for ep in range(NUM_EPISODES): # Number of episodes

    print('********Episode ' + str(ep) + '********')

    # Initialize episode
    states = environment.reset()
    done = False
    step = 0

    while not done: # Episode timestep
        actions = agent.act(states=states)
        states, done, reward = environment.execute(actions=actions)
        agent.observe(terminal=done, reward=reward)
        environment.render() # Gives error

environment.close()
agent.close()

And this is the error I get:

Traceback (most recent call last):

  File "c:\users\user\reinforcement learning\rl.py", line 179, in <module>
    environment.render()

AttributeError: 'OpenAIGym' object has no attribute 'render'


texts:
So I was reading about the LSTM architecture and I was having trouble understanding a certain aspect of it. This article mentions the step in question near the bottom of the page. Here is the image given:
The issue I have is this: If ot is the result of an operation on the concatenation of ht-1 and xt, then ot will be larger than ht-1. Then, ht is calculated by an operation on the now larger ot. ht is then 'passed' on to the next timestep. So wouldn't the size of h continually increase over time?
I'm definitely misunderstanding some part of this, so thanks in advance for any help. :)


texts: Alignment of square nonorientable images/data

texts:
Another post where I don't know enough terminology to describe things efficiently. For the comments, please suggest some tags and keywords I can add to this post to make it better.

Say I have a 2D data structure where 'orientation' doesn't matter. The examples I ran into:

The state of a 2048 game. In terms of symmetry groups this would be D4 / D8, except that an operation doesn't yield an identical state, it just yields another state that has the same solution.
Images of plankton or galaxies (without background). Somewhat similar to above except that any rotation (not just 90o) yields an equally valid image (and one might take scale into account, but let's forget about that).

In both cases I've wanted to transform all these equivalent states/images to remove all but one of the equivalent images. To illustrate with two that worked:

I can use image moments M10 and M01 to transform horizontally and vertically mirrored equivalent data. E.g. apply horizontal mirroring iff it makes M10 bigger. This would transform a 2048 state and it's horizontal mirror image to the same state.
I can use the eigenvector of the covariance matrix which has the largest eigenvalue as the orientation. Then I can rotate the image to align this eigenvector with some predetermined axis (e.g. horizontally).

That still leaves a lot of operations though (diagonal mirroring, rotations around the center, inversion). And these operations do not commute (D8 is non-Abelian). Is there any comprehensive approach?

The reason I want to do this is to help machine learning methods by removing variance that isn't actually meaningful. Hopefully that makes sure they don't have to learn these equivalences, so possibly need less train data (and time).


texts: Force Neural Net to attempt to predict every class

texts:
I am training a (deep) neural net to classify approximately 60 different classes. The range of occurrences of each class in the dataset is wide, 3 orders of magnitude from the most represented to the least represented. The neural net currently never predicts any of the 20 least represented classes.

Are there ways to mitigate this other than per class weighting, such as increased dropout or some other method? Can this be mitigated in the network structure? If per class weighting is the only option, is the best scheme to weight the class inversely to the frequency of occurrence?


texts: how to use Bayesian theorem and probabilistic analysis?

texts:
could someone help me to solve this problem please ?

Your prize Rapid Ripe tomato plant has flowered and is ready to start producing fruit. If all goes well, your plant will produce tomatoes by the end of the week. It will then produce a new set of flowers and blossoms next week. Unfortunately, your area is subject to blossom wilt, which causes the tomato flowers to fall off. If the blossoms fall off, a new set of blossoms will not emerge until next week, and tomatoes will not be ready until the end of that week. Luckily, each time blossoms fall off, the plant builds up its resistance; the probability of each succeeding blossom falling off is then only half as much. You estimate that the probability of this first set of blossoms falling off is 0.40.

What is the probability tomatoes will be ready in
the third week?
What is the expected number of weeks you will have tomatoes over the next three weeks?
What is the probability you will lose the blossoms one or more weeks of the next five weeks?


texts: Prediction from data series with varying features

texts:
I'm looking into a problem where the data points have unequal features.

Each instance represents the progression of an item throughout the system. A number of them have progressed to their end point, others are still at intermediary stages. The number of stages is known (10 in total). I have the time when it enters each stage. The point is to make predictions as to when they will reach their final stage.

There are a number of ways to handle this. It seems that a decision tree might be able to handle this properly. Even if a stage's entry time isn't known, it should still be able to make a reasonable prediction from the data it does have.

Can someone point me in the right direction?


texts: Is something missing in this problem involving dynamic time warping, specifically the description of each sequence?

texts:
this is a question on my assignment I'm having trouble with. I've arrived at an answer, but I'm now beginning to question it since it wasn't discussed in the lecture.

As you can see, the question is to find the best DTW path for the sequences shown in the image. I'm not sure what exactly is going on here, because I don't understand what sequences I'm being shown. To me, it just looks like a matrix of dots with four line segments outlining it. However, I've tried attempting this question regardless.
According to the accompanying lecture, DTW requires two sequences in time which do not need to be the same length. If you assume the first sequence has $m$ elements and the second one has $n$ elements, then you construct an $n*m$ matrix. On the vertical axis, you lay out the second sequence, bottom to top, and you put the first sequence left-to-right on the horizontal access. You then calculate either the Euclidean distance or Manhattan distance for each pair of elements. You put these values in each cell of the matrix. Your DTW path would then be the path starting from the bottom left and ending at the top right, following the trail of minimum values in the matrix. You can move in one of eight directions from an element.
Using the last line of the preceding paragraph, it becomes pretty easy to arrive at an answer for this question. Since you can't go from $(1,1)$ to $(2,3)$ without violating the direction of motion, those answers are ruled out. You can't go from $(2,2)$ to $(2,4)$ in a single step as well, which rules out option B. That leaves me with option C, which I think is the right answer.
Is this correct? Using Inspect Element on both Firefox and Chrome, I've checked the HTML of the webpage this question appears on. This is the only image that's loaded for this specific question. I don't see what two sequences the question seems to be referring to, unless it's something to do with the rectangular border surrounding the dots. I've checked all of the textbooks prescribed for this course; none of them have any diagram like this. I know how the DTW algorithm works, but I'm not sure how it's applied here. Is my answer, option C, correct? Or is there something blindingly obvious I'm missing?


texts: Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?

texts:
Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects:

simple
more biologically plausible
no gradient to vanish
better performance
sparsity

And I see only one advantage of sigmoid/tanh: they are bounded. It means that your activations won't blow up as you keep training and your network parameters won't take off to the sky.

Why should not we forget about sigmoid/tanh for deep neural networks?


texts: What does the inital_state parameter in the GRU call arguments do?

texts: Does the inital_state parameter in the GRU call arguments, specify the inital state of the hidden state, that is, $h_t$ or the weights?


texts: Goodness of fit metric to compare topic models NMF vs LDA

texts:
Does anyone have a good idea for how to compare topic modeling done by NMF and LDA? Let's say I fit LDA to a dataset and generate topic-word and document-topic distributions--I can use perplexity, for instance, to measure the goodness of fit of the model. If I fit NMF to the same dataset and generate topic-word and document-topic distributions, I can take the resultant fitted data as $X^{Est} = HW^T$ and use $\|X^{Original} - X^{Est}\|_{Fro}$ as a goodness of fit of the model.

What is an objective metric that I can use to compare the goodness of fits to each other? Can either of these metrics be used for the other model?


texts: Using a multi-headed neural network, how should I approach the regression head loss

texts:
I have a multi-headed NN where one head performs multi-label classification and the other a regression task on a set of images. The classification head outputs a one-hot vector where each value in the vector represents a probability for the given class being present in the image. The regression head should provide a prediction of the percentage of the image that is covered by each class class.
So a dummy example of the target classes and network predictions would look like:
classes = ['tree', 'car', 'ground', 'sky']
class_prob_preds = np.array([0.03, -0.17, 0.58, 0.73])
area_preds = np.array([0.05, 0.01, 0.42, 0.50]) # regression values don't need to add to 1
class_preds = np.where(class_prob_preds > 0.5) # gives the index of the predicted classes

My question is, how should I pass to the loss function for the regression part? Should I feed the entire vector of area predictions to the regression loss function, or should I use the predicted classes of the classification head to select only the values in the regression vector that correspond to the predicted classes and pass these to the loss function?


texts:
I am trying to build a classifier that detects if I have a malaware by predicting the provenance compiler. To do so I have a dataset composed of assembly code in json format :

In particular, I want to select as features the instructions, so the push, mov, jmp,..etc and create a feature vector that contains the number of times a feature appears. So, I want to apply the bag of words. To do so my code is the following:

#libraries
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

json_data = None;
with open('training_dataset.jsonl') as data_file:
    lines = data_file.readlines()
    joined_lines = "[" + ",".join(lines)+"]"

    json_data = j.loads(joined_lines)

data = pd.DataFrame(json_data)
data.head()

vect = CountVectorizer()

data['instructions'] = data['instructions'].apply(lambda x: ' '.join(x))

vect.fit_transform(data['instructions'])

a =vect.vocabulary_
a

and from this I obtain a dictionary:

with more and more key, value pairs. At this point what I tried to do is to eliminate the registers so my idea to do this is to iterate the dictionary and eliminate the keys that have numbers, and so I have written the following:

def hasNumbers(inputString):
    return any(char.isdigit() for char in inputString)

for k in list(a.keys()):
    if hasNumbers(k):
        del a[k]

but actually it wasn't a really good idea because I have also registers without numbers, for example rdi.

At this point, I don't know how to move, and I am not sure I am going in the correct way. Can somebody please help me? Thank's in advance.

[EDIT 2]
Now I am trying to take the json file, not opening it with pandas, but open it as a dictionary, so:

and my intenction is try to split, as suggested, each key in such a way to eliminate the registers. So, my code for this is :

for key in json_data:
    splitted = key.split[0]

but I recieve the following error message:

AttributeError: 'dict' object has no attribute 'split'

I am getting really confused about how to operate now.

[EDIT 3] or I tried to do the following:

for v in json_data.values():
    splitted = v.split[0]

but it gives me the following error message:

AttributeError: 'list' object has no attribute 'values'

a problem I think is the fact that the various push r12,.. etc are in a list, so I don't know how to do.

[EDIT 4] I actually tried to solve the above problem this way:

for v in json_data[0].values():
    splitted = v.split[0]

but again I get the error message:

AttributeError: 'list' object has no attribute 'split'

[EDIT 5] Now, I am trying to do this using the file opened with pandas, so:

data['features'] = [i.split[0] for i in data['instructions']]

but still I get the following error:

AttributeError: 'list' object has no attribute 'split'

or I tried also:

feat = [i.split[0] for i in json_data[0].values()]

but get the same error as above.


texts: local minima vs saddle points in deep learning

texts:
I heard Andrew Ng (in a video I unfortunately can't find anymore) talk about how the understanding of local minima in deep learning problems has changed in the sense that they are now regarded as less problematic because in high-dimensional spaces (encountered in deep learning) critical points are more likely to be saddle points or plateaus rather than local minima.

I've seen papers (e.g. this one) that discuss assumptions under which "every local minimum is a global minimum". These assumptions are all rather technical, but from what I understand they tend to impose a structure on the neural network that make it somewhat linear.

Is it a valid claim that, in deep learning (incl. nonlinear architectures), plateaus are more likely than local minima? And if so, is there a (possibly mathematical) intuition behind it?

Is there anything particular about deep learning and saddle points?


texts: Incrementally Train BERT with minimum QnA records - to get improved results

texts:
We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)

It generated new checkpoints and BERT is giving good answers for most of questions we asked on our text documents. However, there are some questions which it is answering wrong, so we are trying to further fine tune with our Question and known answer on our text document. We further trained based on last generated checkpoint and got new checkpoint.

With new checkpoint when we are asking the same question, the answer did not got corrected! Previously BERT was giving wrong answer with 99% confidence and now also giving same wrong answer with 95% confidence.

Can someone suggest, if they have same or similar experience, and suggest please.
Following are questions in BERT github Issues, and are unanswered for quite some time:

BERT accuracy reduced after providing custom training..The answer is
also not correct : https://github.com/google-research/bert/issues/492
Unable to incrementally train BERT with custom training: https://github.com/google-research/bert/issues/482
Little training has no impact:    https://github.com/google-research/bert/issues/481
Custom Domain Training: https://github.com/google-research/bert/issues/498


texts: How to combine rows after Pandas Groupby function

texts:
May I know how to combine several rows into one single row after I used Pandas groupby function?

In below example, I would like to to group the data by Employee ID, Customer Last Name and Customer First Name. Then I want all his dependents' data listed in the same row.

Thanks a lot!


texts: How to analyse the results of cross-validation do determine overfitting

texts:
I performed k-fold CV and measured the resulting average error (RMSE) for each fold. This was done with 5 folds, and 4 of the measurements gave similar errors (between 10% and 12%), but one of the tests has given a 4% error.
What can be concluded in regards to overfitting in this experiment?
Is the model overfitted because it works much better in one of the situations than in the others?
Thanks.


texts:
I want to perform Doc2Vec on a twitter dataset. As each tweet consists of a nummber of special characters ,numbers, urls, mentions and hashtags, non-english words, what should be my input for Doc2Vec? How should i approach for initial tweet pre-processing?

I saw many tutorials but all of them used plain texts. I am a newbie in data science.


texts:
My goal is to be able to extract data from web pages such as address, product categories and their products on e-commerce sites.

Performing this operation on a regular expression basis is not possible on a large number of websites.

How could we do that with Tensorflow? Would you have an example in mind or the methodology to achieve this?


texts: Are stationarity and low autocorrelation the prerequisite of regression model?

texts: As said in the title, are stationarity and low autocorrelation the prerequisite of general / linear regression model ? That is, if a time series is non-stationary or has large autocorrelation, would it be easier or harder to be predicted using regression model such as linear model and deep learning ?


texts: Is Oracle 11g is able to ingest big data?

texts: What is the sizing limitation for Oracle 11g? Can I use it as big data platform? From the following link I understand that there is no limitation for the records, only the columns are bounded to 1000. So if my big data has less than 1000 per table but contains many records can I use Oracle 11g as big data platform?


texts: Continous bag of words claimed to be unsupervised, how is it working?

texts:
I'm following these two lectures on CBOW and skip-gram word2vec models. The first is lec 12 and the next lec 13 of a deep learning series

https://www.youtube.com/watch?v=syWB-YMYZvI

https://www.youtube.com/watch?v=GMCwS7tS5ZM&t=548s

Up to about 17 mins into the second video the lecturer says that the approach is unsupervised for CBOW as there are no labels? How can you learn a NN with no labels? This completely confused me as why are we not comparing our softmax probability vector to an actual set of outputs so that we can adjust the v_c and v_w weights accordingly. His liklihood function seems to only be concerned with the parameters v_c and v_w (and completely devoid of some kind of target label) which is bonkers to me, because could i not not just make them whatever i want? In addition how are you learning relationships between pairs of actual words, if actual target variables are not guiding you towards correct labels? Could someone please explain what is going on under the hood as i really want to understand this approach. Perhaps target labels are being considered and i've not noticed it?

Most log likelihood estimates with a window of size $m$ look something like the following $-\mathrm{log} \prod_{j=0,j \neq m}^{2m} \frac{e^{u^T_{c-m+j}v_c}}{\sum_{k=1}^{|v|} e^{u_k^Tv_c}} $

To my knowledge a likelihood function should involve data from actual observations not purely parameters, which are to be estimate, can someone explain this also?

Note i follow it more from 17 mins onwards as he talks about pairs (w,c) in D or D' respectively. I appreciate any help!


texts: F-test for comparing the mean of two groups

texts:
What would be the procedure, in terms of F-test, if I would like to check if the mean of one group is greater than the other group (alternative hypothesis: $\mu_1 > \mu_2$)?

And also what would change for the opposite alternative hypothesis: $\mu_1 < \mu_2$?

I know how to do it with t-test but got confused for the F-test, especially in terms of how to differentiate between left-tail and right-tail in an F distribution.


texts:
I have two models: one a simple linear regression and the other a multi linear regression.

The strongest relationship is between the two variables in the simple linear regression. The multi linear regression includes two extra variables that can be connected with the dependent, however, statistically they are insignificant (P > 0).

How do I interpret whether the best model to use is the simple linear regression or the multi-linear regression?

Here are some values from the regression:

Simple Linear : F(1, 77) = 21.07 , Prob > F = 0.0000 , Rsquared = 0.2148, Adj Rsquared = 0.2046

Multi Linear: F(3, 75) = 7.29 , Prob > F = 0.0002, Rsquared = 0.2258, Adj Rsquared = 0.1948


texts:
I am trying to read and plot several files which looks like as below when I open with python using pandas read.table. The columns to plot are "dev" vs timestamp.

The timestamp should be compose from the columns; mm dd,time and yyyy.

    mm dd time yyyy dev st fault typ
0   Jul 5 2:48:29 2018 aaa STANDBY HRW_FAULT neg
1   Jul 5 2:48:29 2018 aaa  SOFT SWF_FAULT  ack
2   Jul 5 2:48:29 2018 aaa HARDWARE  disable
3   Jul 5 2:50:47 2018 bbb STANDBY HRW_FAULT pos

...............


texts: Why do we need to handle data imbalance?

texts:
I would like to know why we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue - by up sampling or down sampling or by using SMOTE.
For example, if I have a rare disease 1 percent out of 100, and lets say I decided to have a balanced data set for my training set which is:  50/50 sample
won't that make the machine think 50% of patients will have disease? even though the ratio is 1 of 100.
So

Why do we need to deal with data imbalance?
What is the recommended ratio to have balance set?


texts: Is it possible to use Jaccard similarity instead of Cosine similarity in gensim document similarity?

texts: I am using gensim library to compute similarity between documents but it only uses cosine similarity. I was wondering if there was a way to use jaccard similarity or any other similarity measure for that matter instead


texts: clarification on train, test and val and how to use/implement it

texts:
So far I think I understood the differences between the training, test and validation set. Basically it is like in this image:

Training set: The data where the model is trained on

Validation set: Data the model has not been trained on and used to tune hyperparameters

Test set: In principle the same like the validation set.. just used at the final end after the model has been tailored.

The training set is usually set up via some cross validation. When fitting via model.fit(X_train, y_train,..) will Keras shuffle the data autonomously ?

Next, in Keras, you are able to provide the validation set inside the model.fit() method as validation_data=(x_test, y_test) but there is also the possibility to provide e.g. validation_split = 0.2
What is the difference?

And after that, the test/prediction set will be taken into account just as model.pred(X_pred, y_pred,..). Right?


texts: Can natural language generation algorithms generate valid words too?

texts:
Natural Language Generation (NLG) is the natural language
  processing task of generating natural language from a machine
  representation system such as a knowledge base or a logical form. —
  Wikipedia

Is NLG about building meaningful sentences, reports, etc.? Can NLG build valid dictionary words as well? For example, without consulting/reading from an English (or any language) dictionary, can an algorithm generate such words?


texts: Training a model for Single Image Super Reoslution

texts:
I'm trying to implement the Attention-based approach for SISR paper. However, during something odd happens. The MAE for the first output of the model is very small. But as the training progresses, the loss is extremely high as shown below.

  1/101 loss: 0.563045883178711

  1/101 loss: 31225273443.6775

  1/101 loss: 21936102.43

  1/101 loss: 5351808.445

  2/101 loss: 22829.5

  2/101 loss: 2118322.8925

Doesn't this mean that the model is somehow better off without the training procedure?

The train function is shown below. I'm quite new to PyTorch, so please let me know if there's some fairly obvious semantic error.

def train(model, optimizer, criterion, train_loader, epochs=100, batch_size=8, steps=400):
    for epoch in range(epochs):
        train_loss = 0.0
        for i, data in enumerate(train_loader):
            x_train_bic, x_train, y_train = data[0], data[1], data[2]
            x_train_bic = x_train_bic.requires_grad_(False)
            x_train = x_train.requires_grad_(False)
            y_train = y_train.requires_grad_(False)
            optimizer.zero_grad()

            output = model(data[0], data[1])
            # loss = ssim(output, data[2], data_range=data[2].max()-data[2].min(), multichannel=True)
            loss = criterion(output, data[2])
            loss.backward()
            optimizer.step()

            train_loss += loss.item()
            if i%100 == 0:
                print("{}/{} loss: {}".format(epoch+1, epochs+1, train_loss/100))
                train_loss = 0.0

The rest of my code can be found here.


texts:
Does the XGBClassifier method utilizes the two regularization terms reg_alpha and reg_lambda, or are they redundant and only utilized in the regression method (i.e., XGBRegressor)?

I think that some hyperparameters in XGBoost could have no effect on specific methods (e.g., scale_pos_weight in XGBRegressor).


texts: Transforming binary data for decision trees

texts:
I have binary columns in my dataset (20) e.g. hot_weather, discount (y or no),  where in each case 1 = yes no = 0. I am using this data on tree based methods.
It is a regression problem and my RMSE is around 1500! Running feature importance reports I get 0 for all but 2 of these binary columns which makes me think they cannot be good for my tree based model.
How else can I transform these binary columns in my data?


texts:
I have been reading about the validation_curve function from scikit learn. When I run this it takes too long. Therefore, I am plotting the results from grid search instead, which seems to be far quicker. However, when I plot using the code from: https://matthewbilyeu.com/blog/2019-02-05/validation-curve-plot-from-gridsearchcv-results.
I get:

Why are the lines so close together? Shouldn't I get a curve instead? More generally, how can I tell if my model is under-fitting or overfitting? How  can I plot the overfit/under-fit areas on my plot.
edit: i am trying to predict if a client will  default on a loan or not so 1 for yes 0 for no. the data are financials. also the data i have used is about 100k samples


texts: What is the right algorithm to detect segmentations of a line chart?

texts:
To be concrete, given 2D numerical data as is shown as line plots below. There are peaks on a background average movement (with small vibrations). We want to find the values of pairs (x1, x2) if those peaks drops down to average; or (x1) only if the line doesn't back to the average.

There are thousands of such 2D data.

What is the right statistic or machine learning algorithm to find x1 and x2 above without plotting?


texts:
Keras expects an array of a certain shape as an input. So far so good.
How does it treat this array in respect to an order? Will the first row be calculated the first and then one after another?
Or is it vice versa? Or abitrary? Is there any determined sequence?

LSTM-layers are probably something different and, exceptionally, easier to understand in this context.


texts: In which cases should Genetic Programming be preferred over Artificial Neural Network trained with Genetic Algorithm

texts:
I am trying to understand Genetic Programming (GP) but I cannot think of any context where GP can be chosen over training Artificial Neural Networks with genetic algorithms.

What problems each of them is suitable for?
In what context we choose one over the other (if it is not personal preference)?


texts: AttributeError: 'numpy.ndarray' object has no attribute 'nan_to_num'

texts:
I'm trying to run a Random Forest model from sklearn but I keep getting an error: ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
I tried following steps in ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
fillna(0) on my pandas dataframe still gave the ValueError.
So I tried working with my numpy array:
val = setTo.ravel().nan_to_num(0)

But I keep getting an error: 'numpy.ndarray' object has no attribute 'nan_to_num'
I'm wondering how I can deal with the nan values if I have ndarray?
Thanks so much!
Update
Thanks so much to @Beniamin H for all the help, as per suggested, I rescalled the data, which I based on https://stackoverflow.com/questions/34771118/sklearn-random-forest-error-on-input and it worked!


texts:
I'm trying to reproduce an algorithm designed in a paper. And everything is going well except one thing:

It says we considered the lengths zero-meaned accelerometer vectors and created a feature for the mean and standard deviation of this value. and I do not understand what is it zero-meaned vectors?

Example dataset:

-0.6946377  12.680544  0.50395286
5.012288    11.264028  0.95342433
4.903325    10.882658  -0.08172209
-0.61291564 18.496431  3.0237172
-1.1849703  12.108489  7.205164
1.3756552   -2.4925237 -6.510526
-0.61291564 10.56939   5.706926
-0.50395286 13.947236  7.0553403

Can any body help me?

I found only this information https://www.quora.com/What-does-it-mean-when-a-vector-is-zero-mean but I'm not sure about it.

Thank you.


texts: Is always converting a input vector into matrix and apply cnn's good idea?

texts:
I know the benefits of using cnns(reduced size weight matrices). Is it a good idea to convert a input vector(which is not a image) into a matrix and apply cnn's. What I understand is that it should not be done because this would enforce some relationship between input vector value which actually doesn't exist.
Am I correct or there is some way we can apply cnn's to reduce computation?
If cnn's can't be applied what could be the method for reducing computation for very high dimensional input


texts: Tier Splitting for Time Series Analysis K means

texts:
I am currently trying to expand upon some results from the paper "Time Warping Clustering for the Forecast and Analysis of COVID-19" by Qixuan Jin out of Cal Tech, because when it was published they only had 4 months of data. The paper is essentially looking for clusters of covid cases at the FIPS code level. In the methods step the last preprocessing step before it goes into how it clustered talks about "tier splitting the data". I have searched EVERY WHERE looking for what this means and I can not find anything other than when working with customer data you can used and RFM score and tier split based on that. However, since covid cases doesn't related to customer data, I don't think this is what it is talking about. My data frame has 3219 columns with each column a different FIPS code; it has 470 rows each row a different day with the number of covid cases per day. I went a head and totaled the number of cases over the 1 and 104 days for each fips code. Here is what it says about tier splitting :
"Tier Splitting: Magnitude can still be used to guide clustering at a higher level. Counties are split into tiers based on their cumulative case counts. Clustering is then performed within each tier. We propose two tier-splitting methods: splitting into four quartiles and splitting based on the custom thresholds of 10, 100, and 1000 cumulative cases."
Am I understanding this correctly that for one method they went ahead got the quartile values for the cumulative cases and then split up the data frame based on the various quartiles and then clustered the various quartile groups. Then for the custom split the split based on 0-10 total cases, 11-100 total cases, 111-1000 total cases, and 1001+ total cases? Or am I misundertanding?
Thank you


texts: Steps on how to use autoencoders to reduce dimensions

texts:
I have a dataset that contains text columns. I have used tf-idf to convert those text columns to numerical columns. I want to reduce the dimension of the dataset since tf-idf creates a multitude of new features/columns.

I am thinking of using autoencoder to reduce the dimension of the data by using the output of the encoded layer, concatenate those output to my dataframe and use it as a training set for the random forest.

My questions are: Do the above steps sound reasonable? In particular, should I train the autoencoder on the whole dataset and then use the output of the encoded layer (after training) as the new columns to feed random forest? [See code below]

I am using Keras so after defining the autoencoder model, I fit the whole dataset and then also predict on the whole dataset using only the encoded layer. And then using those predictions as my updated data.

Here is the code I am using:

def autoencoder(df):
    #return the lower dimensional data to feed directly into train_test_split
    df_copy = df.copy()
    input_dim = Input(shape = (df_copy.shape[1], ))

    # DEFINE THE ENCODER LAYER
    encoded = Dense(int(df_copy.shape[1]/2), activation = 'relu')(input_dim)

    # DEFINE THE DECODER LAYER
    decoded = Dense(df_copy.shape[1], activation = 'sigmoid')(encoded)

    # COMBINE ENCODER AND DECODER INTO AN AUTOENCODER MODEL
    autoencoder = Model(input = input_dim, output = decoded)

    # CONFIGURE AND TRAIN THE AUTOENCODER
    autoencoder.compile(optimizer = 'adadelta', loss = 'mean_squared_error')
    autoencoder.fit(df_copy.values, df_copy.values, nb_epoch = 10, batch_size = int(df_copy.shape[1]/3))

    # THE ENCODER TO EXTRACT THE REDUCED DIMENSION FROM THE ABOVE AUTOENCODER
    encoder = Model(input = input_dim, output = encoded)
    encoded_input = Input(shape = (int(df_copy.shape[1]/2), ))
    encoded_out = encoder.predict(df_copy.values) #Note how I am training on the same data I am predicting on

    return pd.DataFrame(encoded_out)

I am new to autoencoders so any suggestions and help on how to use autoencoders in this context will be appreciated.


texts:
I was trying to encode the string values of the feature 'ProductCategory' into integer values but I got this error. Kindly help.

And I would also like to ask if label-encoding this feature would not force my model to misrepresent the integer values as 0<1<2. Thanks.


texts: You are in charge of investing Chipotle's E-Coli source/s, what methods do you use?

texts:
With the amount of geographic diversity that is popping up over time with Chipotle and the outbreak of e-coli, if you were investigating the source of all these issues, and where it could occur again in the future, where would you begin?

While I initially thought it would be very simple to narrow it down, the most recent CDC articles indicate they have suspicions but nothing for sure yet to indicate how to track it down.

I am relatively new to machine learning algorithms, but I would think a feature classification that would take into account the ingredients involved in positive cases, the sources of the ingredients, and possibly even the machinery used or methods of cooking to help classify the inputs that relate to a positive or negative output would be a good start.


texts: What is and how to calculate quartic interpolation?

texts:
I was reading the gist on the reward function used in OpenAI Five, but I didn't understand the way they calculate health's reward.

This is what they state: Hero health is scored according to a quartic interpolation between 0 (dead) and 1 (full health) to force heavier weight when the agent is near dying.

I tried googling but didn't manage to find an easy enough explanation for me to understand. What exactly is quartic interpolation and how is it calculated?

I understand the normalization part of scoring it between 0 and 1, but how do they increase the weight of certain values?


texts:
In my work in behavioural cloning, I have been asked 'how is your policy expressed?' and I didn't know the answer to this. I was trying to create apply a behavioural cloning algorithm from the context of driving. I wanted to know what is the answer to the question 'how is your policy expressed'?
Am I supposed to give an analytical function? or can we simply say 'mapping from states to actions' as indicated by the demonstrations, which we regard as true?


texts: When should train and test data be merged?

texts:
I am very new to data science and machine learning. I am following courses on datacamp, and then trying to solve problems on kaggle/drivendata.

Very often, I try to use the sklearn.model_selection train_test_split()-method, but because my training (X) and test (y) data is not same shape, I get the error:

ValueError: Found input variables with inconsistent numbers of samples: [913000, 45000]

When I look at other people's solution, it looks like they very often combine training and test data (datasets in this case: train & test), like this:

all_data = train.append(test, sort = False)

Then later, they split this up again into variables X and y, which they can use on the train_test_split method. (You can probably hear I am very new and don't quite understand what's happening yet).

Can anyone explain me, why I need to combine my training and test data, when I already have them in separate .csv files? Maybe share some resources that explain it? Thanks a lot


texts:
Can someone help me understand how to find the values of a confusion matrix?

I know that essentially a confusion matrix looks like this:

True Positive |False Positive
False Negative|True Negative

I've encountered a problem where I have around 1000 cases in my test data.

In the approximate middle of its ROC chart there is a point where the false positive rate is 0.5, the true positive rate is 0.7, and the accuracy is 0.7.

This is my approach to this problem and I just want to verify if I am doing it correctly:

To calculate the count of True Positives:

$$0.7 * 1000\,Cases = 700\,Cases$$

To calculate the count of False Positives:

$$0.5 * 1000\,Cases = 500\,Cases$$

I would assume calculating the True Negatives would be $1000 - 700 = 300$.

Then solving for False Negatives =

$$.7 (Accuracy) = 700 + 300 / 700 + 300 + 500 + FN$$

Can someone confirm if this is the right approach?


texts: How does Pooling Layer in CNN introduce invariance to other transformations besides translation

texts:
Here is a quote from deeplearningbook which I am trying to process. I am not sure what do they mean by this quote, can someone help me understand please?

  Pooling over spatial regions produces invariance to translation, but
  if we poolover the outputs of separately parametrized convolutions,
  the features can learnwhich transformations to become invariant to
  (see ﬁgure 9.9)


texts: Tell me some things a data scientist/statistician needs to do that can't be done in R/Stata/SAS/SPSS

texts: Why do I need to learn to program in python or Java and learn data structures? I can't think of a single thing statisticians/data scientists need to do that isn't built in as a package in R/stata.


texts: Abbreviation in Orange's contingency table

texts: What do the scores ARI and AMI mean in Orange's contingency table?


texts:
I am new to NLP, I have a bunch of raw data that is not tagged at all of medical questions, I need to extract from them what are the health issues stated in those texts.
I was thinking I need to create two custom POS tags for NER:
-the location on the body
-the problem itself
So if someone asked 'my head hurts' it would understand that the location is the head and the problem is that it hurts, but if someone asked 'my skin is red around my abdomen' it would understand that the location is the abdomen and the problem is that the skin is red.
After I extract this data I need to recommend medical articles based on what that user asked.
I have some questions:
1.Am I on the right path?
2.How would you implement it?
3.Do I need a custom pos tag for the location and the health issue or can it be done easier? How would you extract those informations?
4.I guess I have to manually tag questions right?
5.What framework would you use?
6.To create a recommender system I need to extract the same informations from medical articles?
7.How would you create a recommender system?
As I said, I am new to NLP and I didn't decide on the framework yet but the questions are not in english, I have found however on github a WordNet clone and a Named Entity Corpus for my language, so please keep in mind when recommending frameworks.


texts: Correlation coefficient for weighted value pairs

texts:
I want to obtain an appropriate correlation statistic for sparse matrices, in two different cases.
Case 1
A small rectangular matrix (size: 3-20 elements long or wide) with a single value per row, per column. For example:
1  0  0
0  1  0
0  0  1

0  1  0  0
1  0  0  0
0  0  1  0  => if rectangular, may have a col or row with all zeros.

0  1  0  0
0  0  1  0
1  0  0  0
0  0  0  1

My thinking here was to convert the values to paired coordinates. For the first matrix, we would obtain (1,1), (2,2), (3,3). Then pass these values into a linear correlation statistic like Pearson's, which in this case would give a perfect negative correlation.
Is this the best approach?
Case 2
In second case, the values can be real valued, 0≤x≤1.
0.7  0    0
0    0.5  0
0    0    0.8

Higher values that are also more highly correlated should give a higher statistic than otherwise.
Does an appropriate correlation statistic exist for weighted value pairs like these? (1,1,w=0.7), (2,2.w=0.5), (3,3,w=0.8)


texts: Disproving or proving claim that if VCdim is "n" then it is possible that a set of smaller size is not shattered

texts:
Today in the lecture the lecturer said something I found peculiar, and I felt very inconvenient when I heard it:
He claimed, that if the maximal VCdim of some hypothesis class is $n\in\mathbb N$, then it is possible that there is some $i<n$ such that for every subset C of size i the subset C is not shattered.
Is his claim true? I thought that we can take some subset of size $i,\forall i\in [n]$of the set C* which satisfies the condition for the case where $|C|=n$, and they will shatter as well. Am I missing something?


texts:
Here is a question I am struggling with
Training data:
start = "2013-01-01"
end = "2013-01-01"
temp = [24,
        34.38,
        34.36,
        34.74,
        35.26,
        35.23,
        35.29,
        35.64,
        36.02,
        36.1,
        36.98,
        37.01,
        36.75,
        36.01,
        35.66,
        34.72,
        33.9,
        32.62,
        31.51,
        30.73,
        29.5,
        26.94,
        25.47,
        23.84,
        22.55]

Explanation: Above is the training data of temperature of a room for a single day. Predict for next 2 days.
Visualization of above temperature series

There is a same question (with some more info) at this link
My Question:
How to solve this question ? What model should be used for such a question ?
My thoughts:

It looks like time series model. But to use any time series model training data should have trend and seasonality which does not seems to be there (as seen in the above visualization).
One can try linear/non-linear regression models by making on input feature(s) (eg. depicting hour of day starting with 0 to 23) but that would mean the model would predict same set of temperatures every day.
Decision tree with input feature(s) (eg. depicting hour of day starting with 0 to 23) but decision tree is to big model for such small dataset it may over fit.

Thanks in advance.


texts: I am working on a dataset which contains more than 80 features, and thousands of instances. Among those features there are some nominal ones, such as IP SOURCE, IP DESTINATION, Flow ID, which don't have any signification for my machine learning model. My question is, should I remove those features manually, or do I have to replace their values with numerical ones?


texts: How to find a model for a short discrete time-series?

texts:
I'm examining the activity of customers over the years which have about one event per year. This results is many short time-series for which I found the distributions (hit/miss over 4 years sorted by probability in the data):

0000 : 0.31834
0001 : 0.17582
0010 : 0.13605
0100 : 0.13554
1000 : 0.12886
0011 : 0.01717
1100 : 0.01650
0110 : 0.01578
0101 : 0.01220
1010 : 0.01117
1001 : 0.00883
0111 : 0.00571
1110 : 0.00565
1111 : 0.00496
1101 : 0.00384
1011 : 0.00351

Apparently a purely uncorrelated binomial model wouldn't do, but one can observe that if both, the number of 1's and 11's coincide, then the probabilities are approximately equal (apart from a small recency effect of 0001).

Can you see a way to approach such data to deduce a probabilistic model? Basically where I have only a few probability parameters which roughly explain this distribution?


texts: How to deal with broad and narrow variance within classes in classification tasks

texts:
Let's say I'm doing an animal image classification task (it doesn't have to be image classification - this is just my example), and the training and test data is balanced across classes. The classes might be ['gorilla', 'giraffe', 'dog', 'donkey']. Now we all know that there is relatively a lot of variance within the 'dog' class compared to the other three classes.

So, is there any way one would treat this problem vs another problem where all classes have about the same amount of variance (where I might replace 'dog' with 'sheep' for instance)?


texts: What Regresson Algorithm Would Fit This Data?

texts:
I am using regression on lottery numbers, below is the head of my database. I was wondering if anyone had a suggestions on what the best algorithm would be to use for the regression model.

Data Head:

      Date  b1  b2  b3
1984-09-01   9   4   8
1984-09-03   1   5   3
1984-09-04   8   6   6
1984-09-05   5   6   9
1984-09-06   2   1   9


texts: Sequence multi-class classification only learns a few outputs

texts:
I have a multi-event delineation problem, where given a signal, I have an output with the same signal length. Something like 0011002200, where each unique number represents an event (say 0: no event, 1: event A and 2: event B)
To solve it using neural networks, I frame this setting with NxM prediction, where N is the length of the signal and M the number of classes in one-hot encoding fashion. For training, I use the cross-entropy between outputs and labels; as for prediction, I just take the argmax of each time instant.
The problem is that, using three classes as in above, the network usually distinguishes well one event while it labels the rest as 0, completely neglecting the other event. The data itself seems to be well-behaved, and this pattern happens alternately.
Has anyone faced a similar situation and could spare a few tips on how to approach this?


texts: How to use Keras predict_generator() for segmentation output?

texts:
Below is the code I'm using for segmentation mask prediction after using fit_generator(...) on model named m :-

test_datagen = ImageDataGenerator(rescale=1./255)

test_generator.reset()

test_generator =test_datagen.flow_from_directory('result/test', class_mode=None,seed=1, color_mode="grayscale",target_size=(256,256),batch_size=1)

results = m.predict_generator(test_generator,steps=17, verbose=1)

It runs without any errors but how do I visualise the predicted segmentation masks from results?


texts: How do I fix space allocation error on a machine with plenty of space?

texts:
I'd like to run a model on RStudio Server, but I'm getting this error.

  Error: cannot allocate vector of size 57.8 Gb

This is what my data looks like and it has 10,000 rows.

   latitude longitude                 close_date close_price
1  1.501986  86.35068 2014-08-16 22:25:31.925431   1302246.3
2 36.367095 -98.66428 2014-08-05 06:34:00.165876    147504.5
3 36.599284 -97.92470 2014-08-12 23:48:00.887510    137400.6
4 67.994791  64.68859 2014-08-17 05:27:01.404296    -14112.0

This is my model.

library(caret)
training.samples <- data$close_price %>%
  createDataPartition(p = 0.8, list = FALSE)
train.data  <- data[training.samples, ]
test.data <- datatraining.samples, ]

model <- train(
  close_price~., data = train.data, method = "knn",
  trControl = trainControl("cv", number = 1),
  preProcess = c("center","scale"),
  tuneLength = 1
)

My EC2 instance has more than 57 GB available. This is the memory.

             total       used       free     shared    buffers     cached
Mem:      65951628     830424   65121204         64      23908     215484
-/+ buffers/cache:     591032   65360596
Swap:            0          0          0

And it has enough storage space, too. This is the hard drive space.

Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs        32965196      64  32965132   1% /dev
tmpfs           32975812       0  32975812   0% /dev/shm
/dev/xvda1     103079180 6135168  96843764   6% /

And these are details on the machine.

R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Amazon Linux AMI 2018.03


texts: Is it possible to make an machine learning algorithm, that can outperform humans in complex games with current technology?

texts: Would a company like OpenAi, it the "average" person be able to make a bot, using machine learning, that can outperform the best e-sport players in complex games, like PubG, with the technology we have available today? And if so: what kind of models and approach would most likely be used?


texts: What do positive and negative gradient values mean for Convolutional Neural Network?

texts:
As we have the typicall pass of the neural network we make a forawrd pass to predict classes and then we have cost function and based on that we calculate gradients.

I'm wondering what are the meaning of the positive and negatives values of gradient. The specific question that i want to know answer about is why the positive values of gradient maps mean that object was detected there?

Or posting this question with example. Why here the highlighted gradients maps have parts of dog/bird/spider highlighted?

If I m confusing something correct me.


texts: How to use deep learning to add local (e.g. repairing) transformations to images?

texts:
I want to train a neural network that removes scratches from pictures. I chose a GAN architecture with a generator (G) and a discriminator (D) and two sets of images scratchy and non-scratchy, with similar motives. G in my setting uses mainly convolutional and deconvolutional layers with ReLU. As input it uses the scratchy images. The D discriminates between the output of G and the non-scratchy images.

For example the generator would perform the following transformations:

(128, 128, 3) > (128, 128, 128) > (128, 128, 3)

Where the tuples contain (width, height, channels). Input and output need to have the same format.

But in order to get output that have same global structure as the inputs (i.e. streets, houses etc) it seem I have to use filter-sizes and strides of 1 and basically pass the complete pictures through the network.

However, the features I am looking at are rather small and local. It should be sufficient to use filters of up to 20 pixels in size for the convolutions. And then then apply the network to all parts of the input picture.

What would be a good generator architecture for such a task? Would you agree with the structure of the generator or would you expect different designs to perform better on local changes?

Here is the code for the generator that I use implemented in tensorflow.

def generator(x, batch_size, reuse=False):
    with tf.variable_scope('generator') as scope:
        if (reuse):
            tf.get_variable_scope().reuse_variables()

        s = 1
        f = 1

        assert ((WIDTH + f - 1) / s) % 1 == 0

        keep_prob = 0.5

        n_ch1 = 32
        w = init_weights('g_wc1', [f, f, CHANNELS, n_ch1])
        b = init_bias('g_bc1', [n_ch1])
        h = conv2d(x, w, s, b)
        h = bn(h, 'g_bn1')
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)
        h1 = h

        n_ch2 = 128
        w = init_weights('g_wc2', [f, f, n_ch1, n_ch2])
        b = init_bias('g_bc2', [n_ch2])
        h = conv2d(h, w, s, b)
        h = bn(h, "g_bn2")
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)
        h2 = h

        n_ch3 = 256
        w = init_weights('g_wc3', [f, f, n_ch2, n_ch3])
        b = init_bias('g_bc3', [n_ch3])
        h = conv2d(h, w, s, b)
        h = bn(h, "g_bn3")
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)

        output_shape = [batch_size, HEIGHT//s//s, WIDTH//s//s, n_ch2]
        w = init_weights('g_wdc3', [f, f, n_ch2, n_ch3])
        b = init_bias('g_bdc3', [n_ch2])
        h = deconv2d(h, w, s, b, output_shape)
        h = bn(h, "g_bnd3")
        h = tf.nn.relu(h)
        h = h + h2

        output_shape = [batch_size, HEIGHT//s, WIDTH//s, n_ch1]
        w = init_weights('g_wdc2', [f, f, n_ch1, n_ch2])
        b = init_bias('g_bdc2', [n_ch1])
        h = deconv2d(h, w, s, b, output_shape)
        h = bn(h, "g_bnd2")
        h = tf.nn.relu(h)
        h = h + h1

        output_shape = [batch_size, HEIGHT, WIDTH, CHANNELS]
        w = init_weights('g_wdc1', [f, f, CHANNELS, n_ch1])
        b = init_bias('g_bdc1', [CHANNELS])
        h = deconv2d(h, w, s, b, output_shape)

    return tf.nn.sigmoid(h+x)

When you use a stride s>1 you will get an hourglass, where the layers get smaller but deeper. The depth is independently controlled by the n_ch variables.

BTW, I am running this on a Google colab notebook. Its amazing to have such an engine for free and be able to experiment with deep learning! Fantastic!


texts: Use pretrained model and create another sequential model

texts:
I have a pre-trained model, as below

Code for the above model is as below,which is working as expected and I have saved this model separately in ".h5" keras format

data= pd.read_csv('TrainPWM.csv')
data.values[:,0]
actual_pwm = data.values[:,0]
actual_rpm = data.values[:,1]
actual_pwm = np.reshape(actual_pwm,(36954,1,1))/10000
actual_rpm = np.reshape(actual_rpm,(36954,1,1))/10000
x_train,x_test,y_train,y_test=train_test_split(actual_pwm,actual_rpm,test_size=0.3,random_state=50)
model = models.Sequential()
model.add(layers.LSTM(units=1,return_sequences=True))
model.add(layers.LSTM(units=20,return_sequences=True))
model.add(layers.LSTM(units=20,return_sequences=True))
model.add(layers.LSTM(units=1,return_sequences=True))
model.add(layers.Dense(units=1))
model.compile(optimizer='adam',
          loss='mean_absolute_error',
          metrics=['accuracy'])
model.fit(x_train,y_train,epochs=1)

Now I want to create another sequential model, which must be trained with the above pretrained model(rnn2), below some explanation

I must create a sequential training model based on this where,When I give input x, first rnn1 must train itself based on the output y using rnn2

I have trouble visualizing the way to do this


texts: Is there are way to impute missing values by clustering, regression and stochastic regression

texts:
I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the best imputation method for an small dataset (Iris in this case). I had to delibrately create NaN values since Iris set has none.

My code for KNN imputation:

 import pandas as pd
 import numpy as np
 import random
 from fancyimpute import KNN

 data = pd.read_csv("D:/Iris_classification/train.csv")
 mat = data.iloc[:,:4].as_matrix()

 prop = int(mat.size * 0.5) #Set the % of values to be replaced
 i = [random.choice(range(mat.shape[0])) for _ in range(prop)] #Randomly choose indices of
 j = [random.choice(range(mat.shape[1])) for _ in range(prop)] #the numpy array

 mat[i,j] = np.NaN #replace values with NaN

 mat_filled = pd.DataFrame(KNN(3).complete(mat)) #converted the array back to df

 data_col = data.drop('species', axis = 1)
 mat_filled.columns = data_col.columns  #added column names that went missing in mat_filled

Is there a similar way to impute with the other 3 methods?


texts: loss function for multi-label segmentation with class inbalance

texts:
In order to use a binary segmentation loss function in a multi label problem, I would like to permute the batch axis with the channel axis in the loss computation in order to compute the loss by channel rather than batches to fight a class inbalance problem. (for example, if a class A is very predominant on the others, I could obtain a good loss result by attributing class A to each pixels / voxels.)
Here is an example using the dice loss on 3d images:

def dice_coef(y_true, y_pred, smooth=1e-10):

    # permute channels with batches
    # eg:  [num_batches, height, width, depth, num_channels] --becomes--> [num_channels, height, width, depth, num_batches]
    # in order to perform the computation per channel rather than batches
    y_true = K.permute_dimensions(y_true, (4,1,2,3,0))
    y_pred = K.permute_dimensions(y_pred, (4,1,2,3,0))

    y_true_f = K.batch_flatten(y_true)
    y_pred_f = K.batch_flatten(y_pred)

    intersection = K.sum(y_true_f * y_pred_f)
    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)

def dice_coef_loss(y_true, y_pred):
    return 1 - dice_coef(y_true, y_pred)

is my raisonning correct ?


texts: Is the output of this loss function considered as a percentage or something else?

texts:
I am currently working on fine-tuning the PEGASUS model for abstractive summarization.
The script for fine-tuning the class Trainer from Transformers is imported.
The output of training loss is printed and I don't understand is it a percentage or does it represent something else.

The link for the fine-tuning script: https://gist.github.com/jiahao87/50cec29725824da7ff6dd9314b53c4b3
The link for the Trainer class documentation: https://huggingface.co/transformers/main_classes/trainer.html#transformers.Trainer.compute_loss
The Code to compute the loss:
def compute_loss(self, model, inputs, return_outputs=False):
"""
How the loss is computed by Trainer. By default, all models return the loss in the first element.
    Subclass and override for custom behavior.
    """
    if self.label_smoother is not None and "labels" in inputs:
        labels = inputs.pop("labels")
    else:
        labels = None
    outputs = model(**inputs)
    # Save past state if it exists
    # TODO: this needs to be fixed and made cleaner later.
    if self.args.past_index >= 0:
        self._past = outputs[self.args.past_index]

    if labels is not None:
        loss = self.label_smoother(outputs, labels)
    else:
        # We don't use .loss here since the model may return tuples instead of ModelOutput.
        loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]

    return (loss, outputs) if return_outputs else loss


texts: Is it normal for F1 scores to be lower on a binary classification task as compared to a 3-class classification task?

texts: I am trying to understand if the F1 scores are higher for a binary classification problem than for a multiclass classification problem.


texts: Scikit Learn Missing Data - Categorical values

texts:
I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier)
Label  Feat1 Feat2 Feat3 Feat4
 1      1     1      2     2
 2      3     1      2     2
 3      4     3      3     1
 4      4     1      2     4

I'm using scikit learn, and am considering using Naive Bayes or a Decision Tree. The classifier needs to be able to deal with missing features, and I read on scikit learn's page that Decision Tree does not support missing values.
What I am looking for is advice as to how to approach missing categorical values when using scikit learn. Also, any links to academic papers addressing this would be appreciated.


texts:
Is it possible to connect kobotoolbox to tableau in any way?
Any kind of method would be much appreciated.


texts: How important is outcome variable scaling in SVM regression?

texts: Should I scale outcome variable for SVM regression? What is the magnitude of impact of outcome variable scaling in SVM regression?


texts: Are CNNs insensitive to rotations and shifts in images?

texts:
Can CNNs predict well if they are trained on canonical-like images but tested on a version of images that are little bit shifted?

I tried it using mnist dataset and found the contrary. The accuracy of the test set that was shifted was very low as compared to MLPs.


texts:
For example

Shoes priced at \$10, 30 of 1000 customers buy for revenue of \$300
Shoes priced at \$20, 25 of 1000 customers buy for revenue of \$500

And then now I have the problem of determining how certain I am the second pricing is yields higher returns

My question is: How can I find out X so that I'm X% confident that the \$20 price yields higher returns than the \$10?


texts: SAGAN - what is the correct architecture?

texts:
Hi, in the original paper the following scheme of the self-attention appears:
https://arxiv.org/pdf/1805.08318.pdf

In a later overview:
https://arxiv.org/pdf/1906.01529.pdf
this scheme appears:

referring the original paper.
My understanding more correlates with the second paper scheme, as:

Where there is two dot-product operations and three hidden parametric matrices:
$$W_k, W_v, W_q$$
which corresponds to $W_f, W_g, W_h$ without $W_v$ as it in the original paper explanation, which is as following:

Is this a mistake in the original paper ?


texts: Specifying priors in rstanarm for hierarchical model

texts:
We are given the model $$
\begin{align*}
y_{ij} & \sim \mathsf{Normal}(\alpha_j + \beta x_i, \sigma^2)\\
\alpha_j & \sim \mathsf{Normal}(\gamma_0 + \gamma_1 u_j, \tau^2)
\end{align*}
$$ with priors $$ \begin{align*}
\beta, \gamma_0, \gamma_1 & \sim \mathsf{Normal}(0, 0.5^2)\\
\sigma, \tau & \sim \mathsf{Normal}_+(0, 0.25^2)
\end{align*}
$$  where we have $j$ counties and $i$ observations. I'm having trouble specifying all of my priors for rstanarm. What I have so far is

library(rstan)
library(rstanarm)
data(radon)

fit_lmer <- stan_glmer(log_radon ~ floor + (1 | county),
                       data = radon,
                       prior_intercept = normal(0, 0.5),
                       prior_aux = normal(0, 0.25))

I'm just not entirely sure I properly specified everything.


texts:
I trained a regression TPOT algorithm on Google Colab, where the output of the TPOT process is some boiler plate Python code as shown below.
import numpy as np
import pandas as pd
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline, make_union
from tpot.builtins import StackingEstimator
from tpot.export_utils import set_param_recursive

# NOTE: Make sure that the outcome column is labeled 'target' in the data file
tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)
features = tpot_data.drop('target', axis=1)
training_features, testing_features, training_target, testing_target = \
            train_test_split(features, tpot_data['target'], random_state=1)

# Average CV score on the training set was: -4.881434802676966
exported_pipeline = make_pipeline(
    StackingEstimator(estimator=ExtraTreesRegressor(bootstrap=False, max_features=0.9000000000000001, min_samples_leaf=1, min_samples_split=20, n_estimators=100)),
    ExtraTreesRegressor(bootstrap=True, max_features=0.9000000000000001, min_samples_leaf=6, min_samples_split=13, n_estimators=100)
)
# Fix random state for all the steps in exported pipeline
set_param_recursive(exported_pipeline.steps, 'random_state', 1)

exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)

Would anyone know what is the Sklearn pipeline process like how does this work? When I butter up the boiler plate code and run it with my data set in IPython I can see this output from the pipeline process, what is this all doing?
Pipeline(steps=[('stackingestimator-1',
                 StackingEstimator(estimator=ExtraTreesRegressor(max_features=0.6500000000000001,
                                                                 min_samples_leaf=19,
                                                                 min_samples_split=14,
                                                                 random_state=1))),
                ('maxabsscaler', MaxAbsScaler()),
                ('stackingestimator-2',
                 StackingEstimator(estimator=ExtraTreesRegressor(max_features=0.4,
                                                                 min_samples_leaf=3,
                                                                 min_samples_split=7,
                                                                 random_state=1))),
                ('adaboostregressor',
                 AdaBoostRegressor(learning_rate=0.001, loss='exponential',
                                   n_estimators=100, random_state=1))])

The results look good just curious about how the pipelines processes work, any tips or links to tutorials greatly appreciated. I thought this machinelearningmastery tutorial is also somewhat useful for anyone interested in learning more about TPOT.


texts: Spatial Transformer Networks vs Deformable Convolutions

texts:
As I understand STN as described by the the deepmind paper https://arxiv.org/abs/1506.02025

allow a neural network to learn how to perform spatial transformations on the input image in order to enhance the geometric invariance of the model.

whereas Deformable Convolutions https://arxiv.org/abs/1703.06211

add 2D offsets to the regular grid sampling locations in the standard convolution.

From the paper:

...deformable convolution does
not adopt a global parametric transformation and feature
warping. Instead, it samples the feature map in a local
and dense manner. To generate new feature maps, it has
a weighted summation step, which is absent in STN.

To me STNs augment the input image whereas DC augment the kernel shape. Results should be similar/same in terms of field of view of the network? Please correct me if mistaken.


texts: Data Science Noob - Customer Scoring based on conversion probability

texts:
I work at a University and have a project to be able to score applicants based on their likelihood to enroll (convert), using their answers to application form questions.

The applications contain name, DOB, date & time of application, country, gender, course selected, English proficiency, funding availability and some other similar fields. There are also various free-text fields, but I think these will complicate things too much to begin with.

My initial thought is to use a regression model to do this, using R. But I am a complete noob - I studied regression at uni 10 years ago....

I have had a search around and I think once I know I'm on the right path I will be able to figure the process out but I am unsure where to begin, and do not want to start by going down the wrong avenue. My main concerns are:

Is a regression model the correct approach? If not what is?
Are categorical fields a problem - as opposed to continuous fields?
There is additional information which is only available for some applicants - can this be included, or do we need to use the same information for all applicants?


texts:
I wanted to see if I can simply set new weights for gensim's Word2Vec without training. I get the 20 News Group data set from scikit-learn (from sklearn.datasets import fetch_20newsgroups) and trained an instance of Word2Vec on it:

model_w2v = models.Word2Vec(sg = 1, size=300)
model_w2v.build_vocab(all_tokens)
model_w2v.train(all_tokens, total_examples=model_w2v.corpus_count, epochs = 30)

Here all_tokens is the tokenized data set.
Then I created a new instance of Word2Vec without training

model_w2v_new = models.Word2Vec(sg = 1, size=300)
model_w2v_new.build_vocab(all_tokens)

and set the embeddings of the new Word2Vec equal to the first one

model_w2v_new.wv.vectors = model_w2v.wv.vectors

Most of the functions work as expected, e.g.

model_w2v.wv.similarity( w1='religion', w2 = 'religions')
> 0.4796233
model_w2v_new.wv.similarity( w1='religion', w2 = 'religions')
> 0.4796233

and

model_w2v.wv.words_closer_than(w1='religion', w2 = 'judaism')
> ['religions']
model_w2v_new.wv.words_closer_than(w1='religion', w2 = 'judaism')
> ['religions']

and

entities_list = list(model_w2v.wv.vocab.keys()).remove('religion')

model_w2v.wv.most_similar_to_given(entity1='religion',entities_list = entities_list)
> 'religions'
model_w2v_new.wv.most_similar_to_given(entity1='religion',entities_list = entities_list)
> 'religions'

However, most_similar doesn't work:

model_w2v.wv.most_similar(positive=['religion'], topn=3)
[('religions', 0.4796232581138611),
 ('judaism', 0.4426296651363373),
 ('theists', 0.43141329288482666)]

model_w2v_new.wv.most_similar(positive=['religion'], topn=3)
>[('roderick', 0.22643062472343445),
> ('nci', 0.21744996309280396),
> ('soviet', 0.20012077689170837)]

What am I missing?


texts: In which predictive tasks are the ARIMA model useful?

texts:
I'm working on a project where I have to predict the total quantities sold at the ITEM/DAY level.

As a model, I thought of using an ARIMA (I'm using R), but I wanted to get a confirmation of whether it is a good choice or not. If not, can I get the contexts where the ARIMA model is the most useful?


texts: How to manipulate this column of less than/greater than?

texts:
I have a dataset in which one column is as given in the picture.
What will be the best way to handle such a column?


texts: How to check for overfitting with SVM and Iris Data?

texts:
I am using machine learning predictions for the sample iris dataset. For instance, I am using the support vector machines (SVMs) from scikit-learn in order to predict the accuracy. However, it returns an accuracy of 1.0. Here is the code I am using:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=51)
svm_model = svm.SVC(kernel='linear', C=1, gamma='auto')
svm_model.fit(X_train,y_train)
predictions = svm_model.predict(X_test)
accuracy_score(predictions, y_test)

How to find out or to measure if this over-fitting or if the model is so good? I assume that its not over-fitting but what are the best ways to validate this?


texts: Should I apply PCA on the entire dataset or just the nominal values?

texts:
I have a data-set with 14~ attributes, roughly half of them nominal. I've used a binary vectorizer to convert these values to a number of attributes. The number of attributes, naturally, ballooned up; I'm sitting on roughly 50 at the moment. I've looked at using PCA to reduce this number.

As far as I can understand the things I've been reading, I should exclude my target variable from the analysis. But I'm not sure if I should perform PCA on the whole remaining data set (Including the values that were already numerical, like 'age') or just on the values I converted from Nominal to Numerical and then re-add those to the already numeric values.

To clarify, I've already converted this data-set from nominal to binary, and I'm not sure if I should apply PCA to just the binary columns generated, or the entire thing.


texts: Confused about description of YearPrediction Dataset

texts:
https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD

According to the description given in the above link,
the Attribute information specifies "average and covariance over all 'segments', each segment being described by a 12-dimensional timbre vector". So the covariance matrix should have 12*12 = 144 elements. But why is the number of timbre covariance features only 78 ?


texts: Back-Translation model for German and English

texts:
Do you know of any pre-trained models for back translation between German and English? I am aware that there are ways to include a monolingual corpus into the training of a machine translation model (often referred to as back translation). I am looking for a model to allow me to generate new, semantically similar German texts by translating them into, say, English and then back. Ultimately, to increase my training corpus with examples that are similar to the existing ones, but with slight variation in sentence structure and word choice.

Obviously, I could just stick together two separate models (one for translating the original texts into English and one for translating the English version back to the original language). However, it would be great if you knew of another that I could use.

I'd very much appreciate any kind of help!


texts: Methodologies for predicting missing data

texts:
I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset.

For example: I have a dataset which contains information about a person. This can be gender, weight, age, height etc. Let's assume there are information missing of age and height for a specific person. How can this information be predicted based on the information I have in my dataset?

I have read about PCA-methods, but I would like to get an overview of pros and cons of methodologies and also an overview of recent research (good working algorithms for the given dataset/where to start reading for developing an algorithm and solving the given problem).


texts: I am wanting to use a character level CNN to classify a heap of documents based on the century which they are from. I am having difficulties finding any resources on how to do character level classification in WEKA and was wondering whether anyone has any experience with this. I am using the wekadeeplearning4j package but I can't seem to figure out how their CNN text iterator can work with characters instead of embeddings. I would greatly appreciate any help.