texts:
I am trying to predict the overall age of an opportunity (creation date - closing date) this is my response variable

lets say an opportunity  passes through 3 stages
to close

For example: Opp x stayed in

stage 1 : 30 days
stage 2 : 10 days
stage 3:  20 days

At stage 3 I might close it same date or wait some time

so if I waited some time to close, it will be createdon: 22/11/2018 & closedon:9/2/2019

There is opp y , where i close it in same date of stage 3, so createdon:22/11/2018 and closedate: 21/1/2019

Summary

+---------+--------+--------+--------+--------+
| OppName | oppAge | stage1 | stage2 | stage3 |
+---------+--------+--------+--------+--------+
| x       |     79 |     30 |     10 |     20 |
| y       |     60 |     30 |     10 |     20 |
+---------+--------+--------+--------+--------+

my question is :

Can I include stage1,2,3 as my independent variables to create a regression model?
They seem to nearly make the model ideal, so is it better to include maybe only stage 1? without 2 &3

Work Done ,Added Edits

I transformed stages to a categorical nature, for example <30 days to 1 month
30-60 day transformed it to 1-2months and so on

 +-----------+
|  stage1   |
+-----------+
| <1month   |
| 1~2months |
| 6~7months |
+-----------+

Then I did one-hot encode to the stages like stage 1
Then I stopped ,wasn't sure whether to include everything or what?


texts: What is the dimension of the filters if the input image has only one channel?

texts:
I have a grayscale image with dimension HxWx1 (one channel). To build a CNN using the grayscale image as an input image, what is the dimension of the filters?

I read from some websites, it says that if the input image has only one channel, then the filter must be NxNx1. However, some other websites mentioned differently. It says even if the input image has one channel, it is fine to use filters with dimension NxNx3 by simply converting the grayscale image (HxWx1) to a RGB image (HxWx3).

Therefore, my question is: with input image that has only 1 channel, must the dimension of the filter be NxNx1 or NxNx3?

(I self-study this, so I might get the idea and understanding wrong. Help me, please.)

Note;
H: height, W: width, N: height and width of the filter


texts: Why dont we use 2d cnn filters for Nlp tasks?

texts: CNNs are used in NLP for various tasks. But I cannot find a clear understanding of why do we only use 1d filters in these networks?


texts:
I'm currently reading Mitchell's book for Machine Learning, and he just started gradient descent. There's one part that's really confusing me.

At one point, he gives this equation for the error of a perceptron over a set of training examples.

$$E(\vec{w})\equiv \frac12 \sum_{d \in D}(t_d-o_d)^2$$

$O_d$ is the actual output of $ \vec{W} \cdot \vec{X}$, where $ \vec{X}$ is the input vector and $\vec{W}$ is the weights vector.

$t_d$ is the target output, what we want to get.

The sum over all the $D$ means we sum over every single  $\vec{X}$ we can input.

Okay, so far so good, I understand that.

However, he then gives this example:

But that is just not true!!!! That equation for the error does NOT give us a single minimum!!!

According to his previous rule, if we're considering the error for a single weight vector and a single training vector, the equation for the error would be:

$$E(\vec{w}) = \frac{1}{2}  (t_d - (w_0  x_0 + w_1  x_1))^2$$

Which has an infinite number of minimums!!! Every time $(w_0 x_0 + w_1 x_1) = t_d$

I graphed it here to show you:

In that picture, $x$ and $y$ are the two rows of the weight vector $\vec{w}$.

Please help! I've been confused about this for the last three hours!

Thanks


texts: How to execute a git project with different input data?

texts:
I would like to run this project and receive result from this project

However I have problem to execute the step

  mailcorpus.json created by executing the sql script
  data/get_mail_corpus.sql on the Apache database

because this is the sql query

  https://github.com/collab-uniba/Personality-Recognition-in-SD/blob/master/data/get_mail_corpus.sql

SET group_concat_max_len=18446744073709547520;

SELECT

CONCAT(

'[',

GROUP_CONCAT(

JSON_OBJECT(

'id',messages.message_id,

'mailing_list_url',messages.mailing_list_url,

'type_of_recipient',messages_people.type_of_recipient,

'email_address',messages_people.email_address,

'message_body',messages.message_body,

'is_response_of',messages.is_response_of

) SEPARATOR ',\r'),

']') AS list

FROM messages

LEFT JOIN messages_people

ON messages.message_id = messages_people.message_id

AND messages.mailing_list_url = messages_people.mailing_list_url

WHERE email_address IN ('dr@gorsvet.kz', 'mathews@uk2.net', 'd.jentsch@fu-berlin.de', 'atul.soman@microfocus.com', 'kkrugler_lists@transpac.com', 'hcorg@minions.org.pl', 'bruno.ronchetti@mac.com', 'christophe.thiebaud@sap.com', 'kc.baltz@copart.com', 'havanki4j@gmail.com', 'joseph.obernberger@ngc.com', 'goran@roseen.se', 'hboutemy@apache.org', 'johann@gyger.name', 'fredrik@jonson.org', 'jose-marcio.martins@mines-paristech.fr', 'dmytro.kostiuchenko@gmail.com', 'jochen.wiedmann@softwareag.com', 'guyskk@qq.com', 'alejandroscandroli@gmail.com', '1983-01-06@gmx.net', 'jstrayer@proofpoint.com', 'daniel@degu.cl', 'hontvari@flyordie.com', 'david@andl.org', 'gholmes@pinsightmedia.com', 'dennis.geurts@luminis.eu', 'carlspring@gmail.com', 'jozef.koval@protonmail.ch', 'gabriele.ctn@gmail.com', 'ashish.disawal@evivehealth.com', 'aarslan2@anadolu.edu.tr', 'kshukla@yahoo-inc.com', 'alex.huang@gmail.com', 'markus@openindex.io', 'marohn@sipgate.de', 'ea@apache.org', 'krzysztof.szalast@gmail.com', 'kfoskey@tpg.com.au', 'krauss@in.tum.de', 'kenhans@hotmail.co.uk', 'jokin.c@odeian.com', 'eitch@eitchnet.ch', 'm.kristian@web.de', 'bob.sandiford@sirsidynix.com', 'asandstrom2@eastlink.ca', 'kelly@apache.org', 'antony.blakey@linkuistics.com.au', 'ashawki@hotmail.com', 'h.klocker-mark@tsn.at');

but the database is not available to make the query. How is it possible to use my own input and take the rinal results?


texts: In the context of natural language processing, can anyone give a concrete example of True Positive, True Negative, False Positive, False Negative?

texts:
Google post gives a interesting explanation about True Positive, True Negative, False Positive, False Negative

  True Positive (TP): Reality: A wolf threatened. Shepherd said: "Wolf."
  Outcome: Shepherd is a hero.

  True Negative (TN): Reality: No wolf threatened. Shepherd said: "No
  wolf." Outcome: Everyone is fine.

  False Positive (FP): Reality: No wolf threatened. Shepherd said:
  "Wolf." Outcome: Villagers are angry at shepherd for waking them up.

  False Negative (FN): Reality: A wolf threatened. Shepherd said: "No
  wolf." Outcome: The wolf ate all the sheep.

In the context of CV, the classifier predicts if an image contains cat

True Positive (TP):
Reality: an image contains cat.
classifier predicts: cat.

True Negative (TN):
Reality: an image does not contains cat.
classifier predicts: no cat.

False Positive (FP):
Reality: an image does not contains cat.
classifier predicts: cat.

False Negative (FN):
Reality: an image contains cat.
classifier predicts: no cat.

Can anyone gives a concrete example of TP、TN、FP、FN like above, in the context of natural language processing?


texts:
My data are sequences of real numbers $a_0,a_1,...,a_{n-1}$. The length of a sequence is fixed and equals $n$. Each sequence is mapped to a real number $y$ and I want to predict $y$ given the sequence.
The arrangement of the elements within a sequence is important. However, the sequences are circular, meaning that $a_0$ is not the first element, and $a_{n-1}$ is not the last one. The sequence $a_0,a_1,...,a_{n-1}$ is indistinguishable from the sequence $a_k, a_{k+1}, ..., a_{n-1}, a_0, ..., a_{k-1}$: they are mapped to the same $y$. Moreover, one can circle them in the opposite direction, thus $a_{n-1}, a_{n-2},..., a_0$ maps to the same answer.
I know that recurrent neural networks (RNN) are used for sequences where it is important that the inputs are fed into the network in a specific order. How to make such a network invariant to circular transformations and the change of direction described above?
I don't insist on RNN. Are there any supervised learning algorithms that work with such sequences?


texts: Python Neural Network Library - With Dynamic Topologies

texts:
I know there are questions asking for "best neural network library in Python" but I haven't found any (besides the old deprecated ones like PyBrain) which allow algorithms like NEAT and MultiNEAT and simple non-gradient/differentiation optimization algorithms like PSO to be applied. I'm working on a project involving NEAT in PyBrain but PyBrain is old and as far as I can tell very very slow.

Are there any more recent/in-progress projects in Python which allow custom neural network topologies (for example, Tensorflow and Keras seem to only allow fully-connected recurrent networks, I can't think of a way to, for example, modify those networks so they match up with a network generated by NEAT).


texts: Demand Forecasting for Multiple Products Across Thousands of Stores

texts:
I'm currently working on a demand forecasting task, with data on tens of thousands of products across a couple thousand stores. More specifically,I have 3 years' worth of daily sales data per product in each store, and my goal is to forecast the future sales of each item in each store, one day ahead; then two days ahead, etc.

So far I've considered breaking down each product-store pair into a single time series, and doing a forecast for each time series as was done in Neal Wagner's paper, Intelligent techniques for forecasting multiple time series in real-world systems. In other words, I will use only the historical information of a particular store's sales of the product to forecast the future sales of that product in that store.

However, I've been browsing Kaggle and competitions like Corporación Favorita Grocery Sales Forecasting suggest a different approach, which is to use the information from all stores and all products to predict future sales. As I understand it, historical sales information of all products in all stores are dumped into the training set, from which the model will learn to forecasts future sales. It's very different from traditional time series methods, but apparently, based on the results of the competition, it works.

The latter method seems promising and more robust. However, there's the problem of having to process hundreds of millions of data points.

Which method is more appropriate for my task? For those who have worked on similar problems, which methodology would you recommend?


texts:
I want to train several CNN architectures with Google Colab (GPU), Keras and Tensorflow.
Since the trained models are not reproducible due to GPU support, I would like to train the models several times and determine the mean and the standard deviation of the results.
I'm totally unsure if I should at least try to make the models minimal reproducible? For example with the following code at the beginning of the program:

import numpy
import tensorflow as tf
import random as rn
import os

os.environ['PYTHONHASHSEED']='0'
np.random.seed(1)
rn.seed(1)
tf.set_random_seed(1)

from keras import backend as K
if 'tensorflow' == K.backend():
   import tensorflow as tf
   tf.set_random_seed(1)
   from keras.backend.tensorflow_backend import set_session
   config = tf.ConfigProto()
   config.gpu_options.allow_growth = True
   config.gpu_options.visible_device_list = "0"
   set_session(tf.Session(config=config)

I don't know if this makes sense. Would it be better if I did not use seeds at all?
What do you think?


texts: Predicting and clustering at the same time?

texts:
I want to build a segmentation to substitute the existing RFM segmentation which is a basic segmentation based on the Recency, Frequency and Monetary values.

The new segmentation will be used for two purposes:

Rank customers per value
Have homogeneous groups of customers, so there should be an interpretation and meaning for each group.

I think this shouldn't be an unsupervised learning, but a mix of both supervised and unsupervised learning.

So What I did is that I defined a target variable that takes 1 if the customer is active in the next period and 0 if not. then I used a decision tree to predict this value, so that the leaves represent the clusters. the percentage of active customers in each group is used to rank these clusters.

Do you think that's a good solution and do you have other ideas ?

Thanks in advance


texts: Visualizing ConvNet filters using my own fine-tuned network resulting in a "NoneType" when running: K.gradients(loss, model.input)[0]

texts:
I have a fine-tuned network that I created which uses vgg16 as it's base.  I am following section 5.4.2 Visualizing CovNet Filters in Deep Learning With Python (which is very similar to the guide on the Keras blog to visualize convnet filters here).

The guide simply uses the vgg16 network.  My fine tuned model uses the vgg16 model as the base, for example:

model.summary()

Layer (type)                 Output Shape              Param #
======================================================================= vgg16 (Model)                (None, 4, 4, 512)         14714688
________________________________________________________________________ flatten_1 (Flatten)          (None, 8192)              0
________________________________________________________________________ dense_7 (Dense)              (None, 256)               2097408
________________________________________________________________________ dense_8 (Dense)              (None, 3)                 771
======================================================================== Total params: 16,812,867 Trainable params: 16,812,867 Non-trainable params: 0

I'm running into an issue when I run this line:
grads = K.gradients(loss, model.input)[0] where when I use my fine tuned network I get a result that's a "NoneType"

Here is the code from the guide:

> from keras.applications import VGG16 from keras import backend as K
>
> model = VGG16(weights='imagenet',
>               include_top=False)
>
> layer_name = 'block3_conv1' filter_index = 0
>
> layer_output = model.get_layer(layer_name).output loss =
> K.mean(layer_output[:, :, :, filter_index])
>
> grads = K.gradients(loss, model.input)[0]

To reproduce the on my fine tuned model, I've used the exact same code, except I obviously changed the model that I imported:

model = keras.models.load_model(trained_models_dir + 'fine_tuned_model.h5')

...and I also had to index into the nested Model object (my first layer is a Model object as is shown above) to get the 'block2_con1' layer:

my_Model_object = 'vgg16'
layer_name = 'block3_conv1'
filter_index = 0

layer_output =
model.get_layer(my_Model_object).get_layer(layer_name).output

any idea why running grads = K.gradients(loss, model.input)[0] on my fine tuned network would result in a "NoneType"??

Thanks.


texts:
Let's say I have a data set like the following:

file   group_a_co_1    group_a_co_2    group_b_co_1    group_b_co_2
file_1  0.8 0.2 0.3 0.7
file_2  0.1 0.9 0.2 0.8
file_3  0.5 0.5 0.7 0.3
...

I wonder, whether there are ways/tricks to tell the model about the group information here: since group_a_co_1 + group_a_co_2 = 1 and the same goes for group_b. Somehow I figure if I expose the group information, the performance of my model will improve.


texts: The majority of people use S3. However, Google Drive seems a promising alternative solution for storing large amounts of data. Are there specific reasons why one is better than the other?


texts: Impute missing values in feature column on the basis of Target column

texts:
I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates if a claim was filed or not. To begin with, I imputed missing values with Mean value of feature column and then with Median value. My classification models (Logistic Regression and SVC) gave almost the same prediction metrics (accuracy, precision, recall and F1-Score) for these two imputations strategies.
The third strategy that I tried involved imputing the missing values with the Mean value of each of the two categories of the target variable.
dataframe['Feature'] = dataframe['Feature'].fillna(dataframe.groupby('Target Feature')['Feature'].transform('mean'))

After this step, the prediction metrics of my models increased considerably (5-6%). Now I am wondering if this step is correct or if the model is overfitting the data. Below are the metrics details:
Strategy: Replace Missing Values with Mean Value
Model: Logistic Regression
Accuracy:88.47%
Precision:86.87%
Recall:88.47%
F1 Score:87.67%
Model: Support Vector Classifier
Accuracy:90.66%
Precision:91.46%
Recall:88.94%
F1 Score:90.18%
Strategy: Replace Missing Values with Mean Value of the target feature categories
Model: Logistic Regression
Accuracy:96.44%
Precision:95.06%
Recall:97.69%
F1 Score:96.36%
Model: Support Vector Classifier
Accuracy:96.88%
Precision:97.20%
Recall:96.31%
F1 Score:96.75%


texts: Sliding window leads to overfitting in LSTM?

texts:
Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs?
For a simplified example, assume that we have to predict the sequence of characters:
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Is it bad (or better?) if I keep training my LSTM with the following minibatches:
A B C D E F G H I J K L M N, backprop, erase the cell

B C D E F G H I J K L M N O, backprop, erase the cell

 .... and so on, shifting by 1 every time?

Previously, I always trained it as:
A B C D E F G H I J K L M N,  backprop, erase the cell

O P Q R S T U V W X Y Z,  backprop, erase the cell

Instead of shifting by one, would it be better to slide the window by 2 entries instead, etc? What would that mean (in terms of precision/overfitting)?

Also, if I were to do the sliding-window approach in a Feed-forward network, would it result in overfitting? I would assume yes, because the network is exposed to the same information regions for a very long time. For example, it is exposed to E F G H I J K for a long time.

Edit:
Please remember that cell state is erased between training batches, so the LSTM will have a "hammer to head" at these times. It's unable to remember what was before OPQRSTUVWXYZ. This means that the LSTM is unable to ever learn that "O" follows the "M".
So, I thought (thus my entire question), why not to give it intermediate (overlapping) batch in between...and in that case why not use multiple overlapping minibatches - to me this would provide a smoother training? Ultimately, that would mean a sliding window for an LSTM.

Some useful info I've found after answer was accepted:
from here

The first word of the English translation is probably highly correlated with the first word of the source sentence. But that means decoder has to consider information from 50 steps ago, and that information needs to be somehow encoded in the vector. Recurrent Neural Networks are known to have problems dealing with such long-range dependencies. In theory, architectures like LSTMs should be able to deal with this, but in practice long-range dependencies are still problematic.
For example, researchers have found that reversing the source sequence
(feeding it backwards into the encoder) produces significantly better
results because it shortens the path from the decoder to the relevant
parts of the encoder. Similarly, feeding an input sequence twice also
seems to help a network to better memorize things. For example, if one training example is "John went home", you would give "John went home John went home" to the network as one input.

Edit after accepting the answer:
Several months after, I am more inclined to use the sliding window approach, as it uses the data better. But in that case you probably don't want to train BCDEFGHIJKLMNO right after ABCDEFGHIJKLMNO.  Instead, shuffle your examples, to gradually and uniformly "brush-in" all of the information into your LSTM.  Give it HIJKLMNOPQRSTU after ABCDEFGHIJKLMNO etc.  That's directly related to Catastrophic Forgetting. As always, monitor the Validation and Test set closely, and stop as soon as you see their errors steadily increasing
Also, the "hammer to head" issue can be improved, by using Synthetic Gradients. See its benefit here: (linked answer discusses its benefit of long sequences) https://datascience.stackexchange.com/a/32425/43077


texts: Does Keras allow using independant classifier

texts:
I have spam data set classified into 0:ham or 1:spam. I created the Embedding layer with Keras and then I used Conv1D following with other layers. My question is about adding a classifier like Random Forest or SVM or anything else using Scikit-learn library after the Conv1D layer, so that the output of Conv1D can become as input to Random forest? for example:
model.add(Conv1D(filters=100,kernal_size=2,activation="relu")
model.add(RandomForestClassifier(max_depth=2, random_state=0))
...
model.compile
model.fit

I know the second line gives syntax error but I tried to explain my idea. Thank you.


texts: GloVe vector representation homomorphism question

texts:
In the paper GloVe: Global Vectors for Word Representation, there is this part (bottom of third page) I don't understand:

I understand what groups and homomorphisms are. What I don't understand is what requiring $ F $ to be a homomorphism between $ (\mathbb{R},+) $ and $ (\mathbb{R}_{>0},\times) $ has to do with making $ F $ symmetrical in $ w $ and $ \tilde{w}_k $.

Am I misunderstanding something? We want $ F $ to be unchanged if we either interchange $ w_i $ and $ \tilde{w}_k $ OR interchange $ w_j $ and $ \tilde{w}_k $, right? Is this the only way to achieve the symmetry between  $ w $ and $ \tilde{w}_k $?


texts: Partial autocorrelation and Autocorrelation difference

texts:
I have been trying to understand better the PACF and ACF, but I'm literally struggling.

Have been using a series of articles like:
https://machinelearningmastery.com/gentle-introduction-autocorrelation-partial-autocorrelation/

And even looking at the method signature of statmodels for PACF and ACF I am still not able to get it.

Can somebody be so kind to make a "for dummies" explanation with some example of what is happening when data a considered?

Is it correct saying, among the other things, that

ACF is the plot of the AutoRegressive (AR) modelling whereas
PACF does the same but for the Moving Average (MA)

Or is it the other way round?

ACF is the plot of the Moving Average (MA) modelling whereas
PACF does the same but for the AutoRegressive (AR).


texts: Is it possible to predict based on arbitrary attributes?

texts:
I have a job/task runner system that takes a task description as input from a queue and delegates it to a machine on which to run. Based on the task type and other properties, some which are task-specific, the machine will change what it does (i.e. which app it runs). Each machine runs one task at a time. We send all tasks to a large sized machine right now which can handle anything we throw at it. This works fine, but wastes resources for smaller tasks.
I'm trying to determine if it's possible to predict the best class of machine on which to run a particular task, based on the task descriptor, so we can send wimpy tasks to a wimpy machine and tough ones to a more powerful machine, without needing to manually classify them ahead of time. The descriptors are JSON objects with a task type (app name) and "arbitrary" criteria, specific to that type of task. Various timing differences can affect the resources needed (time of day, week of year, etc.) even for the same task type (i.e. task A at the end of a quarter might take twice the ram as it did the week before).
I took a couple intro to ML courses 10+ years ago. Currently I am exploring ML.NET's text classification, though I feel like an ANN might work as well. Essentially I want a system that can recognize "if it's task A on week Y with criteria Z etc., it should go on a large machine" without predetermining all these heuristics.
One complication is that each task can have different criteria. Most are strings, and can be sparse (empty/null values are normal). An empty criteria is important and shouldn't just be ignored (e.g. it means process all items, not just one). Even if I split into models per-task-type this will be the case, though maybe easier to make accurate.
Another is that some attributes are negative indicators, e.g. if a job uses over 90% memory, crashed/errored or was otherwise unsuccessful, it should be considered "bad" and a larger machine used instead. I'm not sure how to represent or account for this though so that, when combined with the other attributes, a similar task would run on a larger machine. Maybe I could find these failed jobs, bump up the vm size, and train it that way, to nudge it upwards.
The idea is that for new tasks it would run the descriptor through a service and receive the (hopefully) best machine size to run it on, and after it runs/fails, train again to get more accurate.
Is there any algorithm or methodology that would be more applicable to this sort of problem?


texts:
I am trying to build a Recommendation system on my website for recommending similar articles to the user. Eg: Lets say a user is reading an article about sports on a news website. The next article should be related to sports. I have a dataset with a set of links to different articles.
Currently, I have the Page_title, Tags, Article content. I ran some NER's on the article content and found out important keywords. Then I have used Jaccard Similarity for combining the matrix from all 3 features and finding similarity between these links.
Is something wrong with my method?
Can it be improved?
How do websites solve this problem?
I am constrained by the data that I have.


texts: How to make a linear model with a constant value in R?

texts:
I'm working on an unassessed homework problem from unpublished course notes of a statistics module from a second year university mathematics course.
I'm trying to plot a 2-parameter full linear model and a 1-parameter reduced linear model for the same data set. I can't figure out how to plot the 1-parameter model; all attempts so far have either given errors or a non-constant slope.
xs <- c(0,1,2)
ys <- c(0,1,1)
Data <- data.frame(x = xs, y = ys)
mf <- lm(y ~ x, data = Data) # model_full
mr <- lm(y = mean(y), data = Data) # model_reduced; this is the bit I can't get to work
attach(Data)
plot(x, y, xlab="x", ylab="y")
abline(mf$coefficients[1], mf$coefficients[2])
abline(mr$coefficients[1], mr$coefficients[2])


texts: Dropout onto pre-weighted vs onto pre-activated vector?

texts:
For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector?

In other words:

$$\vec q=W\cdot \vec x$$
$$\vec h = activate(drop(\vec q))$$

or:

$$\vec q=W\cdot (drop(\vec x)) $$
$$ \vec h = activate(\vec q)$$

I think the second variant is smoother (none of our current vector is fully dropped out, but is assembled from a mix of the dropped-out input) and is therefore softer.


texts: How do I use NLTK Text object with Re library?

texts:
I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error;

  TypeError: expected string or bytes-like object

I have coded the following;

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

""" importing data set """
from nltk.corpus import PlaintextCorpusReader
corpus_root = './'
wordlists = PlaintextCorpusReader(corpus_root, 'evidence1.txt')
wordlists.fileids()
wordlists.words('evidence1.txt')
stringx = wordlists.words(wordlists.fileids()[0])
print (stringx)

""" cleaning the texts """
from nltk.text import Text
text = Text(stringx)
print(text)

import re
lowerx = re.sub('[^a-zA-Z0-9]','',text)

I think I need right kind of object to pass on to re.sub


texts: Learning resources for Data Science for applications in road traffic data?

texts:
My Background

I am a graduate student in Civil Engineering. For the analyses of road traffic data (vehicle trajectories as time series) I work with big data sets mostly about a million data points or more.
I started using R language when MS Excel could not open the big data files. Using basic statistics knowledge and R code I developed few algorithms to identify certain patterns in the data which worked for many applications. But I still lack serious programming skills in R.
Now, I am familiar with basic inferential statistics and R packages (plyr, dplyr, ggplot2, etc). Recently I came to know that Machine Learning algorithms also help in defining patterns in the data through supervised/ unsupervised learning and their application might improve the accuracy of prediction of certain 'behaviors' of drivers using the traffic data.

Question

Having the basic knowledge of Statistics and R, I want to learn about the data science/ machine learning as a beginner. I know that some concepts in Stats. and ML overlap and that might bridge the gap in my learning of ML. Keeping my background in mind, what resources (books/ online courses) would you recommend me to start learning data science and apply it in my field?


texts: Can I cluster an aggregated data-set (grouped by) and apply dimensionality reduction?

texts:
I have data of sales, however it is in the millions, about 500M rows. I aggregate this data by factors such as location, shoptype, country_of_shop, cardtype, and then the aggregated statistic is:

number of transactions
sum of amount in dollars

My questions is can i cluster this say using kprototypes or something similar? And can i reduce the dimensionality using Factor analysis?


texts:
I'm new into the ML Scene and I want to create a phonegap app involving Tensorflow but I'm unsure where to start or if this is even possible. Can anyone give me a hand (Probably by linking me to some resources)? My app will just use tensor flow image recognition (probably pre-trained).

Thanks, Felix.


texts: why is MSE of prediction way different from loss over batches

texts:
I am new to machine learning so forgive me if i ask stupid question. I have a time series data and i split it into training and test set.

This is my code:

from numpy import array
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# split a univariate sequence into samples
def split_sequence(sequence, n_steps_in, n_steps_out):
    X, y = list(), list()
    for i in range(len(sequence)):
    # find the end of this pattern
        end_ix = i + n_steps_in
        out_end_ix = end_ix + n_steps_out
        # check if we are beyond the sequence
        if out_end_ix > len(sequence):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

# choose a number of time steps
n_steps_in, n_steps_out = 10, 5
# split into samples
X, y = split_sequence(trainlist, n_steps_in, n_steps_out)
# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps_in))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mean_squared_error')
# fit model
history = model.fit(X, y, epochs=2000, verbose=0)
# demonstrate prediction
x_input = array(testlist[0:10])
x_input = x_input.reshape((1, n_steps_in))
yhat = model.predict(x_input, verbose=0)
yhat=list(yhat[0])

when i do print(history.history['loss'][-10:-1]) it gives me roughly 0.55 and when i do

from sklearn.metrics import mean_squared_error
mean_squared_error(testlist[11:16],yhat)

it gives me 0.11. Why is it so different?


texts: The Meaning Behind the Cross Validation Score in Factor Analysis

texts:
In order to choose the best number of underlying factors for my data using factor analysis, I decided to use the tutorial outlined in scikit-learn's documentation.

Running cross_val_score(fa, X) outputs a score (usually a negative number). What is this score actually measuring? Any references with your answer would be much appreciated!


texts: What is the loss function defined by Mnih and Hinton in their paper “Learning to Label Aerial Images from Noisy Data”?

texts:
In section 3.3 of the paper, they state that they use the cross entropy.
Then they define the probability for a label to be a false positive as $\theta_0$ and a false negative as $\theta_1$.

They use it to somehow modify the loss function but never actually state what this new loss is.

I was expecting something like

$f_\theta(\hat{m_i}, \tilde{m_i}) = \tilde{m_i} *g_{\theta_0}(\hat{m_i}) + (1-\tilde{m_i})*g_{\theta_1}(1-\hat{m_i})$

but $g$ is not given.


texts: Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)

texts:
I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many levels (on the order of 100-1000). Given that one-hot encoding creates a dummy variable for each level, the feature importances are for each level and not each feature (column). What is a good way to aggregate these feature importances?

I thought about summing or getting the average importance for all levels of a feature (probably the former will be biased towards those features with more levels). Are there any references on this issue?

What else can one do to decrease the number of features? I am aware of group lasso, could not find anything easy to use for scikit-learn.


texts:
I am looking for a news dataset with semantically duplicate news articles tagged.
Basically all the news articles which talk about the same story should be grouped. The stories can be worded differently but at a high level talk about the same event. Something like what google news does. Are there tagged news datasets for this ?


texts: ROC curve shows strange results for imbalanced dataset

texts:
I have a classifier with a heavily imbalanced dataset (1000 of each negative label for each positive.)

I'm running a GradientBoostingClassifier with moderate success (AUC .75) but the curve has this strange look:

Any good ideas on what would cause the curve to have this behaviour?


texts: Can you do automated feature engineering in R?

texts: Since now Python has its own deep feature synthesis library, is there anything similar available in R? I know of bounceR but I'm not sure if it does the job as DFS does. Is anyone aware of anything similar in R?


texts: Why can distributed deep learning provide higher accuracy (lower error) than non-distributed one with the following cases?

texts:
Based on some papers which I read, distributed deep learning can provide faster training time. In addition, it also provides better accuracy or lower prediction error. What are the reasons?

Question edited:

I am using Tensorflow to run distributed deep learning (DL) and compare the performance with non-distributed DL. I use the number of dataset 1000 samples and step size 10000. The distributed DL uses 2 workers and 1 parameter server. Then, the following cases are considered  when running the code:

Each worker and non-distributed DL use 1000 samples for training sets, same mini-batch size 200
Each worker uses 500 samples for training sets (first 500 samples for worker 1 and the rest 500 samples for worker 2), non-distributed DL use 1000 samples for training sets, same mini-batch size 200
Each worker uses 500 samples for training sets (first 500 samples for worker 1 and the rest 500 samples for worker 2) with mini-batch size 100, non-distributed DL use 1000 samples for training sets with mini-batch size 200

Based on the simulation, for all cases, distributed DL has lower RMSE than non-distributed DL. In this case, the RMSEs of distributed DL are as follows: Distributed DL in Case 2 < Distributed DL in Case 1 < Distributed DL in Case 3 < Non-distributed.

In addition, I also add the training time (i.e., the number of steps is 2 x 10000) for non-distributed DL, the results are still not as good as distributed DL.

One reason can be the mini-batch size, however, I wonder the other reasons why the distributed DL has better performance using the aforementioned cases?


texts: From development environment to production

texts:
I have been working on a project as part of my master degree in participation with a firm.
I developed a predictive model in the past few months that is essentially a document classification model. The biggest limitation of the research and model is the lack of data available for training. I have a small data set of 300 documents, where as the features are in excess of 15000 terms (before feature selection).

How do we identify or estimate the number of data-points (documents) necessary to gain a level of confidence that will allow us to move forward with the testing stage into a production environment ? So how do I estimate the necessary dataset size required for the performance metrics to be viable enough to ensure it is a solid model?
Are there standards or metrics in the industry for when a model is sufficiently tested and should qualify for production?

I cannot find literature about this topic.

Thank you


texts: Can somebody explain me the meaning of this sentence? (Color Similarity - Selective Search Algorithm)

texts:
This is a sentence from this article :

Color similarity: Computing a 25-bin histogram for each channel of an image, concatenating them together, and obtaining a final descriptor that is 25×3=75-d. Color similarity of any two regions is measured by the histogram intersection distance.

But I don't understand it. Can someone explain with an example?


texts: Forcing class imbalance to mirror the target data

texts:
I'm trying to do binary classification on some data, my source data has a class split of 40% A / 60% B while my target data has a split of 70% A / 30% B.

Is it a worthwhile strategy to use SMOTE to over-sample A such that I'm training on a class split that mirrors the data I'm trying to classify? The only metric that concerns me is accuracy.


texts: Using random forest to select important variables & then putting into logistic regression?

texts: I was wondering does it make sense to use random forest to select most important variables then put into logistic regression for prediction? I think that it might not make sense because what's important for random forest might not be important for logistic regression?


texts: How do you apply SMOTE on text classification?

texts: Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it possible to apply it on text classification problem? Which part of the data do you need to oversample? There is already another question about it, but it doesn't have an answer. Where can I possibly learn to get started with this?


texts: How to use Cross Entropy loss in pytorch for binary prediction?

texts:
In the pytorch docs, it says for cross entropy loss:

  input has to be a Tensor of size (minibatch, C)

Does this mean that for binary (0,1) prediction, the input must be converted into an (N,2) tensor where the second dimension is equal to (1-p)?

So for instance if I predict 0.75 for a class with target 1 (true), would I have to stack two values (0.75; 0.25) on top of each other as input?


texts: Do the filters in deconvolution layer same as filters in convolution?

texts:
I am trying to understand how the deconvolution works in Convolutional Neural Network for image segmentation problem. I have seen this definition:

Filters used in deconvolution is just the transpose of the convolution matrix.

Does this mean that the convolution layer and deconvolution layer shares the same filters? Do we train a separate set of filters for deconvolution layer?


texts: How to calculate Temperature variable in softmax(boltzmann) exploration

texts:
Hi I am developing a reinforcement learning agent for a continous state/discrete action space. I am trying to use boltmzann/softmax exploration as action selection strategy. My action space is of size 5000.

My implementation of boltzmann exploration:

def get_action(state,episode,temperature = 1):
    state_encod = np.reshape(state, [1, state_size])
    q_values = model.predict(state_encod)
    prob_act = np.empty(len(q_values[0]))

    for i in range(len(prob_act)):
        prob_act[i] = np.exp(q_values[0][i]/temperature)

    #numpy matrix element-wise division for denominator (sum of numerators)
    prob_act = np.true_divide(prob_act,sum(prob_act))

    action_q_value = np.random.choice(q_values[0],p=prob_act)
    action_keys = np.where(q_values[0] == action_q_value)
    action_key = action_keys[0][0]
    action = index_to_action_mapping[action_key]
    return action

If my temperature variable is 200, after 100 episodes I get an error

ValueError: probabilities contain NaN

If my temperature is 1 in very few episodes i get NaN error.

Why is this happening. Am I doing something wrong here? How to select the temperature variable? Can someone help me with this.

Thanks.


texts: EM parameter estimation for conditional Gaussians

texts:
Let $$X_1\sim N(\mu_{X_1},\sigma_{X_2}^2)$$ $$X_2\sim N(\mu_{X_2},
\sigma_{X_2}^2)$$
where $\mu_{X_2}=c+aX_1$. Also, I have data $D$ (with missing values on $X_1,X_2$).

How can I update/estimate the parameters $\mu_{X_1},\sigma_{X_1},\mu_{X_2},a,c,\sigma_{X_2}$ using EM?i.e. what is the formula for updating $\sigma_{X_2}$?

My model is a conditional Gaussian, which is a conditional form of a bivariate Gaussian $(X_1,X_2)$ with mean vector $(\mu_1,\mu_2)^\top$ and covariance matrix
$$\left( \begin{matrix}
   \Sigma _{11} & \Sigma _{12}  \\
   \Sigma _{21} & \Sigma _{22}  \\
\end{matrix} \right)$$

Here is a reference to convert bivariate Gaussian to conditional Gaussian: $$\mu_{2|1}=\mu_{2}+\Sigma_{21}\Sigma_{11}^{-1}(X_1-\mu_1)\quad,\quad \Sigma_{22|1}=\Sigma_{22}-\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}$$ which yields my model.

It seems that when $X_1$ has different observations and $X_2$ is unobserved, the variance of $X_2$ remains unchanged.So how to update the $\sigma_{x_2}$?

Initial setting for the model
$$X_1\sim N(5,7)\quad,\quad X_2\sim N(0.5X_1,8)$$

Data:

$$X_1:\operatorname{9\,\,4\,\,NA}$$

$$\quad X_2:\operatorname{NA\,\,NA\,\,3}$$


texts: Lowercase texts before tokenizing as pre-processing step for alignment

texts: I am pre-processing some texts and I wonder what the best practice is when preparing your texts for word alignment. I don't know how aligners such as fast align and GIZA++ work under the hood, but I could assume that they take capital letters as distinguished characters. Is there, therefore, any reason to lower case your text in such a situation? Is this language dependent? It might, for instance, be interesting for a language with many capital letters, such as German, to do so or not.


texts: Is it ok to interprete PCA plot this way?

texts:
I have several samples (C2, C4, C5) and want to check if they are at a certain stage. I included some known samples (D0 - D77) which were generated at different stages by another lab. In the PCA plot, my samples cluster together on the left and the known samples are dispersed on the right. I think the major difference among all the samples is different experimental protocols (PC1) and the second is different stages (PC2). So my samples are at the same stage. Is that right? And can we say my samples are at a stage between D12 and D19 (when projected to known samples, my samples are located between D12 and D19)? I have no strong mathematical background. Hope someone with math background can give some explanation. Thanks!

[ UPDATE1 ]
I did this analysis using prcomp in R. The input is a 2D numeric matrix with 17436 rows and 42 columns. Each row represents a gene and each column represents a sample. The number is the gene expression level for a gene in a sample. The gene expression level is normalized using DESeq2 and thus the numbers are comparable across genes (rows) and samples (columns). For the 42 columns, 18 are from my experiments and the rest from published datasets. Besides different protocols, it is possible there are other differences. Generally, the table is a combination of two sources of data. In my data , C2, C4 and C5 are three cell lines which we processed in parallel in experiments. In published datasets, they sampled the cells at different time points (Day 0 to Day 77).

[ UPDATE2 ] R Code for PCA and plotting

# normCounts: normalized count from DESeq2, with 17436 rows and 42 columns
normCounts0 <- normCounts[ rowSums(normCounts) > 0, ]
tab <- t(normCounts0)
pca <- prcomp(tab, scale = TRUE)
tmp.x <- as.data.frame(pca$x)
tmp.x$sample<-c(rep("C2_Con",3),rep("C2_KD",3),rep("C5_Con",3),rep("C5_KD",3),rep("C4_Con",3),rep("C4_KD",3), rep("D0",4),rep("D7",4),rep("D12",2),rep("D19",4),rep("D26",2),rep("D33",2),rep("D49",2),rep("D63",2),rep("D77",2))

require("ggplot2")
p <- ggplot(tmp.x, aes(x=PC1, y=PC2, color=sample))
p + geom_point() + scale_color_discrete(breaks=c("C2_Con","C2_KD","C5_Con","C5_KD","C4_Con","C4_KD", "D0","D7", "D12","D19","D26","D33","D49","D63","D77"))


texts:
I have a model that has to classify inputs into one of 45 categories but those categories actually represent bins (e.g. bins 1, 2 and 3 are between 1 and 10, 11 and 20, 21 and 30 respectively). What I would like is my model to classify properly values into bins but I am not too upset if it puts 19 into bin #3 even in bin #1. What is the loss function that would measure distance from the correct category and will score the classification in neighbouring bins with weight of say 1/n where n is the distance to the correct bin.

mind you that what I am looking for is different from the top_k metric that keras has.


texts: Keras 'cross section' model with time trend

texts:
Problem: I have a problem in which cross-sectional features (X) explain a continuous outcome (y). In addition, there is a linear time trend (t) in the data.
Using OLS, such a model would write: $y = b_0 + b_1 X + b_2 t + u$. Here, $b_1$ captures the cross sectional effects attributable to X and $b_2$ captures the time trend.
I would like to implement the logic of the OLS model in a Keras NN context.
I have tried "normal" Sequential() models with Dense() layers, but these model types do not fit the data very well. In fact, the Keras Sequential() model does not even beat a simple OLS model in terms of MAE.
I suspect that the Sequential() model architecture is not fit to deal with the linear time trend in my data.
I had a look at LSTM() models, but it seems that these models are mostly used to deal with autoregressive processes (at least this is what I could find online).
Question: What would be a good Keras model architecture to deal with a linear time trend as described above?
Can someone point me to some instructive resources?


texts: Case weights with sequence to sequence models?

texts:
I've got a bunch of variable-length sequences.  They're basically vectors of zeros with the occasional one.  Event rate is maybe 1%.
I want to build a model that will take sequences that I get in the wild -- same form and event rate -- and classify their inputs into zeros or ones.
Since the label is so rare, these models will struggle to classify.  But since the "batch" is the sequence, applying weights to the batch doesn't make any sense.  Most of the batch's labels will be zeros, because most of the sequences are zeros.
I had been planning on using a combination of LSTMs and 1D convs.  Is there some way to upweight certain parts of a vector-valued label?
And if it exists, is it implemented in tensorflow?
The simplest solution would be to cut out traning examples that highlight the label:  spans of the sequence with a higher event rate.  But this messes up the natural context that the label is found in, and I'll only do this if I don't think of a better option.


texts: How to use Embedding() with 3D tensor in Keras?

texts:
I have a list of stock price sequences with 20 timesteps each. That's a 2D array of shape (total_seq, 20). I can reshape it into (total_seq, 20, 1) for concatenation to other features.
I also have news title with 10 words for each timestep. So I have 3D array of shape (total_seq, 20, 10) of the news' tokens from Tokenizer.texts_to_sequences() and sequence.pad_sequences().
I want to concatenate the news embedding to the stock price and make predictions.

My idea is that the news embedding should return tensor of shape
(total_seq, 20, embed_size) so that I can concatenate it with the
stock price of shape (total_seq, 20, 1) then connect it to LSTM layers.

To do that, I should convert news embedding of shape (total_seq, 20, 10) to
(total_seq, 20, 10, embed_size) by using Embedding() function.

But in Keras, the Embedding() function takes a 2D tensor instead of 3D tensor. How do I get around with this problem?
Assume that Embedding() accepts 3D tensor, then after I get 4D tensor as output, I would remove the 3rd dimension by using LSTM to return last word's embedding only, so output of shape (total_seq, 20, 10, embed_size) would be converted to (total_seq, 20, embed_size)

But I would encounter another problem again, LSTM accepts 3D tensor not 4D so

How do I get around with Embedding and LSTM not accepting my inputs?


texts:
I am trying to build ARIMA model, I have 144 terms in my standardized time series, which represent residuals form original time series. This residuals, on which I would like to build ARIMA model, are obtained when I subtracted linear trend and periodical component from original time series, so residuals are stochastic component.

Because of that subtraction I modeled residuals like stationary series (d=0), so model is ARIMA(p,d,q)=ARIMA(?,0,?).

ACF and PACF functions of my residuals are not very clear as cases in literature for identification ARIMA models, and when I choose parameters p and q according to criteria that they are last values outside of confidence interval, I got values p=109, q=97. Matlab gave me error for this case:

  Error using arima/estimate (line 386)

  Input response series has an insufficient number of observations.

On the other side, when I am looking only to N/4 length of time series for identifying p and q parameters, I got p=36, q=34. Matlab gave me error for this case

  Warning: Nonlinear inequality constraints are active; standard errors may be inaccurate.

  In arima.estimate at 1113

  Error using arima/validateModel (line 1306)

  The non-seasonal autoregressive polynomial is unstable.

  Error in arima/setLagOp (line 391)
     Mdl = validateModel(Mdl);

  Error in arima/estimate (line 1181)
    Mdl = setLagOp(Mdl, 'AR' , LagOp([1 -coefficients(iAR)' ], 'Lags', [0 LagsAR ]));

How do I need to correct identify p and q parameters and what is wrong here? And wwhat does it mean in this partial autocorrelation diagram, why are last values so big?


texts: Multiple Time Series Forecasting with limited data

texts:
I'm doing time series forecasting for the sales of an assortment of items. The data set I have contains two years of sales record of items that are may or may not still be selling. The challenging part is that the product life cycles are usually one year, after which the sales would stop and the new items would be introduced. So anything with more than a year of sales history is likely to be outside of my forecasting scope as it would soon be removed from the assortment.
Intuitively, I'd think the algorithm should fit a model on all the active items (those with sales numbers) within a certain timeframe and extrapolate the components for the assortment as a whole, and then predict the sales of the currently active items using their historical data.
I'm new to ML and have tried several time-series forecasting methods in Python but so far none of them seems to yield any meaningful results. I'd appreciate it if you can point me in the right direction.


texts: How can I finetune XLM-R for neural machine translation between the same language(Catalan to Catalan-with different grammar structure))?

texts:
Can I fine-tune XLM-R to do something similar to https://www.guru99.com/seq2seq-model.html
(encoder-decoder architecture or any other way that perform good in small dataset - around 1000)
For example:
Input: He sells food(in Catalan)
Output: Food he sells(in Catalan)


texts: Estimating Variance Reduction Resultant from Additional Data

texts:
I couldn't quite think of how best to title this, so recommendations are welcome. Same goes for the tags (I don't have the reputation to use the tags that I thought were appropriate). The question is this:

"Suppose you have N pairs of observations, (x,y), and you have a model with some unknown parameters, B, that estimates the relationship between x and y, F(x,B) -> y. Now suppose you determine B using the method of least-squares (and, implicitly, that all the assumptions of least-squares are satisfied). The parameters, B, are themselves random variables, each with its own variance. Is there any way to estimate the reduction (or increase) in the variance of B that would result from applying the same method of least-squares to N+1 pairs of observations?"

The question is asked in the context of experimentation. If each data point costs $X, an affirmative answer to the question would go a long way in determining whether or not to continue testing.


texts: Generate dataframe series from current series which is a list of objects


texts: "Hadoop" formats for user database: online advertising

texts:
I was wondering if someone could point me to suitable database formats for building up a user database:

basically I am collecting logs of impressions data, and I want to compile a user database

which sites user visits, country/gender/..? and other categorisations with the aim of
a) doing searches: give me all users visiting games sites from france...
b) machine learning: eg clustering users by the sites they visit

so I am interested in storing info about 100's of millions of users

with indexes? on user, sites, geo-location

and the idea would be that this data would be continually updated ( eg nightly update to user database of new sites visited etc)

what are suitable database systems. Can someone suggest suitable reading material?
I was imagining Hbase might be suitable...


texts: Heat maps in R with more than 2 categorical variables

texts:
I have a dataset which looks like-

Variable value year Quarter Location
A 48.235 2011 Q1 North
B 65.444 2011 Q2 North
C 77.453 2011 Q3 North
D 44.678 2011 Q4 North
A 88.542 2012 Q1 South
B 66.566 2012 Q2 South
C 55.443 2012 Q3 South
D 78.990 2012 Q4 South

Can anybody help me with a code to generate heatmap with more than 2 categorical variables? I need 'variable' and 'Location' to be on y axis and 'Quarter'&' year' to be on x axis. The shades of the data inside heat map should correspond to 'value'.


texts:
I just recently got interested in Data Science, and just encountered concepts such as embedding from high dimensions to low dimensions. Maybe on an irrelevant note, but in this video: Geometry of Singular Value Decomposition (SVD) | Lê Nguyên Hoang, he mentioned that we can think of SVD as mapping of movies into concepts and mapping of users into concepts. Furthermore, he also mentioned that such concepts are to be data-driven. That makes me wonder if the theorem of SVD imply that there exists intrinsic categorizations of data.

Can someone give me any analysis on why this might be true or false ? Also it would be appreciated if you can appeal to intuitions, since I am very new to this area.

Edit The url was corrected, i mistook it for another video


texts:
can anyone explain difference between Alternating Least Squares(ALS) and the recommendation systems?
it will be helpful if you give me an example.


texts:
I am working on the Boston challenge hosted on Kaggle and I'm still refining my features.  Looking at the dataset, I realize that some columns need to be encoded in binary, some encoded in decimals (ranking them out of a scale of n) and some need to be one-hot-encoded.  I've collected these columns and categorized them in distinct lists (at least based on my judgement on how their data should be encoded):

categorical_columns = ['MSSubClass', 'MSZoning', 'Alley', 'LandContour', 'Neighborhood', 'Condition1', 'Condition2',
                       'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'Foundation', 'Heating',
                       'Functional', 'GarageType', 'PavedDrive', 'SaleType', 'SaleCondition']

binary_columns = ['Street', 'CentralAir']

ranked_columns = ['LotShape', 'Utilities', 'LandSlope', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond',
                  'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'FireplaceQu',
                  'GareFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']

One fellow stackexchange user suggested that I use pandas.get_dummies() method to one-hot-encode categorical variables like MSZoning and attach it to a variable like this:

OHE_MSZoning = pd.get_dummies(train['MSZoning'])

I'd like to know how I can automate this process using functions and control-flow statements like for-loop.


texts: Data sets for evaluating text retrieval quality

texts:
I'm searching for data sets for evaluating text retrieval quality.

TF-IDF is a popular similarity measure, but is it the best choice? And which variant is the best choice? Lucenes Scoring for example uses IDF^2, and IDF defined as 1+log(numdocs/(docFreq+1)). TF in lucene is defined as sqrt(frequency)...

Many more variants exist, including Okapi BM25, which is used by the Xapian search engine...

I'd like to study the different variants, and I'm looking for evaluation data sets. Thanks!


texts: What are Some Good ML/Deep Learning Approaches to This Type of Problem

texts:
Problem Definition:

You are given a dataset of $N$ different features.
Most of the features are actually calculations of their own time series of set length (i.e A 5-time step weighted moving average). These time series are not necessarily related (There may be a moving average of hospital capacity and another moving average of infection rate).
At each sample, there is a time series of variable lengths (data is not missing, you can choose to put in the missing values) for each feature.
Despite the fact time series for each feature are variable lengths, these time series lengths are consistent throughout the samples. For example, say the max time series length is $D$ and in sample $X$, feature $A$'s time series length is $D - K$ for some natural number $K$. Then in sample $Y$, feature $A$'s time series length will still be $D - K$.
The time series of the different samples are not continuous. For example, feature $A$ might look like this in sample $X$ $(A_{Z}, A_{Z+1}, ..., A_{Z + (D-K-1)})$, but feature $A$ in sample $X+1$ might not look like $(A_{Z+1}, A_{Z+2}, ..., A_{Z + (D-K)})$, but as stated before, you can be guaranteed the time series length will be of $D-K$.
The goal is to predict the likelihood of events $A$ and $\neg A$ happening (binary classification) within the next $y\neq 1$ (to be optimized) time steps.
The dataset is unbiased and large ($\gt 1,000,000$ samples)

In summary: After some signal is generated (non-continuous time series from sample to sample), the associated time series of the features are then generated thereafter. Using these features we predict the likelihood of a particular event occurring in the near future.
What I am currently thinking:
I am currently deciding between an LSTM and a single-layered CNN model to tackle this type of problem as it is my understanding that LSTM's and CNN's work best for multivariate time series classification problems. I am also weighing the possibility of merely using the most recent values instead of the time-series structure and going with a simple ANN. The reason I am going with a neural network approach is due to the fact they are more friendly on a large dataset.


texts:
I am new to neural networks. I tried coding the backpropogation alogrithm and tried running it on a test set which gave wrong results.
I have used the following knowledge to code it,

For the forward pass,
$$z^l = w^la^{l-1} + b^l$$
$$a^l = g^l (z^l)$$

For the backward pass, (Here $\circ\text{ - Element wise Product}$)

For the last layer,
$$\delta ^L = (a^l - y) \circ g'^L(z^L)$$
$$\frac{\partial L}{\partial w^L} = \delta^L (a^{L-1})^T$$
$$\frac{\partial L}{\partial b^L} = \delta^L$$

For the other layers,
$$\delta ^l = (w^{l+1})^T(\delta^{l+1}) \circ g'^l(z^l)$$
$$\frac{\partial L}{\partial w^l} = \delta^l (a^{l-1})^T$$
$$\frac{\partial L}{\partial b^l} = \delta^l$$

For the Update,
$$W : = W - \alpha \frac{\partial L}{\partial w^l}  $$
$$b : = b - \alpha\frac{\partial L}{\partial b^l} $$

The following is the code which I have written,

    X = load('iris.csv');
    W1 = rand(3,4);
    W2 = rand(1,3);
    b1 = rand(3,1);
    b2 = rand(1,1);
    y = [ones(1,50) 2*ones(1,50) 3*ones(1,50)]';

    for j = 1:100    %for epoch
    for i = 1:150    %for iteration through dataset
    x1 = X(i,:)';
    z1 = W1*x1 + b1;
    a1 = tanh(z1);
    z2 = W2*a1 + b2;
    a2 = relu(z2);

    dz2 = (a2 - y(i)).*reluGradient(a2);  %this is delta L
    dw2 = dz2*transpose(a1);              %The derivative term of Lth
    db2 = dz2;                            %for the bias

    g1 = 1 - a1.^2;                       %Derivative
    dz1 = transpose(W2)*dz2 .* g1;        %FOr delta l
    dw1 = dz1*transpose(x1);              %derivative term for lth layer
    db1 = dz1;                            %bias update

    W1 = W1 - 0.1*dw1;
    b1 = b1 - 0.1*db1;

    W2 = W2 - 0.1*dw2;
    b2 = b2 - 0.1*db2;

    end
    end

I am trying to train the net for the iris data set (150 X 4 - dataset Size).
I have considered 4 input units, 1 hidden layer with 3 hidden units and 1 output unit. Hence the dimension of the weight matrix for first layer is 3 X 4 and for the last layer is 1 X 3. When I try to test the network I always get the input classified to class 3. I tried changing the hyper parameters, but it seems there is something wrong with the code.

In the code , I first load the CSV file, and then initialize the weight matrices accordingly. I have run two for loops one for the epoch and other for the iteration. I do a forward pass first using the above equations then a backward pass. I have made functions for the RELU and RELU_GRADIENT.

For relu:

function g = relu(z)
g = max(0,z);
end

For relu gradient:

function z = reluGradient(z)
z(z>=0) = 1;
z(z<0) = 0;
end

I would be obliged if someone can direct to me to the solution to this problem

About IRIS.csv:

Has four input features and 150 data samples.
50 of each class. Hence there are 3 classes.

Updates:

The cost function here I have used is the quadratic cost function:

$$C = \Sigma (a^L - y)^2 $$

Hence all the derivatives have been calculated with respect to that cost function. For example for the last layer,

$$\frac{\partial C}{\partial z} = (a^L - y) \circ g'^L(z^L)$$ which is $\delta^L$

Also,

I have tried using the sigmoidal transfer function instead of the relu transfer function with all the proper changes at all derivatives,weight and output vectors. (The output vector was then of (3,1) dim)
As 3 classes hence - $[1,0,0]^T$ , $[0,1,0]^T$ and $[0,0,1]^T$

Thanks


texts: What is the positional encoding in the transformer model?

texts:
I'm trying to read and understand the paper Attention is all you need and in it, there is a picture:

I don't know what positional encoding is. by listening to some youtube videos I've found out that it is an embedding having both meaning and position of a word in it and has something to do with $sin(x)$ or $cos(x)$

but I couldn't understand what exactly it is and how exactly it is doing that. so I'm here for some help. thanks in advance.


texts: Difference between CBOW and SKIP Gram word vectors

texts:
I have gone through several links but was not able to understand how CBOW and Skip Gram is trained from scratch?
Any good link/blogs or books would be very helpful.
Word2Vec - CBOW and Skip-Grams
What's the difference between Skipgram word2vec and CBOW word2vec during training and when to use CBOW Skip-gram. ?
Example or Application where CBOW would be preferable choice but not Skip-gram and vice versa.


texts: How to measure the information of covariates in a ML task?

texts:
Background

Recently, I do 2 different ML projects.

One is lending club loan prediction, another is a pravite dataset in online experiment field to predict whether a customer will take the treatment.

Both 2 tasks are binary classification with 100+Million observations and a hundred covariates. However, my model of lending club have a very high PR-ROC(0.86), which shows the good performance of model. My model of online experiment suffers, only have a 0.03 PR-AUC. The model is somekind useless.

I tried to explain to my leader that the dataset is so uninformative and that's the real reason why my project failed. I use low metric scores(PR-AUC = 0.03) and high loss function values as proofs to show the model I build is useless.

Question

Later, I came across one question, how do we measure the information contains in the covariates regardless of the model we built?

If we perform regression tasks using linear model, we can use RMSE, AIC, BIC to select a model, choosing the best model who mined the covariates best. If we perform binary classification ML task like I did before, we can use F1, ROC and PR-ROC metrics etc. I think these metrics helps us to compare model performance instead of the potential data quality.

The solution I want is something like entropy. For instance, we can calculate the entropy between 2 probability distributions or between covariate and target label. Entropy shows us the relation intensity between covariates and label. Is there any better solution to measure the information contains in the covariates?

I am a graduate of statistics, thanks a lot if you guys can provide me any source to learn!


texts: Apply feature importance obtained from entire dataset to individual row

texts:
There are various methods for calculating feature importance. These are generally obtained from computing the entire dataset. Can this feature importance be then applied to specific rows?

In other words, what is a practical way of applying feature importance to a specific row? An application is as follows. Say we have a binary classification problem with 10 features, X1, X2, ...., X10. Also, from the feature importance analysis, X1 is more important than X2, X2 is more important than X3, and so on and so forth. Also assumed is that these are numerical features and range between 0 and 5.

Upon the binary prediction, if the prediction is negative, we have one chance to pick a feature and try to influence this feature so that it can yield a positive outcome. Since we can pick only one feature to influence, the feature importance ranking is important.

How do we apply the previously determined feature ranking (from the entire dataset) to this individual?


texts:
Every one. I have EEG dataset with 80 subjects, 3072 data points and 100 trials. This a univariate data, it mean there is only one channel. I am confused how to feed this data to convolution neural network.
 Most of blog and tutorial deals with multivariate data. But i have univariate and i am clueless how to move ahead


texts:
I am trying to do svm model training and it gives this error:

 ValueError: Found input variables with inconsistent numbers of samples: [91, 212]

Code:

target = data.target
target.head()

##splitiing data into training and test data
X_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.3, random_state=101)
print("X_train size ===>", X_train.shape)
print("y_train size ===>", y_train.shape)
print("X_test size ===>", X_test.shape)
print("y_test size ===>", y_test.shape)

#Create a svm Classifier

clf = svm.SVC(kernel='linear')

#training the model using the training sets

clf.fit(X_train, y_train)

#predict the response for test dataset

y_pred = clf.predict(X_train)


texts: YOLOv3 Predicting bounding boxes for grid containing multiple centers

texts:
I recently started learning about YOLO and object detection, and I am kind of stuck on something. I was wondering if someone could explain to me what happens when a grid cell contains the centers of more than one bounding box. Does the grid cell still predict a single bounding box?
In YOLOv3 the image is divided into several grid cells of size $13\times13$, $26\times26$ and $52\times52$ Each of these grid cells are responsible for predicting a bounding box if the center of the ground truth bounding box is in the grid cell.
I know that each grid cell predicts $B$ boxes, where $B$ is the number of anchor boxes. The dimensions of each of these predictions is $4+1+n$, where $n$ is the number of classes. $4$ for $b_x,b_y,b_w,b_h$, and 1 for the objectness score. $b_x,b_y,b_w,b_h$ is defined below:
$$
b_x = \sigma(t_x)+c_x \\
b_y = \sigma(t_y)+c_y \\
b_h = e^{t_h}p_h \\
b_w = e^{t_w}p_w
$$
$p_h$ and $p_w$ is the height and width of the anchor boxes. $t_x, t_y, t_h, t_w$ are predictions made from a single anchor box of a grid cell. My confusion is when a grid cell contains the centers of more than one bounding box, there are more than one values for $t_x, t_y, t_h, t_w$, but the description of the YOLOv3 training process or the above equations does not seem to take this into account.


texts: Can cluster analysis of preclassified items gives idea about the classification performance?

texts:
Suppose in a classification we have a dataset with many features and their class, we want to select some features using which we can construct a classifier.

We perform the cluster evaluation for the given dataset taking different features (or derived features) and find cluster purity and entropy.

Can these purity and entropy calculated for certain feature vector be used to predict the classification performance of the classifier constructed for those set of features.

For eg.

I have set of features vector {A,B,C,D,E,F} and n no dataset. on selection of features there are many possible feature vectors I am considering two: X = {A,B,C,D} and Y = {A,E,F}. Now I perform cluster evaluation and find cluster purity for both X and Y. Can these purity(X) and purity(Y) predict which one is a better selection of feature for classification.

I think this purity may be a lower bound for the performance of classification.

PS: The clustering used is assumed to be k-means clustering with k = no of classes.

originally posted here, but could not find the answer, so posting here too.


texts: Recommender systems and Machine Learning

texts:
According to Mitchell:

  “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T , as measured by P , improves with experience E” (Mitchell, 1997) .

If we put this in terms of recommender systems:

Task: recommend products

Experience: includes the experience of observing a set of examples encoded in a ratings matrix

Performance: metric i.e. rmse between real and predicted ratings

Is my approach okay?

Any suggestions are welcomed!


texts: How do you integrate nlp into an existing website?

texts:
Let's say I have a db driven app which has two tables: people and organizations. I run my documents through a named entity recognition program. Now - what do I do with this additional ner information?

Does it go in my existing db tables?
Do I have to rewrite my existing tables to accommodate the ner data?
Do I make new tables for the ner data?
Do I connect them to my older tables with foreign keys?
Do I need new functions that pull response data from these new ner objects instead of, or with, my original ones?
Are these new entities "objects" in the traditional OOP sense?
But they are different objects from the ones in my original db tables, so do they replace the old, non-ner objects?
Does it matter what kind of datastore the new ner objects go into in order to be fully utilized? rmdbs? key/value? document based?

The only thing I’ve been able to find online about this is a Medium post that pickles the model - it says nothing about any other kind of datastore - and the Flask website just displays the information by accessing it through an api.

A search here on datascience.stackexchange.com got this:

  We couldn't find anything for integrate with existing website

And now I get a warning that my question "appears to be subjective and is likely to be closed". I'm sure that's machine learning at work. Granted, the step by step details might be different if your site is php vs python, but the issue is a major one I assume a lot of people are going to run into sooner if not later. My site is in Django.

How do you integrate the product of nlp into an existing, complex website with many apps and functions?


texts: How to use in built Keras ADDITIVE ATTENTION Layer for image captioning?

texts:
I have Designed an Encoder-Decoder Model for Image Captioning. Now, I want to improve my Model. So, I thought of putting an Attention Layer in my Encoder-Decoder model. But, I am struggling with how to use Keras Attention layer API in my model. What are the input and output of the Attention layer(i.e what is query, key ,value provided in documentation) and how to get them?
Here we are using a feature vector of the image from a pre-trained Xception Model.
How to manipulate or change this code to add the Attention Layer of Keras Library (https://keras.io/api/layers/attention_layers/additive_attention/) to built an image captioning Model to this Encoder-Decoder Model?

Here is my Previous Encoder-Decoder Model..
def caption_model(vocab_size,max_len):

    # features from the CNN model squeezed from 2048 to 256 nodes
    inputs1 = Input(shape=(2048,))
    fe1 = Dropout(0.5)(inputs1)
    fe2 = Dense(256, activation='relu')(fe1)

    # LSTM sequence model
    inputs2 = Input(shape=(max_length,))
    se1 = Embedding(vocab_size, 256, mask_zero=True)(inputs2)
    se2 = Dropout(0.5)(se1)
    se3 = LSTM(256)(se2)

    # Merging both models
    decoder1 = add([fe2, se3])
    decoder2 = Dense(256, activation='relu')(decoder1)
    outputs = Dense(vocab_size, activation='softmax')(decoder2)

    # tie it together [image, seq] [word]
    model = Model(inputs=[inputs1, inputs2], outputs=outputs)
    model.compile(loss='categorical_crossentropy', optimizer='adam')
    # summarize model
    print(model.summary())
    plot_model(model, to_file='model.png', show_shapes=True)

    return model
```


texts: Is it required to use Tensorflow in Google ML Engine, or can Theano etc be used?

texts:
I am just getting into Googles Cloud services, and have not been doing much ML in Python. I have done one or two ML projects in school using Tensorflow, but not really accustomed with the other available libraries. Is it so that if I am interested in using the Google Cloud services I should definitely be using Tensorflow, or can they be integrated with Theano or other ML libraries too? If they can be used, can they only be used in part and some functionality is lost, or are they fully available?

Just looking for some guidance if there is any argument for using something else than Tensorflow for someone in my shoes, and don't really find good guidance using Google as they promote their own product.


texts:
I'm working on a multiclass classifier with 6 classes on the target column and I was thinking about Hot Encoding the classes, thus having 6 target columns. Will this improve efficiency? I am using sklearn.
L.E: improve efficiency compared to just label encode


texts: Can we use package "dplyr" on R base 3.0.2?

texts:
Is there any way to use package dplyr on RStudio having R base 3.0.2 ?

I am not interested in plyr package.


texts: Predict words in a given corpus from jumbled incomplete characters

texts:
A complete newbie to data science. I have to predict the word using characters given. But the characters given can be in any order, and some might be missing or wrong. The problem is I get too many suggestions that arent right.

Trial 1:

I have tried permutating the given characters and checking if it is a valid english word : https://pastebin.com/ZUZDZPPa
This does not take into account autocorrect like features.
With autocorrect: https://pastebin.com/mDjVn0Rz

Trial 2:

Tried limiting editdistance (levenshtein distance) to 2 and used my custom corpus that I built as I knew the context of the word (Occupations or hobbies).https://pastebin.com/iC5TYxzS The problem is that the predictions I get arent close to the answer. I got pretty good results in Trial 2, but I am wondering if there was a better approach as edit distance of 2 is pretty limiting.


texts: Why do the deconv outputs of layer >= 2 of Zeiler&Fergus look so unrealistic?

texts:
Reading the Zeiler&Fergus paper (my summary), I wonder how exactly they trained the deconv net. What was their data?

I think for one CNN which they want to analyze, they train exactly one deconv net (in contrast to training one deconv net per layer). The featuers (inputs) of the deconv net are the activations of the layer they want to analyze. The output they train them on are the activations that actually was the input of the layer they want to analyze. So although they have one deconv-net in total, they train it layer-wise. So for each training run, the weights of only one deconv layer are adjusted.

However, I wonder why the images look that unrealistic:

Is it gray because MSE is the training objective? Why aren't the first layer filter outputs gray then, too?


texts: variables selection in regression models

texts:
I develop price prediction data model using multiple linear regression, ridge, lasso and elastic net regression, initially I had 215 variables. after creating models I ran a python code to check how many variables have used in final models, this is python code which i use for the detect number of variables in ridge regression,

print("Ridge Regression Selected " + str(sum(coef_ridge != 0)) + " Variables and Neglected " +  str(sum(coef_ridge == 0)) + " Variables")

This is a out put which I got

Ridge Regression Selected 209 Variables and Neglected 6 Variables

above code ran for all other modeling methods separably. then Multiple regression   212 variables selected, Ridge 209 variables selected, Lasso 68 variables selected and Elastic net regression 77 variables selected.

my question is according to my knowledge lasso and elastic net regression already use variables selection method because of that, number of selected variables reduced up to 68 and 77 but how multiple linear regression neglect 3 variables and ridge regression reduce 6 variables? that mean multiple and ridge regression also use variable selection method in their code? please someone explain these scenario


texts: How to encode H3 geohash in regression model

texts:
I'm trying to train a random forest regression model based on a number of features, including location. I know that raw lat/long can't be used directly, so I've bucketed them using H3.

I'm struggling to figure out how to encode the hex grid IDs in the training data. The IDs are just 64bit integers, so can I just feed them directly into my model, or do I need to do another kind of encoding?


texts: Chossing between Prediction vs classification model for dataset having daily record(date value)

texts:
I have a use case, where I have 4 classes based on the score, for example

class 1 : when the score < 10
class 2 : when the score between 11 to 20
class 3 : when the score between 21 to 30
class 4 : when the score > 30

The score value is continous data and is derived from the given aggregated dataset for each day (date) for different stocks.

My training dataset contains both the score and class value for each day.

The problem statement is, I need to build a model which will predict if there will be any movement from the current class to a another class
(i.e. class 1 to class2 or class 3 to class2 etc.) in the subsequent day.

For this, I thought of below options and want to check for better ways.
Option 1:
1. Build a linear model (using Linear Regression or SVM, NN etc) to predict the score which is required to predict the class movement.
2. Use the predicted score to decide if there will be any change in class based on the range provided above.
3. Build a confusion matrix for the different class (1 to 4) to show the actual vs predicted score
Option 2:
1. Directly Build a multi-class classification model to predict the the class movement.
2. Build a confusion matrix for the different class (1 to 4) to show the actual vs predicted score.

Since the dataset contains the date and stock name for each day, Is it necessary to use the Timeseries analysis for this use case?
Instead of that can i split the date into year, month & day and use it for training?


texts: Error plotting with datetime and value using matplotlib in python

texts:
Here I have a dataset with three values. I plot these three values in one graph using python. While I plot the graph first plot line is not coming properly. Because start point and end point combined with a line. So does anyone know how to remove this line? I'm stuck with this error. Please can you all help me to solve this error?
Here is my code

data=pd.read_csv('data1.csv')
data = pd.DataFrame(['date', 'time','x1','x2','x3'])
x = pd.to_datetime(data['date'] + ' ' + data['time'], format='%d/%m/%Y %H:%M:%S')
y =data.iloc[:,2]
y1=data.iloc[:,3]
y2=data.iloc[:,4]
idx = ~np.isnan(y)
idx1=~np.isnan(y1)
idx2=~np.isnan(y2)
import matplotlib.dates as mdates
fig = plt.figure(figsize=(6,6))
ax=fig.add_subplot(111, label="1")
ax2=fig.add_subplot(111,label="2", frame_on=False)
ax3=fig.add_subplot(111,label="3", frame_on=False)
ax.plot(x[idx],y[idx], color="C0")
ax.set_xlabel("date1 label 1", color="C0")
ax.set_ylabel("y label 1", color="C0")
hours = mdates.HourLocator(interval =10)
h_fmt = mdates.DateFormatter('%H')
ax.xaxis.set_major_locator(hours)
ax.xaxis.set_major_formatter(h_fmt)
#ax.set_xlim([xmin,xmax])
#ax.set_ylim([ymin,ymax])
ax.tick_params(axis='x', colors="C0")
ax.tick_params(axis='y', colors="C0")
ax2.plot(x[idx1], y2[idx1],'g*')
ax2.xaxis.tick_top()
ax2.yaxis.tick_right()
ax2.set_xlabel('x label 2', color="C1")
ax2.set_ylabel('y label 2', color="C1")
ax2.xaxis.set_label_position('top')
ax2.yaxis.set_label_position('right')
hours = mdates.HourLocator(interval =10)
h_fmt = mdates.DateFormatter('%H')
ax2.xaxis.set_major_locator(hours)
ax2.xaxis.set_major_formatter(h_fmt)
#ax2.set_xlim([xmin,xmax])
#ax2.set_y1lim([y1min,y1max])
ax2.tick_params(axis='x', colors="C1")
ax2.tick_params(axis='y1', colors="C1")
ax3.plot(x[idx2], y1[idx2],'k.')
hours = mdates.HourLocator(interval =10)
h_fmt = mdates.DateFormatter('%H')
ax3.xaxis.set_major_locator(hours)
ax3.xaxis.set_major_formatter(h_fmt)
#ax3.set_xlim([xmin,xmax])
#ax3.set_y3lim([y3min,y3max])
ax3.set_xticks([])
ax3.set_yticks([])
fig.autofmt_xdate(rotation=90)
plt.show()

Here I attached the my csv file link.
https://docs.google.com/spreadsheets/d/1y6LKGIftb7F0ovF2dYsMwmvkR_diJBhBNC6sIiaagbI/edit?usp=sharing

my graph:


texts: How to calculate joint feature contribution for XGBoost Classifier in python?

texts:
I referred to this beautiful document to research about joint feature contibutions. But this works only for RandomForest algorithms because of treeinterpreter (does not work with xgboost). Is there a similar way out for XGBoost as well?
Basically what I want to achieve is to find out the joint contributions of all the combination of features towards the prediction. For example if I have a, b and c as my features, I want to know what is the effect of ab, bc and ca towards the prediction result. It is very similar to shap and lime, but for combinations of features.


texts: Why does SimpleNet achieve accuracy 0.9+ but AlexNet does not?

texts:
I am trying to classify some 224x224 RGB images.

I have 3 potential labels, but am currently only supplying training images for 2. My training set has approx. 2K images for both labels.

When I train with simple net I get 0.9+ accuracy within 6 epochs.

When I train with AlexNet I am still only around 0.5 accuracy after 50 epochs.

Why might this be?

def SimpleNet():
    model = keras.Sequential([
        keras.layers.Conv2D(64, kernel_size=3, activation=tf.nn.relu, input_shape=(224, 224, 3)),
        keras.layers.Dropout(0.75),
        keras.layers.Conv2D(32, kernel_size=3, activation=tf.nn.relu),
        keras.layers.Dropout(0.75),
        keras.layers.Flatten(),
        keras.layers.Dense(3, activation=tf.nn.softmax)
    ])
    return model

def AlexNet():
    model = keras.Sequential()

    # 1st Convolutional Layer
    model.add(keras.layers.Conv2D(filters=96, input_shape=(224, 224, 3), kernel_size=(11, 11), strides=(4, 4), padding='valid'))
    model.add(keras.layers.Activation('relu'))
    # Max Pooling
    model.add(keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))

    # 2nd Convolutional Layer
    model.add(keras.layers.Conv2D(filters=256, kernel_size=(11, 11), strides=(1, 1), padding='valid'))
    model.add(keras.layers.Activation('relu'))
    # Max Pooling
    model.add(keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))

    # 3rd Convolutional Layer
    model.add(keras.layers.Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), padding='valid'))
    model.add(keras.layers.Activation('relu'))

    # 4th Convolutional Layer
    model.add(keras.layers.Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), padding='valid'))
    model.add(keras.layers.Activation('relu'))

    # 5th Convolutional Layer
    model.add(keras.layers.Conv2D(filters=256, kernel_size=(3, 3), strides=(1, 1), padding='valid'))
    model.add(keras.layers.Activation('relu'))
    # Max Pooling
    model.add(keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))

    # Passing it to a Fully Connected layer
    model.add(keras.layers.Flatten())
    # 1st Fully Connected Layer
    model.add(keras.layers.Dense(4096, input_shape=(224 * 224 * 3,)))
    model.add(keras.layers.Activation('relu'))
    # Add Dropout to prevent overfitting
    model.add(keras.layers.Dropout(0.4))

    # 2nd Fully Connected Layer
    model.add(keras.layers.Dense(4096))
    model.add(keras.layers.Activation('relu'))
    # Add Dropout
    model.add(keras.layers.Dropout(0.4))

    # 3rd Fully Connected Layer
    model.add(keras.layers.Dense(1000))
    model.add(keras.layers.Activation('relu'))
    # Add Dropout
    model.add(keras.layers.Dropout(0.4))

    # Output Layer
    model.add(keras.layers.Dense(3))
    model.add(keras.layers.Activation('softmax'))

    return model


texts: Best Python library for statistical inference

texts:
I'm curious if anyone has Python library suggestions for inferential statistics. I'm currently reading An Introduction to Statistical Learning, which uses R for the example code, but ideally I'd like to use Python as well.

Most of my data experience is with Pandas, Matplotlib, and Sklearn doing predictive modeling.

So far I've found statsmodels. Is this what is recommended or is there something else?

Thanks!


texts: Why do popular search engines not follow the usual AND, OR logic for queries?

texts:
I am teaching myself Information Retrieval from Christopher Manning's book (PDF link: http://nlp.stanford.edu/IR-book/pdf/01bool.pdf). I tried Exercise 1.13:

"Try using the Boolean search features on a couple of major web search engines. For
instance, choose a word, such as burglar, and submit the queries (i) burglar, (ii) burglar
AND burglar, and (iii) burglar OR burglar. Look at the estimated number of results and
top hits. Do they make sense in terms of Boolean logic? Often they haven’t for major
search engines. Can you make sense of what is going on?"

By my knowledge of Boolean logic, the number of results should be like this:

burglar AND burglar <= burglar OR burglar = burglar

But this isn't so. In fact, on Google, it is:

burglar > burglar OR burglar > burglar AND burglar

So, what exactly is happening behind the scenes? Any pointers?

Note: This is NOT a homework problem, even though it is from the exercise of a textbook.


texts: Repeated features in Neural Networks with tabular data

texts:
When using algorithms like linear regression or least-squares methods, having repeated or highly correlated features can be harmful for the model. For tree based models, they are generally not too strongly affected by highly correlated features. There are no numeric stability issues as with least squares.
But what happens with Neural Networks? Most of the literature on NN is made for images, signal and there is not too much about tabular data.
Having repeated features in a tabular data Neural Network model, does it harm the accuracy? Or NN are able to select features?


texts: Using pandas get_dummies() on real world unseen data

texts:
I made a ML model, trained and tested it with my data containing categorical variables.
To create dummy variables I used pd.get_dummies() before the split.
I now want to use my model on previously unseen data where, of course, I need to re create my dummies. Should I do it still with pd.get_dummies()? In this way isn't the encoding lost? Any suggestion on how to do it?
Thanks


texts: How to manually calculate AUC and Accuracy

texts:
How to calculate AUC using some formula? What are the parameters required and what formula to use.

For accuracy, $$ \frac{TP+TN}{Total} $$ is this right way to calculate AUC?


texts: Error with negative bins while melting dataframe

texts:
I have a dataframe 'd' like this:

   breaks counts.x counts.y
1     -20        1        1
2     -15        0        1
3     -10        0        5
4      -5        4       18
5       0       13       27
6       5       18       25
7      10        9       12
8      15        2        1
9      20        1        7
10     25       NA        0
11     30       NA        1

When I try to melt using 'breaks'  as id variable it is giving me the following error:

 d=melt(d,id=breaks)
Error in varnames[id.vars] :
  only 0's may be mixed with negative subscripts

How can I resolve this?
I have to draw a ggplot using breaks on x axis and counts on y axis.


texts:
I came across an interesting book about neural network basics, and the formula for gradient descent from one of the first chapters says:

  Gradient descent: For each layer update the weights according to
  the rule

  $w^l \rightarrow   w^l-\frac{\eta}{m} \sum_x \delta^{x,l} (a^{x,l-1})^T$

where $w^l$ is the weights matrix in layer $l$, and $x$ is the index of a specific training example.

I don't want to rewrite all formulas from the chapter, but the important part one is BP4 - equation for the rate of change of the cost with respect to any weight in the network:

$\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j$

Am I missing something or the first formula is incorrect? Shouldn't we use Hadamard product instead, like this?

$w^l \rightarrow
  w^l-\frac{\eta}{m} \sum_x \delta^{x,l} \odot a^{x,l-1}$

Thanks for help.


texts: I only found stanza that has sentence segmentation for hindi. All other available tools do word tokenization but not sentence segmentation. Could someone suggest any other available tools for sentence segmentation for Hindi


texts: How to improve the accuracy of a Doc2Vec model (Gensim) in case of a toy-sized data set?

texts:
I'm building an NLP question-answering application using Doc2Vec technique in gensim package of Python. My training questions is very small, only 20 documents and I am getting very inaccurate and different similarities even for same document while running at multiple instances. Almost all the sources which I referred trained data set containing thousands of documents. So I infer the reason behind my model's inaccuracy is the size of my data set.

Is there any way to improve the similarity between documents, maybe by changing parameters or feature engineering? If yes, what are those parameters and by what ratio should I change them? If no, what are other ways or perhaps other neural network models to tackle the problem?


texts: Neural Net Accuracy: Test Set vs Real World Data

texts:
Neural Net accuracy is high on test set but low on new real world image examples.

Looking for advice regarding what generally causes this scenario and how to fix it.

Sampling basis? Training/test set is not representative of real world data? Obtain more training/test data?


texts: Why does horizontal lines in plt.plot(feature, '.') mean that the data have been properly shuffled?

texts:
I am following a Mooc and in this lecture about visualisation in explenatory data analysis the lecturer claims that when plotting the row indexes against feature values, if we have lines on the feature value axis it means that the data have been properly shuffled. I can't see why.

Shouldn't an index have only one value in the feature axis?
One horizontal line should mean that the feature values for all indexes have been uniformized, not randomized?

On the contrary, in the following lecture, the lecturer claims that from the absence of vertical lines, the data hasn't been properly shuffled:

I think I get it as if it was, I would have seen clear lines. But how can I bee sure there isn't more classes hidden in these subs?


texts: Optimizing the non-differentiable minimum in tensorflow

texts:
I have the following loss function

and I optimize it using SGD in tensorflow. The $\phi$ is a soft approximation of the minimum since the minimum is not differentiable, I used the following approximations:

where k is a negative integer I also tried the formulas here,
yet the model converges very quickly to a local minimum.

Is there any other way to calculate the minimum?


texts: Are CNNs best approaches to find and count objects in images?

texts:
I'm starting my first real data-science project, I made a reserach and want to ask if my approach is correct:

I HAVE:
600 photos of electronic components, hundreds of components in a single photograph, components shape and distance between them may vary between photographs but there is only 1 shape in a single photo. components shapes are squares, rectangles, ovals, "E"-shape and similar

I NEED: generic counter for components (no matter what is the type of elements on the photo)

PLANNED APPROACH: analyze each picture with openCV to gain predictions and then train CNN with Tensorflow and GPU (no previous experience).

is it the correct way of thinking?


texts:
I am working in a deep learning project for image classification. I am using Vgg16 to create the model and the dataset has 100 classes. The testing accuracy is 98.9% and loss is 0.1731

And I got the following ROC and Average Precision curves:

I do not now if the ROC and Average Precision result are good results or there is a problem in the curves because the curves are perfect. Do you know why I got ROC =1? Is there any problem in ROC and average precision?


texts: What is the tag mapping for entity recognition in nltk?

texts:
When doing entity recognition using NLTK, one gets as a result a Tree with a bunch of words mapped to tags (eg. Mark -> NNP, first -> JJ, ...). It's not at all clear what all the tags stand for at first glance and I was unable to find any documentation about these tags in the NLTK docs.

>>> from nltk import word_tokenize, pos_tag, ne_chunk
>>> sentence = "Mark and John are the first to work at Google from one years old in 39 years."
>>> print ne_chunk(pos_tag(word_tokenize(sentence)))
(S
  (PERSON Mark/NNP)
  and/CC
  (PERSON John/NNP)
  are/VBP
  the/DT
  first/JJ
  to/TO
  work/VB
  at/IN
  (ORGANIZATION Google/NNP)
  from/IN
  one/CD
  years/NNS
  old/JJ
  in/IN
  39/CD
  years/NNS
  ./.)

I ended up looking into the source code to get the mapping. Posting in case anyone else runs into the same problem.


texts: Any differences in regularisation in MLP between batch and individual updates?

texts:
I have just learned about regularisation as an approach to control over-fitting, and I would like to incorporate the idea into a simple implementation of backpropagation and Multilayer perceptron (MLP) that I put together.

Currently to avoid over-fitting, I cross-validate and keep the network with best score so far on the validation set. This works OK, but adding regularisation would benefit me in that correct choice of the regularisation algorithm and parameter would make my network converge on a non-overfit model more systematically.

The formula I have for the update term (from Coursera ML course) is stated as a batch update e.g. for each weight, after summing all the applicable deltas for the whole training set from error propagation, an adjustment of lambda * current_weight is added as well before the combined delta is subtracted at the end of the batch, where lambda is the regularisation parameter.

My implementation of backpropagation uses per-item weight updates. I am concerned that I cannot just copy the batch approach, although it looks OK intuitively to me. Does a smaller regularisation term per item work just as well?

For instance lambda * current_weight / N where N is size of training set - at first glance this looks reasonable. I could not find anything on the subject though, and I wonder if that is because regularisation does not work as well with a per-item update, or even goes under a different name or altered formula.


texts: Frequency of occurrence - dummy variables

texts:
I am thinking about it not the first time, namely if I have a variable that I want to convert later to the variable dummy (cities in this case), should I delete lines that occur less often than N times?

For example, the value of new york has occurred 400+ times but there are cities that only appeared once or twice.

What should I do with values that have appeared only once or twice?

print(df[cities].value_counts())

Output:

city1         424
city2         107
city3          35
city4          33
city5          28
city6          24
city7          15
city8           7
city9           4
city10          3
city11          2
city12          1
city13          1
city14          1
city15          1
city16          1
city17          1