texts:
I have set of documents as follows where each document has set of words that represents the content of it.

Doc1: {fish, moose, wildlife, hunting, bears, polar}
Doc2: {energy, fuel, costs, oil, gas}
Doc3: {wildlife, hunt, polar, fishing}

So, if I look at my documents I can deduce that Doc1 and Doc3 are very much similar.

I want distance metrics for bag-of-words. I followed some tutorials in Gensim about how to do it. However, as I understand, initially they train a model and then use that model to calculate the Hellinger Distance. However, in my case, I do not have any training data. Hence, please let me know how to achieve this with no training data.


texts: What is difference between one hot encoding and leave one out encoding?

texts: I am reading a presentation and it recommends not using leave one out encoding, but it is okay with one hot encoding. I thought they both were the same. Can anyone describe what the differences between them are?


texts: When a relational database has better performance than a no relational

texts:
When a relational database, like MySQL, has better performance than a no relational, like MongoDB?

I saw a question on Quora other day, about why Quora still uses MySQL as their backend, and that their performance is still good.


texts: How distribution of data effects model performance?

texts:
I am working on House Prices: Advanced Regression Techniques dataset. I was going through some kernels noticed many people converted SalePrice to log(SalePrice) see below:

I can see that taking a log transform reduced the skewness of data and made it more normal-like. But, i was wondering will it improve my model's performance or is use-full in any way. If it is then how a normal distribution of target variable was a catalyst?


texts: Handling a feature with multiple categorical values for the same instance value

texts:
I have data in the following form:

table 1
id, feature1, predict
1, xyz,yes
2, abc, yes

table2
id, feature2
1, class1
1, class2
1, class3
2, class2

I could perform a one many join and train on the resultant set- which is one way to go about it. But If I rather wanted to maintain the length of the resultant set equal length of table 1, what is the technique?


texts:
I have a problem where I have a list of n words with truly k different ones (k is unknown) because some may be malformed or contracted. I would like to automatically cluster them.
I thought about using something like a dendogram with a distance between words (I don't know which), how can this be handled efficiently?


texts: Can we predict when an event will occur in the future from time series data?

texts:
I would like to predict a few possible times when a particular event may occur. For instance, I have the daily activity data of a person that consists of what the person doing and when over a period of time. Currently, my data are like the following format:

Date    Time    Activity

01-01-2017  08:23:30    Breakfast

01-01-2017  16:20:30    Reading

01-01-2017  19:00:00    Dinner

02-01-2017  08:00:10    Breakfast

02-01-2017  17:40:30    Reading

02-01-2017  19:30:00    Dinner

03-01-2017  08:15:30    Breakfast

03-01-2017  16:20:30    Reading

03-01-2017  20:30:00    Dinner

Let's just say, in most of the cases, the person takes his dinner around 7-7:30 pm and sometimes at 8-8:30 pm, read books around 4 pm mostly and sometimes at 5,6 pm too. From this type of data, I would like to predict when he will take breakfast, dinner or read tomorrow. For instance, dinner at 8pm (confidence/probability 65%), or 8:30pm (20%), 9 pm (10%), 9:30pm (5%).

The challenge I am facing is, I can't identify which technique to choose to achieve the goal. Can you please provide me a few hints?


texts: How to handle columns with large/infinte values in dataset for ML classification

texts:
Computed a column using a formula (formula does't involve any log functions, just a group by with .sum()), but as expected this column would result in infinite/exponential values like below:

-inf
 nan
 inf
-3.000e+32
 7.3297+23 ...etc (similar data)

My doubt is how should I handle this column as I could also not avoid this column for prediction for a binary classification problem. What are the techniques to use before using this column in model classifier.

Thanks!


texts: Stanford parser Python : Combine NER and POS tags

texts:
Hi I am experimenting with stanford parser and NER with python

 Input = "Rami Eid is studying at Stony Brook University in NY"

Parser Output:

NER Output :

[(u'Rami', u'PERSON'), (u'Eid', u'PERSON'), (u'is', u'O'), (u'studying', u'O'), (u'at', u'O'), (u'Stony', u'ORGANIZATION'), (u'Brook', u'ORGANIZATION'), (u'University', u'ORGANIZATION'), (u'in', u'O'), (u'NY', u'O')]

Now can I combine NER results with Parser result ?

So that

(u'Rami', u'NNP'), (u'Eid', u'NNP') ==> u('Rami EID', u'PERSON')

(u'Stony', u'NNP'), (u'Brook', u'NNP'), (u'University', u'NNP') ==> (u'Stony Brook University',u'ORGANIZATION')

will get replaced in the graph

How this can be done ?


texts: What is the definition of knowledge within data science?

texts:
"Knowledge" is crucial within several fields like Knowledge Discovery, Knowledge Distraction, Natural Language Processing, Data Mining, Big Data, etc etc etc.

What is the definition of knowledge within these fields?

Is there 1 common definition, or does it depend on the exact context?


texts: How prevalent are traditional CS questions in data science interviews?

texts:
I've started interviewing for data science roles and I've noticed that a lot of companies ask traditional computer science questions. I have no formal CS education. In fact, I was a business major in college which is pretty much the opposite. I don't know anything about traditional CS algorithms like binary search, bubble sort, or dynamic programming.

I have a very strong background in machine learning / data science but pretty much everything I know is through years of Googling, doing interesting ML side projects, Kaggle, and generally learning things the hard way. I know that a data scientist needs to be able to code. Ironically I'm a professional software engineer and I code all day. I just don't have a theoretical background. Do I need to read something like Introduction to Algorithms and brush up on CS undergrad work before I start applying to data science jobs in earnest?

Please note: I know that there are a lot of other questions on here about what to study to become a good data scientist, but this question is subtly different. I personally disagree that knowledge of traditional CS algorithms is relevant for most day-to-day data science work and that knowing these things makes you a better data scientist.

Most machine learning algorithms (gradient boosting, random forests, linear models, SVMs, neural nets, etc) are available in easy-to-use libraries like caret (R); scikit-learn, and TensorFlow (Python); or H2O, and MLlib (Scala/Java). There are also easy-to-use tools, like Spark, to make these algorithms scale. I feel that understanding how the learning algorithms (like gradient descent) work is relevant, but I don't agree that recursion or dynamic programming is relevant. Am I wrong? Should I be more open minded?


texts:
I've written a 2 layer Neural Network in Python for binary classification. The first layer uses ReLU activation, and the output layer is sigmoid activation. I'm using cross entropy to calculate loss. I've got the following for the forward pass:

$Z^{[1]} = W^{[1]} . x + b^{[1]}$

$A^{[1]} = max(0, Z^{[1]})$

$Z^{[2]} = W^{[2]} . A^{[1]} + b^{[2]}$

$A^{[2]} = \sigma(Z^{[2]})$

Something seems to be wrong in my backward pass for calculating the coefficients:

$L = -y*log(A) - (1-y) * log (1-A)$

$\frac{\partial L}{\partial W^{[2]}} =
\frac{\partial L}{\partial A^{[2]}}.
\frac{\partial A^{[2]}}{\partial Z^{[2]}}.
\frac{\partial Z^{[2]}}{\partial W^{[2]}}=
(A^{[2]} - Y).A^{[1]}$

$\frac{\partial L}{\partial b^{[2]}} =
\frac{\partial L}{\partial A^{[2]}}.
\frac{\partial A^{[2]}}{\partial Z^{[2]}}.
\frac{\partial Z^{[2]}}{\partial b^{[2]}}=
(A^{[2]} - Y)$

$\frac{\partial L}{\partial b^{[1]}} =
\frac{\partial L}{\partial Z^{[2]}}.
\frac{\partial Z^{[2]}}{\partial A^{[1]}}.
\frac{\partial A^{[1]}}{\partial Z^{[1]}}.
\frac{\partial Z^{[1]}}{\partial b^{[1]}} =
(A^{[2]} - Y) * W^{[1]}*relu'(Z^{[1]})$

$\frac{\partial L}{\partial W^{[1]}} =
\frac{\partial L}{\partial Z^{[2]}}.
\frac{\partial Z^{[2]}}{\partial A^{[1]}}.
\frac{\partial A^{[1]}}{\partial Z^{[1]}}.
\frac{\partial Z^{[1]}}{\partial W^{[1]}} =
(A^{[2]} - Y) * W^{[1]}*relu'(Z^{[1]})*A^{[0]}$

Here is my code implementation, it stops at the weight update due to a broadcasting error:

def initialize_coefficients(layers):
    L = len(layers)
    weights = [np.random.randn(layers[l], layers[l - 1]) * 0.01 for l in range(1, L)]
    biases = [np.zeros((layers[l], 1)) for l in range(1, L)]
    return weights, biases

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def relu(x):
    return np.maximum(0, x)

alpha = 0.01
X, y = load_dataset()
weights, biases = initialize_coefficients([4, 30, 1])

for i in range(10000):
    # Feed forward
    Z1 = np.dot(weights[0], X) + biases[0]
    A1 = relu(Z1)
    A2 = sigmoid(np.dot(weights[1], A1) + biases[1])
    loss = - (y * np.log(A2) + (1-y) * np.log(1 - A2))
    print(1/y.shape[0] * np.sum(loss))

    # Backpropagate
    dL_dW2 = np.dot(X, (A2 - y).T)
    dL_dB2 = A2 - y

    dL_dW1 = (A2 - y) * weights[0] * relu_derivative(Z1) * X
    dL_dB1 = (A2 - y) * weights[0] * relu_derivative(Z1)

    # Gradient descent
    weights[0] -= alpha * dL_dW1.T
    weights[1] -= alpha * dL_dW2
    biases[0] -= alpha * dL_dB1.T
    biases[1] -= alpha * dL_dB2.T

Where are the errors in this network?


texts:
I have a profile hidden Markov model that I use to identify all instances of a user-defined pattern of symbols in a long sequence of symbols. I use the Viterbi algorithm to find the most probable path that generates that long sequence of symbols, and it all works very well to identify the user's pattern. But I'm now interested in extending this, and identifying the k-most probable paths through my model that would generate some long sequence. Is there an existing algorithm to do this? I've come across something called the "1-best" or "k-best" algorithm, but I can't find much of anything describing it.

I've also considered what I'm sure would be a heuristic, by finding the most probable path and then returning the neighbourhood around it. This would be found by iterating over each move between states in the most probable path, setting that particular step to 0, and resolving using Viterbi. This would obviously increase the run-time by a factor of (length of the path ~= length of pattern), which I suspect will be reasonable for my purposes.

Has anybody come across something like this, or can anybody see anything obviously wrong in my neighbourhood idea? I'd appreciate any feedback. As a note, this has been cross-posted to the Theoretical Computer Science forum.


texts: Loss Function - Decreasing a lot at beginning of the epoch

texts:
I noticed something when looking at the output of the verbose. When I train my model, in the early part of the epoch (first 20 %), the loss is decreasing a lot. And then in the rest of the epoch (last 80%), the loss is very stable and doesn't change that much until the next epoch. It does the same thing.

I build a model that is training a kind of large dataset (60000 entries). I am using Keras and Tensorflow and my model is just a simple regression model with conv2d and dense layers. I am trying to get the best loss function possible. The loss function that I am using is just a simple Mean Squared Error (MSE). I am also using Adam optimizer.

I don't understand how the loss is decreasing a lot at the beginning of the epoch and less for the rest of the training. Should I reduced the size of my database ? Is it considered overtraining then ?

Is there a way to speed up this process (like early stopping but just for the epoch) ? As I see it most of the training is done for the first part of the data. Maybe I am wrong.


texts: Can you give me a idea about image processing segmentation algorithm invariant to light that can be implemented in Opencv in Jupyter Notebook?Please!


texts: R Script to generate random dataset in 2d space

texts:
I want to analyze the effectiveness and efficiency of kernel methods for which I would require 3 different data-set in 2 dimensional space for each of the following cases:

BAD_kmeans: The data set for which the kmeans clustering algorithm
will not perform well.
BAD_pca: The data set for which the Principal Component Analysis
(PCA) dimension reduction method upon projection of the original
points into 1-dimensional space (i.e., the first eigenvector) will
not perform well.
BAD_svm: The data set for which the linear Support Vector Machine
(SVM) supervised classification method using two classes of points
(positive and negative)  will not perform well.

Which packages can I use in R to generate the random 2d data-set for each of the above cases ? A sample script in R would help in understanding


texts: What is the correct file extension for Orange image embedding?

texts:
I am using Orange data mining in order to create a facial recognition workflow with some prototype widgets. To do this, I am trying to import some images (test data) using the import widget.
My question is: which file format do I need to use in order for the image data to be compatible with the import widget? In other words, which file format should I use to upload the image?
I have tried using JPEG but when I click "browse for images" to search for images on my desktop, it doesn't allow me to select anything to import.
Does this imply that it is not compatible? Is there a specific file format that I should be using in order to do the image detection/embedding process?
Thanks.


texts:
I have been trying around the EXTERNAL table concepts in HIVE

CREATE EXTERNAL TABLE IF NOT EXISTS MovieData
(id INT, title STRING,releasedate date, videodate date,
URL STRING,unknown TINYINT, Action TINYINT, Adventure TINYINT,
Animation TINYINT,Children TINYINT, Comedy TINYINT, Crime TINYINT,
Documentary TINYINT, Drama TINYINT, Fantasy TINYINT,
Film-Noir TINYINT, Horror TINYINT, Musical TINYINT,
Mystery TINYINT, Romance TINYINT, Sci-Fi TINYINT,
Thriller TINYINT, War TINYINT, Western TINYINT)
COMMENT 'This is a list of movies and its genre'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;

Created a table using the above statement and then used the LOAD statement to get the data populated.

LOAD DATA LOCAL INPATH '/home/ubuntu/MovieLens.txt' INTO TABLE MovieData;

Next time I DROP the table in HIVE and recreate it again and LOAD the data... But when I do a COUNT operation on the table I get double the values that's present in the file that I loaded.

I read through few articles that EXTERNAL table does not delete the data but the schema alone from the HIVE metastore... External Table

Can you please advise why does HIVE behave this way...


texts: time series reasoning/explanation for prediction value

texts:
I doing prediction for univariate time series model. But I want to know the reason for the prediction value.
Eg: Today sales will be low due to weekend.  It's christmas time our sale will be tripled during this . Our growth will 5% for this year since past years we have steep growth around 4-6%
In Machine learning we can able to  get the reasoning for the prediction for the algorithms like random forest. Is it possible for the same here
Same way I want this for multivariate time series too


texts: Why 100% accuracy on test data is not good?

texts: I was asked this question in an interview and wasn’t able to give a satisfactory answer not only upto the interviewers' expectations but of my own as well. The question was as above only, he later gave an example as if why if my model predicted the prices of oil of tomorrow 100% accurately why that might be bad or why having a model 100% accurate bad or is it? Is there something in the question or is there a deeper explanation?


texts:
Is there any algorithm/method/system/application that combine all predictive methods into one?

so for users instead of deciding which method they should use, is there any platform that you just feed the data and it gives you the predictions without the decide which method of predictive analysis you want to use?


texts:
I'm solving a system of equations in excel on a rolling data series. In each row is the data for a matrix that I need to invert, but I'm having a difficult time creating the matrix inline. I will do calculations based on the matrix after I create it. It needs to be inline so I can get the inverse in the next row and all subsequent rows.
This is what I'm trying ... and every other combination I can think of, but it fails.
=MINVERSE({I271:L271;M271:P271;Q271:T271;U271:X271})

How do I create a matrix based on this data? i:l is the column, i,m,q,u are the rows.


texts:
Is there some interesting work on modeling relationship between objects in images? This seems like a natural extension to object detection/segmentation, but I couldn't find anything well-cited.

I'm looking for something like the following: a convnet predicts objects in the image (e.g. 'person', 'house', 'dog'). Each objects is examined wrt to relationship to other objects, e.g. 'person' 'in front of the' 'house', 'dog', 'to the left of the' 'person'.

Here the network must understand at least three semantic objects, but also learn to connects them in some sense with additional relationship objects like 'in front of', 'next to', etc. These relationship objects should probably also be pre-defined, like the classes. I can imagine learning some sort of graph edges, but not sure.


texts: Orange: Data has no target variable error

texts:
I am trying to apply Random Forest algorithm on a data set using Orange. The target variable is not set in the data set. However, I know which column is the target variable.

How can I specify the target variable in a .csv file using Orange or any other tools?


texts: How to find average of Duplicate Column Headers in Python dataframe


texts: Parameter of Conditional Gaussian Distribution

texts:
I'd like to understand how to determine the parameter of conditional gaussian distribution. Following is the network architecture of VUNET which learns the conditional gaussian distribution $q(z|x, \hat y)$ of appearance of human($z$) upon given two conditional inputs skeleton of body pose and groundTruth image($x$) of 128x128 resolution image

I don't understand how $2\times2\times256$ vector and
$1\times1$ vector can characterize  $q(z|x, \hat y)$. Any advice to understand this point clearly?


texts:
I'm trying to use L1 regularization to select features in XGBoost classifier. However, I don't see any example code on how to specify the penalty of l1.

This is how I do in sklearn's LogisticRegression.

C = [10, 1, .1, .001]

for c in C:
    clf = LogisticRegression(penalty='l1', C=c)
    clf.fit(X_train, y_train)
    print('C:', c)
    print('Training accuracy:', clf.score(X_train, y_train))

How should I specify penalty and C in XGBoost?


texts: Can Random Forest regressor or Decision trees handle missing values and outliers?

texts:
I have below assumptions about RF & Decision trees in general, please correct me if the assumptions are incorrect.

It takes care of missing values
It handles outliers
It handles skewness in the data so the transformation is not required.
Feature scaling is not required
Feature selection is not required.

Please correct if my observations are incorrect as you can see there is a nan value error in the dataset. Also how to rectify it?
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

importing dataset
share3.info()
share2.isnull().sum()

XRD = share2.drop(['Close'], axis = 1)
YRD = share2['Close']

training & test set
from sklearn.model_selection import train_test_split
X_trainRD, X_testRD, Y_trainRD, Y_testRD = train_test_split(XRD,YRD,test_size = 0.2, random_state = 0)

fitting random forest regresson to the dataset
from sklearn.ensemble import RandomForestRegressor
regressor = RandomForestRegressor(n_estimators= 350,random_state = 0)
regressor.fit(X_trainRD, Y_trainRD)

***ValueError: Input contains NaN, infinity or a value too large for dtype('float32')***
```


texts:
I'm a begginer in data science but I tried to build a classifier for my own bank transactions, I collected ~ 50.000 in total. My intention is to create a relationship between the statement of the transaction and the type of transaction. For example:
Statement: Payment with card number XXXXXX in Wallmart.
Type/label: Buy in a supermarket
For doing so I trained my model with 80% of the total of statements (previously labeled) and I reached a so high accuracy (85%). After that I realized that something was wrong, there are so many statements with the text:
Statement: Payment with card number XXXXX in _______ .
Because this is a "default" statement text in my bank when someone buys a product in a supermarket. Then, if the model tries to find out in what category the statement is it will be pretty simple because there will be many statements repeated and the probability of predict one of those will be high. What I did later was trying to delete all the repeated statements, since the repeated ones will not affect in the training process, but because of this my total of statements went down to ~ 2.000, and reaching then only a 10% of accuracy in the prediction.
My questions are: What is correct process to train my model? Should I let the repeated statements? (I feel like I'm cheating the prediction letting the repeated ones) What I'm doing wrong?


texts: Does a model trained on a larger training set created by duplicating records perform better on the test data than the one trained on data without duplicates?


texts: Scoring metric for recommendation system

texts:
I'm working on a project that involves building a news recommendation system. I've come as far as quantifying user interaction with different articles  on the site into user's affinity towards atopic using a bayesian function. I also have quantified the recent articles using LDA into the proportion an article talk about each topic.
my users topic-affinity for a user x looks like this(target-x):
 user_id  interest-topic-0  interest-topic-1  interest-topic-2  interest-topic-3  interest-topic-4  interest-topic-5  interest-topic-6  interest-topic-7  interest-topic-8  interest-topic-9
       0            0.0257            0.2956            0.0386            0.0643            0.1285            0.0000               0.0            0.0257            0.0386            0.1671

My quantified articles looks something like this(vectors-v):
post_id   topic-0   topic-1   topic-2   topic-3   topic-4   topic-5   topic-6  topic-7  topic-8   topic-9
      x  0.055048  0.000000  0.742544  0.032286  0.059630  0.000000  0.000000  0.01173      0.0  0.095441
      y  0.000000  0.051172  0.000000  0.000000  0.158314  0.042632  0.022281  0.00000      0.0  0.720676
      z  0.028615  0.000000  0.020919  0.000000  0.000000  0.018940  0.882862  0.00000      0.0  0.046078

The shape of target will always be (10,)
The shape of vectors will always be (num_articles, 10)
Both vectors do not follow the same distribution.
Now I'm trying to figure out the best way to recommend articles from vectors v, for a target user x, given x and v. I've tried distance similarity functions like cosine similarity to find the distance between vectors. The results are satisfactory but I'm looking for a better function/metric to pick out top n recommendations for a user.


texts:
I was going through GAN's notebook by fchallot on Generative Adversarial Networks where, in the Generator Network, he creates a Dense layer with $16*16 * 128$ (where 128 is the number of channels).

How exactly does latent_dim=32  becomes of shape $16 * 16$ in the network.
How are these values decided?


texts: Should I scale or normalise my dataset before clustering?

texts: So i have a dataset with variables with unit of measurement as milligrams, kgs and quintals. Should i use standard scaler or minmaxscaler to scale the dataset.


texts: What does the below phrase in the lstm blog mean? - Data Science

texts:
I am a newbie to data science. I was reading this blog

When I was half way through, I came into this sentence

  Further, each series of data has been partitioned into overlapping
  windows of 2.56 seconds of data, or 128 time steps. These windows of
  data correspond to the windows of engineered features (rows) in the
  previous section.

What does 'series' means here and what is window partitioning?

Further I would like to ask all the data science enthusiasts there,What do I lack here.Is it comprehension or logical skills needed .What can I do to improve upon this.I encounter such doubts often and what I do is search for the terms I dont understand or leave it and get ahead.When I search for the terms I get broad answers leaving me more confused.If i leave it like that I dont understand the rest.What should I do here?I cant post every small doubts in stack exchange.It would be great If you could share some light on this as well.


texts: What should the size of the decoder output be in a sequence to sequence model

texts:
In a sequence to sequence model, a lot of the tutorials I have read state that the decoder target length should be the same as the encoder input length (https://blog.keras.io/building-autoencoders-in-keras.html, https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html). Often another separate model is built to be used for inference.  How important is it for the accuracy of the model that the decoder target length is actually the same as the encoder input length?

In my setup I have many variable length sequences which I have applied a mask to so that any sequences that have missing values or are simply shorter than the longest sequence length are padded to the length of the longest sequence in the dataset and the padded data are filled with a mask value.  Using the masking layer in keras these masked points get ignored in the update of the weights and loss.

However this means that if I make the decoder target length the same as the encoder input length, it will try and reproduce a shorter series that has been masked and make it the actual input length which could be much greater.  I am not sure if this would cause a problem or not?

I therefore wonder if it would be better to perhaps make the  decoder target length and make it the mode of the series I have?  Or if infact if I know that I always want to predict H steps into the future, why not make the initial model being trained have a decoder target length of H instead of the same length as the encoder input?


texts: What's wrong with RF/SVM with word embedding (GloVe)?

texts: I searched many times in google for examples on word embedding (specifically GloVe) with Random forest and I couldn't find any single example. For GloVe, it was all either LSTM or CNN. Maybe there's something I don't understand, but why only these 2 algorithms are used with GloVe, what's wrong with the other algorithms?


texts: Triangle Pattern Recognition on Financial Market with Python

texts:
I'm working on a personal project to find Triangles on any stock in Python.
I detect the max and min points (shift(-5,+5) because if I consider only shift(-1+1) I have a lot of lines) and write lines between 2 max points and 2 min points.

I should continue with this or I should try with another method?

What I'm trying to get is this:


texts: Are Multi-Input models better than ensembling different models?

texts:
Suppose I have a classification task, with images and lots of useful metadata. I want to utilize both the datasets even though I can't unify them.
In this case, is it much more beneficial to run and ensemble: make a seperate model for images, then a seperate model for the tabular metadata?
OR
Is making a model Multi-input with multiple branches (Like this shown below)

where each branch respresents a fully fledge model for that data, concatenating it in the end, then a simple F.C layer for classification better?

Which do you think theoretically offers more performance? any estimates as to how much increase (5%, 10%) by those who have tried both methods?


texts: Reinforcement Learning on data only (NO emulators)

texts:
My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...).

It is too costly for us to emulate the agent (executing the action and assessing the reward) meaning our only option is to learn the optimal policy on our dataset using an off-policy algorithm. However I still do not know how to do that, Q-learning, while off-policy, still require an emulator.

Could you please provide me some guideline on how to do that? Which type of algorithm should I use?


texts: Advice on how to get started for someone a bit later in life?

texts:
I am 33 years old and graduated in 2006 from college with a degree in analytic philosophy. For a long time I thought that I was going to get a phd in phil and teach but that dream is long gone, and I am intent on having a career in data science/analysis.

I do have a history in mathematics; I finished calculus 3 in high school and finished a set theory course in college as well as advanced logic courses (symbolic, modal, predicate, etc), but I have never studied how to code, nor have I taken any statistics courses.

Are there any suggestions on how to best proceed?  I of course want to be as attractive to employers as possible.  I have no debt or family and am open and willing to take multiple years (and take on loans) studying for this profession.

Would getting a math degree (with an emphasis in statistics and taking courses like linear algebra) be the right (or at least, good) way to go?  A local university offers a concentration in data science within the mathematics department, so I could do that as well. And in addition, taking courses in computer science to learn how to code?

Basically I'm looking to push a reset button (well not entirely since I have a background in mathematics, and I believe my background in philosophy will help too) on my career path and will do whatever it takes.  I want a career with a future and with hopefully opportunity to choose where I work (city), and a career that fits with how my mind works (I believe I could thrive in this field).

Jobs that I have had since college will not help in any way towards future jobs in this field, so I will be fully leaning on future classes/preparation.

Thanks in advance and for reading this partial ramble!  I'm motivated but don't know exactly what I should do.  I realize that there are many paths people take to get into this profession but would love advice for someone like me in my situation.


texts: Understanding ELBO Learning Dynamics for VAE?

texts: As I understand it, I'm basically minimizing the KL Divergence between the Prior and Encoder latent distribution, and the log probability of the decoder distribution. I have a model that does generate some useful latent distribution for classification using this loss function, however, it has weird learning dynamics. My decoder distribution is a 300 dimension Gaussian and my latent is a 40 dimension Gaussian, so in all instances, both terms of my ELBO should vary wildly in order of magnitude. It seems from looking around on the internet, most use cases of VAEs look similar - high dimension decoders and low dimensional encoders, so I would imagine some people see a similar a problem, but I haven't mentioned any note of this in any literature or tutorial. Is there any way to correct for this?


texts:
When you train the word2vec model (using for instance, gensim) you supply a list of words/sentences.  But there does not seem to be a way to specify weights for the words calculated for instance using TF-IDF.

Is the usual practice to multiply the word vector embeddings with the associated TF-IDF weight? Or can word2vec organically take advantage of these somehow?


texts: How to improve a model with a high cross validation score yet with low accuracy on unseen data?

texts:
I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows:

Record raw data from the subjects
For each subject, compute the feature from the raw data (thus, we obtain 100 datasets with 25 features)
Reserve the data of one subject for model validation
Combine the remaining 99 datasets into one large dataset (thus, we obtain a dataset with up to 1 million rows and 25 columns)
The obtained large dataset is used to train a model and evaluate it using a 10-folds cross-validation.
Use the reserved dataset (see step 3) to validate the model obtained in step 5 above

Unfortunately, the model's performance is very confusing to me (is this an overfitting? a data leakage?) :

The cross-validated model achieves 99.9% accuracy and 99.7% recall)
However, when the same model is tested using the validate test set (see step 3), I get a very low accuracy (40.2% recall and 39.8% precision)

What could be the reason for this discrepancy? Any suggestions on how this could be improved?


texts: I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between two documents which don't seem very similar when reading through the document pair. Are there conventional ways of evaluating which words contribute to the high similarity score between two documents using TF-IDF and cosine similarity?


texts: Using a neural network to learn regression in image processing

texts:
I have a camera system with some special optics that warp the field of view of the camera, dependent on two variables, $\theta_1$ and $\theta_2$. Given a specific configuration of these two variables, each pixel on my camera (which is 500x600 resolution) will see a specific coordinate on a screen in front of the camera. I can calculate this for each pixel, but it requires too many computations and is too slow. So, I want to learn a model that fits this function, but computes much faster.

I have plenty of input/output data that I have generated, mapping the 500x600 input points to the 500x600 output points for different $\theta_i$ values, and I have already used some 2D polynomial least squares regression to learn these functions. They perform adequately, but I was wondering if a neural net could be used to learn a better function.

My question comes down to this: can a neural network learn what basically amounts to a regression problem trying to learn $f_{\theta_1,\theta_2}(px_1,px_2)=(a_1,b_1)$?

I know that neural nets excel in classification problems, which this is not, but I have also heard that a neural network can "learn arbitrary functions."


texts: Is TensorFlow a complete Machine Learning Library?

texts:
I am new to TensorFlow and I need to understand the capabilities and shortcomings of TensorFlow before I can use it. I know that it is a deep learning framework, but apart from that which other machine learning algorithms can we use with tensor flow. For example can we use SVMs or random forests using TensorFlow? (I know this sounds crazy)

In short, I want to know which Machine Learning Algorithms are supported by TensorFlow. Is it just deep learning or something more?


texts: Correlation based Feature Selection vs Feature Engineering

texts:
I'm a bit confused about the superiority of Feature Selection over Feature Engineering or vice versa.

Let's say I just want to get the best possible performance on a couple of models like a neural network, something tree-based and a Naive Bayes Classifier.

Before starting any training I looked at my features and engineered some additional (hopefully) even more expressive features. I did this from a domain expert point of view. For instance I added a new ratio feature C = A / B because I think this will be a very expressive information for the model.

Furthermore I added several features measuring basically the same thing but in different ways. Lets say a feature D measuring the the length of any text including empty lines and an additional feature E measures the length of any text excluding empty lines.

So this leads to very many features in my data set with also very high correlation / multi-collinearity. (of course D and E are very high correlated and A, B and C are also high correlated / multi-collinear.

So any kind of correlated based feature selection (between the features) would remove a lot of the engineered features, but could the removal provide the model any kind of better discriminative power with just less information? What helps the model more, keeping all features or removing correlated ones?


texts:
If I applied PCA on feature vectors and then I do clustering, such like following:

reduced_data = PCA(n_components=2).fit_transform(data)
kmeans = KMeans(init='k-means++', n_clusters=n_digits, n_init=10)
kmeans.fit(reduced_data)

The reduced data will be the in terms of PCA components, so after
clustering in kmean, you can get a label for each point
(reduced_data), how to know which one from the origin data?
how to play with a number of PCA components regarding the number of
clusters? Thanks.


texts: Non-parametric regression on set of time series: One model for each or one for all series?

texts:
Let's say I have a set of 1D time series which values have been samples in equip-distant time steps with timestamps $1,2,3,...$, they have all  the same lengths and are somewhat similar in shape. I want to apply non-parametric regression (e.g. with Gaussian Processes or Kernel Regression) on the time series in order to infer values for timestamps that are between sample timestamps (e.g. $5.3$).

The obvious way of doing this would be to simply build a regression model for each time series. Overfitting, if a single regression model is build for each time series, can be avoided by using established techniques in using the right smoothing parameter in the respektive kernels.

Is there some game-changing merit in taking the entire time series data and applying the regression model to the entire time series dataset? So far I only see the disadvantage of over-smoothing (aka underfitting).


texts: How to train an existing word2vec gensim model on new words?

texts:
According to gensim docs, you can take an existing word2vec model and further train it on new words.

  The training is streamed, meaning sentences can be a generator,
  reading input data from disk on-the-fly, without loading the entire
  corpus into RAM.

  It also means you can continue training the model later:

        model = Word2Vec.load("word2vec.model")
        model.train([["hello", "world"]], total_examples=1, epochs=1)

Source: docs

But when I actually try it, it doesn't seem to learn the new terms.

from gensim.models import Word2Vec

# initial a model
model = Word2Vec([["cat", "say", "meow"], ["dog", "say", "woof"]], min_count=1)

# count terms in model
print( len(model.wv.vocab) )

=> 5

# train existing model on new terms
model.train([['potoatoes', 'and', 'farmers']], total_examples=model.corpus_count, epochs=model.epochs)

# count terms in model
print( len(model.wv.vocab) )

=> 5

After adding new terms in the 2nd code block, the model still only has the same number of terms as before.

How can I make this work?


texts: R: Comparing dissimilarity between metabolic models with discrete wavelet transformation

texts:
I’m working on comparing bacteria metabolic models. Each model has a set of metabolites (around 2000) and their concentration for 200 time points. I’m in the process of comparing the models to cluster them based on their similarity. One method I followed is I did a pair wise comparison for each of the metabolite pairs in two models using Euclidean distance. Below is how my data look like. This is a sample data file.

I computed pair wise Euclidean distance for Met1 from Model A and Met1 from Model B. Likewise computed the distances for all the common metabolites between the 2 models (Met4 in Model A and Met4 in Model B) and summed up the distances to get a distance (dissimilarity) between the two models. Similarly I computed the dissimilarity matrix for all the models and I used hierarchical clustering to cluster them.

As mentioned above, now I want to compute the dissimilarity of the models using Discrete Wavelet Transformation as my distance measure. I would like to know how to use Discrete Wavelet Transformation to compute a dissimilarity distance between 2 time series and hence for my models.

Previously I used DWT as a distance measure with DBSCAN for clustering metabolites in one model according to their behavior. It worked fine.


texts: How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

texts:
I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I am trying to conduct a (modified) student's t-test on these models. I am able to conduct one (according to THIS link) where I compare only ONE variable common to only TWO models. I want to perform a (or multiple) t-tests with MULTIPLE variables and MULTIPLE models at once.

As mentioned, I can only perform the test with one variable (let's say F-measure) among two models (let's say decision table and neural net).

Here's the code for that. I am performing a Kolmogorov-Smirnov test (modified t):

from matplotlib import pyplot
from pandas import read_csv, DataFrame
from scipy.stats import ks_2samp

results = DataFrame()
results['A'] = read_csv('LMT (f-measure).csv', header=None).values[:, 0]
results['B'] = read_csv('LWL (f-measure).csv', header=None).values[:, 0]
print(results.describe())
results.boxplot()
pyplot.show()
results.hist()
pyplot.show()

value, pvalue = ks_2samp(results['A'], results['B'])
alpha = 0.05
print(value, pvalue)
if pvalue > alpha:
    print('Samples are likely drawn from the same distributions (fail to reject H0)')
else:
    print('Samples are likely drawn from different distributions (reject H0)')

Any ideas?


texts: How to save google colab notebook outputs and variables

texts:
I want to save variables and outputs in text file with colab!

Any bright idea?


texts:
I've created a Python code that reads the data from an excel file using Pandas.

Code for your reference:-

import pandas as pd

def myFunc():
   file = r"C:\Documents\myFile.xlsx"
   new_dataframe = pd.read_excel(file,'Sheet1')
   new_dataframe.fillna(value="No Data Found",inplace=True)
   print new_dataframe

myFunc

Current Output:-

             Name        date    amount_used
0             P1   2018-07-01          40.0
1             P1   2018-07-01          40.0
2             P1   2018-07-15          40.0
3             P2   2018-08-01          20.0
4             P2   2018-09-15          50.0
5             P2   2018-08-15          40.0
6             P3   2018-08-10          20.0
7             P3   2018-08-10          50.0
8             P3   2018-08-10          40.0

In the final output, I need to sum the amount_used column based on Name and date column.

Expected Output:-

             Name        date    amount_used
0             P1   2018-07-01          80.0
1             P1   2018-07-15          40.0
2             P2   2018-07-01          20.0
3             P2   2018-08-15          90.0
4             P3   2018-08-10         110.0

How can I achieve this using pandas ?


texts: How do I build an image dataset for CNN?

texts: I don't understand how images are actually fed into a CNN. If I have a directory containing a few thousand images, what steps do I need to take in order to feed them to a neural network (for instance resizing, grey scale, labeling, etc)? I don't understand how the labeling of an image works. What would this dataset actually look like? Or can you not look at it at all (something like a table)?


texts: Model selection for a "set of linear models"?

texts:
I would appreciate your input on which predictive ML model(s) could fit our dataset the best.

The primary features of the dataset are x,y (continous) and c (factor with around L=500 unordered levels). The secondary features (~20) are both continous and factors.
We know that the lowest-order correlation is x~y.
We also know that modeling x~y for each level in c improves the fit significantly.

We could of course create L independent linear models, but would like to explore a single model where these sub-linear relationships are partially correlated. The reason being that some levels of c have relatively low statistics and would benefit from the global x~y relationship. Also, we would like a single model when extending to secondary features.

Any suggestions are welcome. (In case of specific libraries, R is preferred.)


texts: How to create separate nodes for each column in a spreadsheet using Neo4J?

texts:
I have an excel file containing lot of columns. I want to create a graph database in Neo4J with all these columns as a separate node so that I can establish relationship between them and play around with the cypher query to get to know my data quite well. I did write a cypher query to make each column as a node, but when I run the relationship query it shows me the name of one node and gives a random number for all the related nodes despite changing the captions in the properties section.

Sample Create statement:

LOAD CSV WITH HEADERS FROM "File Location" AS row CREATE (:NodeName {DATE_OCCURED: row.Date});

Query to visualize the relationship between nodes:

MATCH (a)-[:`REL NAME`]->(b) RETURN a,b LIMIT 25;

This gives me the values the node "a" and random numbers for all the node "b". I don't know where I am going wrong. I thought it has something to do with the way the data has been set up in my file. So I created a file for each column of my main file and wrote the create statements to get the nodes. I still don't get the actual values of the related nodes.


texts:
I trained my CNN (model) classifier and want to check it on some new images. I have image x, so this syntax works for me for one image:
torch.argmax(model(x))

What if I want to classify 2 more images (different classes),  let's say images y and z? Should I for every image write a new line or is it possible to put in the code above 3 together?


texts: Input pipeline with an autoencoder and tf.data

texts:
I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...))
def categorical_normalization(dataframe, name):
  dummies = pd.get_dummies(dataframe[name])
  for x in dummies.columns:
    newname = f"{name}-{x}"
    dataframe[newname] = dummies[x]
  dataframe.drop(name, axis=1, inplace=True)

Therefore the process is taking too much ressources. After few research, I found tf.data.
I've created a preprocessing layer to encode categorical features and normalize the dataset.
def preprocess_layer(dataset, feature_layer, normalizer_layer):
  dataset = dataset.map(lambda x, y: (feature_layer(x), y))
  norm = normalizer_layer(dataset)
  dataset = dataset.map(lambda x, y: (norm(x), y))
  return dataset.batch(batch_size)

I tired to pass x_train (tf.data.Dataset) in the autoencoder model.fit(x = x_train, y= x_val) its not working
I also did dataset.map(lambda x, y: x, x) so it matches
Even inside the autoencoder model
# Autoencoder class
def call(self, x):
    preprocessed_x = self.preprocess(x)
    encoded = self.encoder(preprocessed_x)
    decoded = self.decoder(encoded)
    return decoded

Got the same error
ValueError: logits and labels must have the same shape ((None, 33) vs (None, 1))
And I receive this error with this code :
def model(preprocessing_head, inputs):
  body = AutoEncoder(33)

  preprocessed_inputs = preprocessing_head(inputs)
  result = body(preprocessed_inputs)
  model = tf.keras.Model(inputs, result)

  model.compile(loss=tf.losses.MeanSquaredError,
                optimizer=tf.optimizers.Adam())
  return model

anomaly = model(preprocessing_model, inputs)
anomaly.fit(x=features_dict, y=features_dict, epochs=10)

 ValueError: Found unexpected keys that do not correspond to any Model output: dict_keys([*features_keys...*]). Expected: ['auto_encoder_22']

How to preprocess the dataset and train it with the autoencoder


texts: Gradient Boosted Decision Trees How to Find Prediction of Each Tree?

texts:
I'm doing a project. I have a classification problem that I should solve using gradient boosted decision trees. What I want to do is create a matrix that gives prediction of each decision tree for each sample. For example if I have 100 samples and 100 trees, I should have 100x100 matrix. i, j th entry gives the prediction of jth tree for ith sample.

I'm using sklearn and problem is I can't get prediction by each tree.

So far I tried:

newgb=gb.estimators_[0][0].fit(X_train, y_train)
print(newgb.score(X_train, y_train))

where gb is already a fitted model. What I understood from documentation of sklearn

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor.staged_predict

  .estimators_

should return (number-of-trees x 1) matrix, each entry contains a tree that used by our model. By gb.estimators_[0][0] I tried to access to the first tree, and predict it with score. What I get as output is:

[0.12048193 0.95       0.95       0.95       0.95       0.95
 0.95       0.95       0.95       0.95       0.12048193 0.95
 0.95       0.95       0.12048193 0.12048193 0.12048193 0.12048193
...]

None of them are 1 or 0, like it should be(it is binary classification) and values repeat themselves like 0.95 and 0.12. I didn't use any likelihood function either so

  .score()

supposed to give me only 1's and 0's.

I don't know how to get predictions for each individual tree. I don't know what I do wrong either.


texts:
Since the first Epoch of the RNN, the loss value is being outputted as nan.

  Epoch 1/100
  9787/9787 [==============================] - 22s 2ms/step - loss: nan

I have normalized the data.

    ...,
    [9.78344703e-01],
    [1.00000000e+00],
    [9.94293976e-01]]])

  Example of my X_train (float64 of size (9787,60,1))

-

array([6.59848480e-04, 6.98212803e-04, 6.90540626e-04, ...,
   1.00000000e+00, 9.94293976e-01, 9.95909540e-01])

  Example of my y_train (float64 of size (9787,))

My RNN:

# Initialising the RNN
regressor = Sequential()

# Adding the first LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True, input_shape =
(X_train.shape[1], 1)))
regressor.add(Dropout(0.2))

# Adding a second LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a third LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a fourth LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.2))

# Adding the output layer
regressor.add(Dense(units = 1))

# Compiling the RNN
regressor.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the RNN to the Training set
regressor.fit(X_train, y_train, epochs = 100, batch_size = 32)


texts:
I am trying to build a multi-label classifier for suggesting tags on blog posts. The textual data is full of noise. The approach I have been following until now was a BOW approach with Tf-idf weighting. However I could only get an accuracy of around 0.35 on the test set using OnevsRest approach and a SVC. I then decided to eliminate the noise and I applied TextRank on my data to obtain a more meaningful summary and then applied the same BOW approach which however resulted in further loss of accuracy. I am quite new to machine learning and data science in general. Can someone please guide me with the approach? I am looking specifically on how to create my features and what else can I try.
I have 12 labels and around 1500 blog posts. I know it isn't much data, but the data-set is still well-balanced and won't be the primary reason for such low accuracy.


texts: Since is not possible test all the possible combination, what is the correct procedure to follow on building Machine Learning?

texts:
Sorry, I'm a little confused and this is a general question. How can I be sure that the procedure that I am following is the correct one? Following the steps for building a machine learning model, we start by analyzing the data and then by prepare that data (among other steps).
In the data preparation, we have various options. For example, if the data have missing values, I can choose to remove those values or replace them (among other techniques). How can I be sure what is the “correct” technique to follow? I mean I can always use an algorithm for getting an initial accuracy and then I can compare the changes that I will do to that accuracy. However, can I rely on that accuracy? The same happens with feature selection. I cannot be sure among all the techniques witch one is the best.

The question is the same if we talk about choosing a model and then make the hyper parametrization of that model. How can I be sure that the model that I choose is the correct one? Because without the hyperparameters tunning can be better for one specific model but after the hyperparameters tunning(in all of them) I can have better accuracy in another one. For example, between SVM and KNN without tunning the accuracy is better in SVM, so I will choose that one and then make the tunning. However, if I try tunning on both models maybe KNN will be better than SVM.

In short, my question is, they’re a lot of possible combinations between all the steps to build a machine learning model. I know it's practically impossible to test all these combinations (computation power, and so on), however, it is correct to rely on an “ephemeral” accuracy? Or there is a “correct” procedure to follow?

I also read that there is a dummy classifier for getting a baseline accuracy. Should I use that method to evaluate, for example, different data preparing methods? I mean, I can obtain a baseline accuracy. Then use a random ML algorithm and compare it with that baseline accuracy, and make changes step by step. But I have the same problem because some types of algorithms may be good with specific types of data preprocessing, and I am not testing all the hypotheses.

Thanks in advance, and sorry for my English because it’s not my maternal language.


texts:
Can you please show the step by step calculation of Entropy(Ssun)?  I do not understand how 0.918 is arrived at.

I tried but I get the values as 0.521089678, 0.528771238, 0.521089678 for Sunny, Windy, Rainy.

I was able to calculate the target entropy (Decision) correctly as = -(6/10)*log(6/10) + -(2/10)log(2/10) + -(1/10)log(1/10) + -(1/10)log(1/10) = 1.570950594

I am totally stuck at the next step.  Request your help.

Reference: http://www.doc.ic.ac.uk/~sgc/teaching/pre2012/v231/lecture11.html
Please search for "The first thing we need to do is work out which attribute will be put into the node at the top of our tree:" to reach the line I am referring to.


texts:
I have a mysql database with the following format:

id      string
1        foo1...
2        foo2...
..       ...

There are >100k entries in this db.

What I want to do is for each string, compare it to each other string and store some metric of the comparison.  Doing this will essentially yield a 2D matrix of size NxN where N is the number of row in the db.

My initial thought was creating another db where each index corresponds to the string of the index in the first db and each column is the value from comparing the two strings. For example, id 1 column 2 in the second db would be the value outputted from comparing id1 and id2 in the first db.

The format of the second db:

id    col1    col2    col3    ....
1       1      0.4     0.5    .....
...    ...     ...      ...

This way of creating the second db would result in 100k rows x 100k columns, which is the issue at hand.  What is the best way to handle large data sets like this?  Is storing the data in a text file more efficient (say each text file corresponds to one row in the second db.)


texts: What information does output of [SEP] token captures in BERT?

texts:
After reading around on the web I came to understand that the output representation of the special token [CLS] captures the representation of a sentence (am I correct?).

My primary question is what information does the output embedding of [SEP] token (T_SEP) captures?

My other doubt is if I input a bunch of sentences into BERT separated by [SEP] does the output embedding of [CLS] contain information about all the sentences?


texts: Identifying templates with parameters in text fragments

texts:
I have a data set with text fragments having a fixed structure that can contain parameters.
Examples are:

 Temperature today is 20 centigrades
 Temperature today is 28 centigrades

or

 Her eyes are blue and hair black.
 Her eyes are green and hair brown.

The first example show a template with one numerical parameter. The second one is a template with two factor parameters.

The number of templates and the number of parameter is not know.

The problem is to identify the templates and assign each text fragment to the corresponding template.

The obvious first idea is to use clustering. The distance measure is defined as a number of non matching words. I.e. the records in example one have distance 1, in example two distance is 2. The distance between the record in example one and two is 7.
This approach works fine, providing the number of clusters is know, which is not the case, so it is not useful.

I can imagine a programmatic approach scanning the distance matrix searching for records with lot of neighbors in distance 1 (or 2,3,..), but I'm curious if I can apply some unsupervised machine learning algorithm to solve the problem. R is preferred, but not required.


texts: Text classification: Choosing training and testing set for each category

texts:
I am very new to machine learning.
I have a text classification problem in hand. I have a tagged dataset of around 750 documents( short texts), categorized manually into 16 buckets. I want to train a classifier on this data. I know that there should be a training set and a test set (an option could be 80-20 ). In my understanding, this should be for the complete set( 80% of my 750 documents- training, 20% of 750 documents - testing ).
1. They should be randomly generated or is there some condition for category? ie. if category A constitutes 60%,category B 5%, C 7% etc. how to choose the training set?


texts: How does GlobalMaxPooling work on the output of Conv1D?

texts:
In the field of text classification, it is common to use Conv1D filters running over word embeddings and then getting a single value on the output for each filter using GlobalMaxPooling1D.

As I understand the process, the convolutional filter is a matrix of the same size as the $$\text{size of filter matrix} = \text{embedding dim}\cdot\text{width of the filter}$$ The filter matrix is then applied to the input embeddings (multiplied element by element) which produces a matrix of the same size for each filter position. Not a single number.

So how does the global max pooling get a single number on the output? Does it simply take a maximum over all the values in all the output matrices, or is there any other processing?

Please correct me if I'm wrong.


texts: Decision boundary for KMeans clustering algorithm

texts:
Below code is used for plotting decision boundary for KMeans clustering. I understand contour in general, my question is, how does plot (contour method) know where to draw decision boundary based on predicted Z values, without knowing X and Y coordinates?
 xx, yy = np.meshgrid(np.linspace(mins[0], maxs[0], resolution),
                         np.linspace(mins[1], maxs[1], resolution))
 Z = clusterer.predict(np.c_[xx.ravel(), yy.ravel()])
 Z = Z.reshape(xx.shape)

 plt.contourf(Z, extent=(mins[0], maxs[0], mins[1], maxs[1]),
                cmap="Pastel2")
 plt.contour(Z, extent=(mins[0], maxs[0], mins[1], maxs[1]),
                linewidths=1, colors='k')


texts: How does the forward method get called in this pyTorch conv net?

texts:
In this example network from pyTorch tutorial

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

net = Net()
input = torch.randn(1, 1, 32, 32)
out = net(input)
print(out)

Why is the method forward() not explicitely called? I mean how does just calling net(output) calls forward() ? (which is what happens as far as I understand)
By the way I dont understand what this line means:

super(Net, self).__init__()

I can imagine super() is calling the constructor of a parent class but …?


texts: What tool to find expected and hidden patterns in data?

texts:
I have no background AT ALL in data science/stats/mathematics. However, I've always been interested in what data shows.

I have a huge dataset right now - daily attendance figures for a factory of ~300 for the past 10 years. I'm interested in finding out answers to questions like is there a pattern of leaves correlated with public holidays? For example, around which holiday (+/- 2 days) are workers most likely to take a leave? This is an expected pattern. Or, was there a significant increase (+10%) in on-time reporting after bonuses were issued. Maybe there are hidden patterns which an algorithm can find.

Is there a tool I can plug this data in which can help me find these patterns? Google tells me there's a tool http://www.i-programmer.info/news/84-database/3501-mine-finding-patterns-in-big-data.html but I'm not sure if this is the right direction for me.

I'd appreciate any advice!


texts: Python : data type handling by sklearn and impact on memory usage and performance

texts:
I am currently working on reducing the memory usage of my data (Pandas DataFrame). This is going quite well : downcasting floats into smaller floats, integers into smaller ones and transforming string objects into categories, then using these new defined types directly when reading the data. I would even be interested in going further (typically rounding some floats to downcast them into int).
However I suspect that it may not have so much impact on the rest of my sklearn Pipeline (pre-processing of my data types ints, floats and categories, and calibration of a small ANN). More specifically :

I can't find anything in the documentation, but there seems to be a tremendous amount of discussion on GitHub on which transformer can handle float32. This might limit the usefullness of downcasting my float, but also for some transformation (min/max scaler, logarithmic transformation) might 'upcast' my ints into floats.

Category seems handled like objects in the pre-processing step. Those features mainly go trough categorical encoding. So as far as the labels are handled correctly, I don't think it should have much impact.

It's unclear to me how those preprocessed data are handled trough the Main model fitting sequence. From litterature, it seems that the float precision may have a significant impact overall (reducing float precision, without getting rid of too much information, can improve speed of calibration and overall statistical performance of the model). But it's unclear how to ensure that enough, but not too much, float precision is used trough the whole Sklearn process.

So, quite generally speaking, what are the steps to ensure that Sklearn is efficient in terms of memory usage and precisions of operations ?


texts:
Hello!

I started learning the Gradient Descend Method in order to solve some regression problems, I can say I know what the algorithm does overall but I can't understand why in every cost function, the x^i is not taken as an argument!

Lets suppose we have n sized input data (x1,x2,...,xn) , each input data has m attributes. In the cost fucntion we basically have an xi from (x1,x2,...,xn), but which one? We just take one at random? And that remains starndard throughout the algorithm? I did attach the image to provide an example of what I'm saying


texts: Feature Engineering Lists\Vectors as values in dataframe

texts:
Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe.

In this case, I can:

I could add many columns of 1's and 0's for each string that may appear in some list, which seems to me terribly inefficient as some of the items in the lists only appear very few times. If there were maximum of 10 different items that could be listed in all of the lists , I guess it's quite all right to use/
I know some use SVD in these cases (even though I don't completely understand it yet.)
could put inside the dataframe pandas series as values like some kind of vectors.

So I ask:

what is the best way to feature engineer it assuming there are few items that could appear in the lists?
what is the best way to feature engineer it assuming there are lots of items that could appear in the lists?
Is there a certain way to do it that runs most efficiently with the popular ML\DL python packages?


texts: 28x28 mnist image unrolling to concatenate using numpy.vstack. Why is numpy.vstack so slow?

texts:
I'm trying to do a simple reshape of a 60000,28,28 list of mnist digits into a 60000,784 numpy array where the digits have been unrolled.
To do this the code is this:
(xdata,xlabel),(ydata,ylabel)=tf.keras.datasets.mnist.load_data()
newxdata=np.array([])
cnt=0
for i in xdata:
 tmpx=i.ravel()
 if cnt == 0:
  newxdata=np.concatenate((newxdata,tmpx))
 else:
  newxdata=np.vstack((newxdata,tmpx))
 cnt=cnt+1

Why does this take so long to run? Is there a way to speed it up? Ultimately the data will be fed into a keras model in smaller batches. Would writing a generator that does the loop unrolling when a batchsize is asked for be more performant or would it not make a difference?


texts:
I have a code that computes the accuracy, but now I would like to compute the F1 score.
accuracy_1 = tf.reduce_mean(tf.cast(tf.equal(
    tf.argmax(output_1, axis=-1),
    tf.argmax(y_1, axis=-1)), tf.float32), name="accuracy_1")
accuracy_2 = tf.reduce_mean(tf.cast(tf.equal(
    tf.argmax(output_2, axis=-1),
    tf.argmax(y_2, axis=-1)), tf.float32), name="accuracy_2")

How can I compute F1 equivalent for the above code? I'm finding it difficult as I am very new to TensorFlow.


texts: Should I take random elements for mini-batch gradient descent?

texts:
When implementing mini-batch gradient descent for neural networks, is it important to take random elements in each mini-batch? Or is it enough to shuffle the elements at the beginning of the training once?

(I'm also interested in sources which definitely say what they do.)


texts: What will be the policy if the state space is continuous in Reinforcement learning

texts:
I have started recently with reinforcement learning. I have few doubts regarding the policy of an agent when it comes to continuous space. From  my understanding, policy tells the agent which action to perform given a particular state. This makes sense when it comes to the maze example, where the state space is descrete and limited. What if the state space is continuous, will the agent have information of every possible state in the state space?
Also will an RL agent be able to take decision if its in a new state that it has not encountered during training ?


texts: No result for scored labels in Azure ML Web Service

texts:
I am trying to predict scored labels using regression. But when I am about to get the result from Azure ML Web Service in Excel 2016, there is no result appeared in the scored label column. How should I fix this? Below is all my process...

Here is my problem I always get. As you are seeing now, there is no result in scored label column when I try to predict.


texts: Feature importance after PCA (or other dimensionality reduction methods)

texts:
I have text data which I one hot encoded and then used PCA on it (although I'm experimenting with other methods as well, LDA, NMF..). I am using the result of the dimensionality reduction as an input for a supervised classification task.

Now I can use the random forest feature importance or other methods to get feature importance of the input to the supervised cls task. However, naturally these features are meaningless. I would like to know which words are most the most important for this classification. In other words, somehow propagate the feature importance score back through the PCA.

Is there any known method to do it?


texts: how to get the Polysemes of a word in wordnet or any other api?

texts: how to get the Polysemes of a word in wordnet or any other api. I am looking for any api. with java any idea is appreciated?


texts:
I am trying to start training Imagenet classification training using Tensorflow's inception model. I am a bit confused as I am not sure how to fully train the model.

Wherever I go everyone seems to be using the 1000 classes training list. However, while digging in tensorflows inception model files I found a list of labels containing 21000+ classes.

So I am a bit confused as to whether I should train using the 1000 classes that everyone seems to be using or should I train on the 21000 classes. I have the full imagenet data and I checked the images I downloaded and almost all 21000 classes are there (missing only 200 classes).

I really want to train against the full imagenet dataset as this should give more accuracy but I am not sure what the downfalls will be or how to properly prepare the data and how many steps are appropriate for that.


texts:
I'm trying to build a CNN similar to this:

For this purpose I chose to use Keras since I worked with it before (simple RNN and FFNN only).
My training data consists of the input data (25 x 25px images) and the output, being 25 x 25px images as well.
The problem is, that I don't know how to build the Model using Keras. How do I specify the output to be of the same shape as the input?

Here is what I have so far:

batch_size = 5
input_shape = (batch_size, 28, 28,  32)

model = Sequential()
model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),
                 activation='relu',
                 input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Conv2D(64, (5, 5), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(1000, activation='relu'))
model.add(Conv2D(64, (5, 5), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),
                 activation='relu',
                 input_shape=input_shape))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.SGD(lr=0.01),
              metrics=['accuracy'])

I know, that the code will not work like that. I just want to clarify what I'm trying to do. The code above was taken from: Keras tutorial – build a convolutional neural network in 11 lines

As of now I just tried to reverse the code from the point, where the tutorial would output the predicted label for the given image to transform the result back to the wanted output image.


texts: What is the use of additional column of 1s in normal equation?

texts:
Currently I am going through Normal Equation in Machine Learning.

$$
\hat\theta = (X^T \cdot X)^{-1} \cdot X^T \cdot y
$$

But when I see how they use this equation, I found they always add an additional column of 1s in the starting of matrix X before transposing.

I don't understand why. What's the logic behind this?

The places where I found such things

1) Coursera - Theory

2) Implementation

  Now let’s compute using the Normal Equation. We will use the inv()
  function from NumPy’s Linear Algebra module (np.linalg) to compute the
  inverse of a matrix, and the dot() method for matrix multiplication:

  X_b = np.c_[np.ones(( 100, 1)), X] # add x0 = 1 to each instance

  Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn and
  TensorFlow: Concepts, Tools, and Techniques to Build Intelligent
  Systems (p. 111). O'Reilly Media. Kindle Edition.


texts: What does dimension represent in GloVe pre-trained word vectors?

texts:
I'm using GloVe pre-trained word vectors (glove.6b.50d.txt, glove.6b.300d.txt) as word embedding.
I have a conceptual question:

What is the difference between these files?
On the other hand, what does the dimension represent in the GloVe pre-trained word vectors?


texts: How can I detect anomalies/outliers in my online streaming data on a real-time basis?

texts:
Say, I've a huge set of data(infinite in size) consisting of alternating sine wave and step pulses one after the other. What I want from my model is to parse the data sequence wise or point wise and the first time it parses a sine wave and starts facing the step pulses raise an alert as an outlier but as it goes on parsing the data it must recognise the alternating sine and step pulses and treat them as normal pattern. But then if it faces something out of this trend it must treat them as outlier, however if that new pattern repeats constantly it must treat them as normal again. In other words, my model must "remember" what it saw in the past to some extent to predict what is "normal" in the near future and on the basis of that detect anomalies in my constantly streaming data.

I've tried implementing the conventional stateless LSTM to achieve my requirements but LSTM being a supervised learning process needs an initial training and always predicts based on that initially given data. So what happens is if the pattern it recognised while training initially deviates in the test phase it always treats the pattern in the test phase to be an outlier irrespective of how many times it is repeating. Simply put, it fails to update itself with time.

I've gone through relevant papers on 'Anomaly Detection of online streaming data' and found HTM implemented by Numenta and tested on NAB benchmark is the best solution in this respect but I am looking for something open source and absolutely free to use.

Being a newbie in this field, any existing open source implementation will be highly appreciated as writing something from scratch is not preferred but if required that'll be my last option.


texts: Strange Behavior for trying to Predict Tennis Millionaires with Keras (Validation Accuracy)

texts:
I'm trying to make an NN with Keras to predict the ATP players that will get more than US$1 million in prize money based on their weight and height (from a dataset I mined some weeks ago), but I have been getting a weird behavior especially for the validation accuracy. Sometimes it gets to 84-85%, which is reasonable since SVMs and GaussianNB seem to be able to hit only 83.3% at best (check this post for more info), but sometimes it gets only around 15% accuracy. And the accuracy mostly doesn't change at all if I add capacity or more epochs. I guess one of the weirdest trends in this problem is that the loss is comparably quite small when I get low accuracies:

All of the code is available in this repo (specially in ATP NN.ipynb and the atp_python_2018-08-27_1-1500.json), but the main part of it is:

model = Sequential()
model.add(Dense(20, activation = 'relu', input_shape = (2,)))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(2))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
model_training = model.fit(x = hw_a, y = prize_a_bin, epochs = 4, validation_split = 0.1)

At first, I thought this was happening because the validation split done by Keras would be getting somewhat improper sets that were not very consistent with the rest of the data, since the millionaires are only 10% of the whole dataset, but that doesn't seem to be the case, the weirdness persists. The only pattern I can kind of recognize is that the normal performance and the strange one are complementary, i.e., if one is 85%, the other will be 100% - 85% = 15%. Is that linked to some mistake I've made?

On a sidenote, since I'm a beginner with Keras, should I have specified the last layer's activation function? I assume he is doing either a softmax or a sigmoid by default, but it seems make no difference if I specify either of them. I wonder why with only class (either millionaire or not), you have reshape y into a to_categorical and have two units - instead of two - in the output layer.


texts:
When dealing with the class imbalance problem in a binary classifier, there are three ways I know of to address it: over-sampling, under-sampling and using cost-sensitive methods.

Are there any guidelines, rules of thumb or general strategies to choose among these methods? A possible answer would be: over-sample when the positive class has more than 100 instances (I just made it up).


texts: How do you map phrase IDs to sentence IDs in the Stanford Sentiment Analysis dataset?

texts:
I am trying to reproduce results obtained on the Stanford Sentiment Analysis dataset, which contains sentiment annotations for syntactic components of sentences taken from Rotten Tomatoes reviews.

The data is partitioned into training, development, and test sets by sentence in the datasetSentences.txt and datasetSplit.txt files. Components within these sentences are mapped to sentiment annotations in the dictionary.txt and sentiment_labels.txt files. The README states that the sentence and phrase IDs are different, but I can't find any mapping between them, so I don't know how to split the sentiment annotations up into the partitions used in the experiment I am trying to reproduce.

Of course I would expect the mapping from phrases to sentences to be many-to-one (because the same phrase might appear in multiple sentences) but I'd still expect to be given an explicit mapping instead of having to compare substrings.

Has anybody worked with this data set? Is there something I am overlooking?


texts:
I have an input data file which contains list of drug names.

I have more than 1000 unique drug names. However, the drug names has spelling mistakes and space character issues.

For ex: we have ISONIAZID 300MG TAB , ISONAZID300MG TAB, ISNIAZID 300MG

You can see how the above 3 terms are different in representation (due to spelling mistake) but actually indicate the same drug ISONIAZID 300MG TAB (which is the right spelling)

But the problem is there are several other drugs with such spelling mistakes and I am not sure how can I group all of them into one (meaning rename it with right spelling)? ex: All the 3 terms above should be renamed to ISONIAZID 300MG TAB (which is the right spelling)

I am posting it here to seek your opinion on is there any medical dictionary or automated approach which can take my raw csv file as input and output proper drug names?


texts:
Is it a good idea majoring in data science for undergraduate?

Auckland University offers an undergraduate major in data science, which has courses in computer science and statistics.

I read somewhere that it's a bad idea to to undergraduate in data science.


texts: What does many low important feature indicate?

texts:
I have a dataset where I am focusing on binary classification problem. In total,I have around 60 features in my dataset

When I used Xgboost Feature Importance, I was able to see that the top 5 features account for 42% whereas rest of the of the 50 features account for 40-49 % (each feature about 1%) and remaing 8-10 features have zero importance or less than 1% of importance.

This is my best paramter list for Xgboost after gridsearch

op_params = {'alpha': [10],
 'as_pandas': [True],
 'colsample_bytree': [0.5],
 'early_stopping_rounds': [100],
 'learning_rate': [0.04],
 'max_depth': [6],
 'metrics': ['auc'],
 'num_boost_round': [10000],
 'objective': ['reg:logistic'],
 'scale_pos_weight': [3.08],
 'seed': [123],
 'subsample': [0.75]}

Since I have many low importance features, should I try to use them all in my model to increase the model metrics?

When I built the model only with top 5 features, I was able to get 80% accuracy.

I am trying to understand is it even useful to make use of these low importance feature for prediction?

Shown below is my feature importance in descending order

Do they even really help?

Any insights would really be helpful


texts: Kmeans cluster validation when I have labeled test data

texts:
I'm trying to implement the unsupervised k-means algorithm for sentiment analysis of imdb movie dataset created by stanford.
The steps that I followed is :
1) Load the comments
2) Apply tokenization and stemmetion ,use tf-idf algo to create tfidf matrix.
3) Use k-means algo to divide the data into 2 clusters.

My problem is how do I validate the the clusters
I have labeled test data. I want to check if all the negative examples go in one cluster and all the positive examples go in another cluster.

For this I used the scikits predict() command. But the parameters that need to be provided to the predict() command should be same as the parameters of fit() command. But as tfidf matrix that is passed has the dimensions dependent on the data passed.


texts: Does AWS Sagemaker continue running when I go offline?

texts:
I'm pulling data from web services in an AWS Sagemaker Notebook.

I'd like to be able to close my computer for an hour, reconnect my computer and see that the data collection has continued to run.
Additionally, I'd like my notebook's kernel sessions to remain active so that I can access the data in memory without having to write to file and reload, etc.

Is there a setting to accomplish this, and is it enabled by default?


texts:
I was recently thinking about the memory cost of (a) training a CNN and (b) inference with a CNN. Please note, that I am not talking about the storage (which is simply the number of parameters).

How much memory does a given CNN (e.g. VGG-16 D) need for

(a) Training (with ADAM)
(b) Inference on a single image?

My thoughts

Basically, I want to make sure that I didn't forget anything with this question. If you have other sources which explain this kind of thought, please share them with me.

(a) Training

For training with ADAM, I will now assume that I have a Mini-batch size of $B \in \mathbb{N}$ and $w \in \mathbb{N}$ is the number of parameters of the CNN. Then the memory footprint (the maximum amount of memory I need at any point while training) for a single training pass is:

$2w$: Keep the weights and the weight updates in memory
$B \cdot $ Size of all generated feature maps (forward pass)
$w$: Gradients for each weight (backpropagation)
$w$: Learning rates for each weight (ADAM)

(b) Inference

In inference, it is not necessary to store a feature map of layer $i-1$ if the feature maps of layer $i$ are already calculated. So the memory footprint while inference is:

$w$: The model
The two most expensive successive layers (one which is already calculated, the net one which gets calculated)


texts: Step-by-step construction of an RBF neural network

texts:
I have started to look at Radial-Basis-Function Neural Networks (RBF-NN), and would like to solve the following task as an exercise: Given is the data in the image below. Each output $Y_j$ of the network is defined as:

$(*)$  $Y_j = \sum_{i}^N w_{ij}* exp(-\frac{||x-\mu_i||^2}{2\sigma_i^2})$

for the -ith neuron. The task is to draw an RBF network that perfectly classifies the data, with suitable means, covariances and weights. In a second step a point has to be taken and classified in the worked-out model.

My ideas We have two nodes in the input layer, one for each dimension. The hidden-layer has as many neurons as there are training samples. Each of these calculates the activation given by the exponential above. The output layer has three output nodes as there are three classes. I would determine $\Sigma$ and $\mu$ as follows: For each class and training-sample $x_i$ of this class, $\mu_j$ is just the centroid of the training samples, $\sigma = \frac{1}{m}\sum_i ||x_i-\mu||$, and $\Sigma = \sigma*I_d$.

Questions That does not seem to be correct however - if I have the same $\Sigma$ and $\mu$ for all the hidden nodes of the same class, each new test-input would result in the same activation for all these nodes..so what exactly are the particular $u_i$ and $\sigma_i$? Also, for perfect classification, I would set all the weights which belong to the correct class to 1, and all the others to zero - would that make sense?


texts: Understanding of threshold value in a neural network

texts:
Consider the hypothetical neural network here

$o_1$ is the output of neuron 1.
$o_2$ is the output of neuron 2.
$w_1$ is the weight of connection between 1 and 3.
$w_2$ is the weight of connection between 2 and 3.
So the input to neuron 3 is $i =o_1w_1 +o_2w_2$
Let the activation function of neuron 3 be sigmoid function.
$f(x) = \dfrac{1}{1+e^{-x}}$ and the threshold value of neuron 3 be $\theta$.
Therefore, output of neuron 3 will be $f(i)$ if $i\geq\theta$ and $0$ if $i\lt\theta$.
Am I correct?


texts: What expresses if/how two variables are dependent on each other?

texts:
The accepted answer to question Why is a correlation matrix symmetric? includes this:

  The correlation matrix is a measure of linearity. It does not express how two variables are dependent on each other.

My question is : What is there that is related to if/how two variables are dependent on each other?

Edit 1 : When asking this question, not considering anything related to correlation, only want to know if there are ways to see if two or more variables have dependency on each other or not, irrespective high or no correlation between them.

I used the correlation tag because I could not think of what tag to use, please feel free to edit/correct the tag.


texts: What could cause validation set to consistently perform better than training?

texts:
I'm training a neural network with a very small dataset just to get things set up, before training on a much larger set. (I only have about 500 data points available to me at this time, with more coming over the weekend.) I should also mention that I'm exploring whether or not this data has enough of a signal to even warrant trying to train a model on it.

I'm currently working with ~500 feature vectors with 4 features each.

I find that when I have a training set containing about 90% of my samples, randomized, that it typically performs significantly worse than the small set of validation samples being validated on that modal. I would expect the opposite.

Edit: I should mention that I'm using RMSE for my loss function

What could this mean?

Some thoughts I have are that I just need more data points to get a good read from the validation set. But I'm also wondering if it could mean that there just isn't a good signal in the features I'm using. But if that was the case, wouldn't my model overfit to the random-ish data and then still perform worse on the validation set?


texts: How to store subset of columns from a csv file?

texts:
I need to create a table in hive (or Impala) by reading from a csv file (named file.csv), the problem is that this csv file could have a different number of columns each time I read it. The only thing I am sure of is that it will always have three columns called A, B, and C.

For example, the first csv I get could be (the first row is the header):

 ------------------------
|  X | Y | A | Z | B | C |
 ------------------------
|  1 | 2 | 3 | 4 | 5 | 6 |

and the second:

 ------------
|  C | A | B |
 -------------
|  1 | 2 | 3 |

And I need to store this in a table, maybe an external table. Something like this:

CREATE EXTERNAL TABLE file (A STRING, B STRING, C STRING)
AS
SELECT A, B, C
USING HEADER
LOCATION 'input/loading/';

That obviously does not work. Any ideas?


texts: Weighted linear regression with a DNN (in Keras)

texts:
I have a fairly small dataset of 225 points. I have a target (labelled numeric), a feature (normalised numeric) and a quality index with set of normalised weights that describe the likelihood that the observation will contribute to a linear relationship between target and feature. Good data for weighted linear regression.

The dataset is currently at only 225 points but it's growing so I'd like to explore DNNs to improve on the performance of the weighted linear approach.  Can Keras do this?

What I am curious about is the possibility that the weight optimization process in the DNN training could itself eliminate points with a low quality index, i.e. those points that do not contribute to a linear fit and result in a high loss function return.

But I am very new to deep learning so my understanding could be wrong.


texts: Rule of thumb for good number of features when dealing with grouped data

texts:
I have a classification problem on clinical data where I have multiple samples for each patient. So the samples related to the same patient are somehow dependent from each other.

I know that is not possible to know a priori the optimal number of features to use, but there are some rule of thumb that works in many cases.

My question is: are those rules valid also in my case? In particular, I should relate the number of features to the number of instances or to the number of groups?

Thanks


texts: Time series finance -Correlation between a sector and MSCI ACWI returns

texts:
I have daily return data of 11 sectors of MSCI World Index and the MSCI ACWI index. I want to know the stationarity of correlations between the sectors and MSCI ACWI index.

This is what I have done:

1) aggregate the daily returns into month and calculate correlation between each of the sectors and the MSCI ACWI index (so for each monthly correlation, around 20 daily returns were used). Then I used the Augmented Dickey-Fuller test to see if the monthly correlation time series is stationary or not.

2) aggregate the daily returns into week and calculate correlation between each of the sectors and the MSCI ACWI index (so for each weekly correlation, around 5 daily returns were used). Then I used the Augmented Dickey-Fuller test to see if the weekly correlation time series is stationary or not.

The result is reproduced here:

As you can see, the stationary result is different for monthly and weekly correlations.

My question is:

1) which model is a better one?

2) what other kinds of tests or useful stuff can I do with this stationary test?

Thank you in advance!


texts: How increase forecast accuracy in sklearn linear regression

texts:
I have this code -

wd_val = load_workbook(filename = 'table.xlsx')
sheet_form = wd_val['2017 год']

x = []
y = []

for a in range(6, 996):
    x.append([sheet_form['C' + str(a)].value,
              sheet_form['D' + str(a)].value,
              sheet_form['E' + str(a)].value,
              sheet_form['F' + str(a)].value,
              sheet_form['G' + str(a)].value])

    y.append(sheet_form['H' + str(a)].value)

xTrain, xTest, yTrain, yTest = train_test_split(x,y,test_size=0.2)

model = LinearRegression(fit_intercept=False, normalize=True)
model.fit(x, y)

#xInput - 2107, 746, 1037, 1691, 26
predict = model.predict(xInput)
#predict = 562108

This forecast is a little inaccurate, it should be equal to ~596.495
How i can improve prediction accuracy? Maybe you need to use a different regression model?
I train the model according to the data from the Excel table (there are more than 900 lines for training).


texts: How can I perform multi-label classification if many labels are missing?

texts:
I have a large set of documents, usually 500-2,000 words each, and for several different labels, there are about 20-100 samples with those labels, and hundreds to millions more that should be labeled but do not have that label. Over time, I expect new labels to be created and new documents to be added to the data set, although relatively few old labels added to any documents. It is possible that some documents will have many labels or possibly no labels. It is also possible for a label to be a subclass of another label, though not manually indicated.

I have a keyword tagging system in place that utilizes TF-IDF metrics to extract ngrams from the documents, which is my current feature set from them.

My initial thought is to use Naive Bayes classifiers (one for each label) and perform ROC analysis to see how well it classifies the category. However, if, say, 5% of the data should actually be labeled with Label A, and the amount of data supplied with Label A amounts to 1% of the data, then I would expect an ideal false discovery rate of 80%. On top of that, since the prior distribution is unknown, dealing with this issue is even more difficult.

Another idea I had is to use the vectors of word/ngram counts and look at the distribution of the cosine similarities of the documents within the sample and use a simple one-sided significance test with p-values to determine if they are similar enough. This relies heavily on the idea that the ngram vectors for each label are dissimilar enough to consider this form of hypothesis testing. A big problem with this is that I am heavily relying on the assumption that similarity is a highly discriminative metric, and because labels can be extremely unbalanced (by up to 5 orders of magnitude), I may need to employ additional methods, such as limiting the number of labels (perhaps by only taking the top N labels by p-value).

The training portion of the algorithm does not need to be extremely fast, but the labeling portion should be very quick once the document is parsed/tokenized.

There is a greater penalty for mislabeling a document than leaving it unlabeled, but future, human-feedback data may be provided to indicate incorrect labels. I will not have access to anything of the sort in the meantime, though.


texts: What is the difference between spiral, flame, aggregation data


texts: How to predict based on multiple samples?

texts:
I am relatively new to ML so I apologies in advance if my question shows lack of understating of the field.

The problem

A particular study course has a high drop-out rate and we want to reduce it.

The dataset

We have two data sets:

Data about the student themselves (e.g. scored of initial aptitude tests, demographic). This dataset includes the feature to predict. E.g.

ID, AGE, RESULT_TEST1, RESULT_TEST2,DROPPED_OUT
AAAA, 21, 0.6, 0.4, TRUE
BBBB, 20, 0.3, 0.9, FALSE

Exams taken by the students during the course. This dataset contains data about the type of the exam (e.g. subject, level of complexity) and the results obtained by the student. Some of the exams are required, some are mandatory and some can be taken on voluntary bases. That is, each student can take an arbitrary list of exams. E.g.

ID, EXAM_ID, EXAM_COMPLEXITY, EXAM_SCORE
AAAA, XXXXX, 0.8, 0.4
BBBB, YYYYY, 0.2, 0.8

The goal

The idea is to use ML to calculate the likelihood of a particular student to drop-out during the course using historical data. The system should be able to predict based on the two datasets, how likely is a student to at drop-out so that we can give him/her more support.

The challenge

How do I combine the two datasets to train a model? I could create a superset joining the two but what would I then use to predict a result?

E.g. If I train the model on:

ID, AGE, RESULT_TEST1, RESULT_TEST2, EXAM_ID, EXAM_COMPLEXITY, EXAM_SCORE
AAAA, 21, 0.6, 0.4,XXXXX, 0.8, 0.4
BBBB, 20, 0.3, 0.9,YYYYY, 0.2, 0.8

How do I predict the likelihood of DROPPED_OUT passing all the data I have for a new student (multiple samples)?

Any explanation or pointers to documentation would be greatly appreciated.

Thank you in advance.

Alex