texts: LSTM: Taking previous output values as feature

texts:
As far as I know, there is practically no limit on the number of dimensions of input feature for LSTM. And it apparently can learn the sequence of data.

My question is does LSTM by nature, also take previous output values besides feature vector as new feature element for the next output values? i.e. If we have:

[t]  -> [x1, ..., xn] [y_t]
[t+1] -> [x1, ..., xn] [y_(t+1)]

Is it necessary to manipulate data like this

[t]  -> [x1, ..., xn, y_(t-1)] [y_t]
[t+1] -> [x1, ..., xn, y_t]    [y_(t+1)]

or LSTM is already handling this for us?


texts: How to train continuous/soft classification model?

texts:
The classic classification problem is like finding the function $F:\mathbb{R}^n\mapsto \{0,1\}$. The label set will be [Apple,Banana,Banana,...,Apple].

What if I want to train a function $F:\mathbb{R}\mapsto[0,1]$? My samples could be something like "this sample has 80% probability to be Apple and 20% to be Banana".

It seems a multi-output neural network works, as we can apply the softmax loss with cross entropy loss. What about random forest or other algorithms? I have tried some common algorithms in scikit-learn without any luck.

For example, this code:

import numpy as np
from sklearn.ensemble import RandomForestClassifier

N_FEATURES = 10
N_SAMPLES = 1000
N_CLASSES = 2

train_x = np.random.rand(N_SAMPLES, N_FEATURES)
train_y = np.random.rand(N_SAMPLES, N_CLASSES)
train_y = np.apply_along_axis(lambda x: x/x.sum(), 1, train_y)

model = RandomForestClassifier(n_estimators=10).fit(train_x, train_y)

Yields a ValueError: Unknown label type: 'continuous-multioutput'.


texts: Understanding time series anomaly detection using Autoencoder

texts:
I'm studying how to detect anomalies in the time series using an Autoeconder. In particular, I'm following the guide posted in the Keras website, but I don't understand why they are creating and how can I adapt it to my dataset. In their guide they load the dataset and create a sequence:
TIME_STEPS = 288

# Generated training sequences for use in the model.
def create_sequences(values, time_steps=TIME_STEPS):
    output = []
    for i in range(len(values) - time_steps):
        output.append(values[i : (i + time_steps)])
    return np.stack(output)

x_train = create_sequences(df_training_value.values)
print("Training input shape: ", x_train.shape)

The reason why they are using 288 as TIME_STEPS is because they have a value for every 5 mins for 14 days. My questions are:

Does this method split the data or is he just creating a 3D variable in the correct format for the Convolutional Autoencoder?

I have a dataset where there are stored the measurements of 30 devices. Each device has about 4000 values and it is structured as well:
 device01 0.02;0.13;1.15;0.10;8.30;........;4.20
 device02 0.06;0.13;1.40;0.03;7.40;........;6.30
 ........
 device30 0.03;0.24;1.10;0.43;4.40;........;2.30

Since I do not have a timestamp reference in my dataset, how can I define the TIME_STEPS variable?


texts: how does feature selection for random forest help

texts:
I came across this link which talks about feature selections in random forest

It introduces two methods

Mean decrease impurity
Mean decrease accuracy

I would like to know if there are more methods and when should one method be preferred.

Also is it correct to say that less feature for random forest always gives worse result and the only gain is in computation speed?


texts: Why does Keras' loss decreases very slow?

texts:
I'm training an LSTM with Keras. I notice that the loss value decreases, but very slowly.

11748/11748 [==============================] - 71s 6ms/step - loss: 0.4564 - val_loss: 0.4543
11748/11748 [==============================] - 71s 6ms/step - loss: 0.4524 - val_loss: 0.4519
11748/11748 [==============================] - 71s 6ms/step - loss: 0.4517 - val_loss: 0.4515
11748/11748 [==============================] - 71s 6ms/step - loss: 0.4514 - val_loss: 0.4513
11748/11748 [==============================] - 75s 6ms/step - loss: 0.4512 - val_loss: 0.4511
11748/11748 [==============================] - 72s 6ms/step - loss: 0.4511 - val_loss: 0.4510

My model:

# Model definition
model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=100, return_sequences=True), input_shape=(timesteps, features)))
model.add(Dense(1, activation='sigmoid'))

model.summary()
model.compile(loss=listnet_loss, optimizer=keras.optimizers.Adam(learning_rate=0.00005, beta_1=0.9, beta_2=0.999, amsgrad=False))

My loss function:

def get_top_one_probability(vector):
  return (K.exp(vector) / K.sum(K.exp(vector)))

def listnet_loss(real_labels, predicted_labels):
  return -K.sum(get_top_one_probability(real_labels) * tf.math.log(get_top_one_probability(predicted_labels)))

I'm also standardizing my training set using the StandardScaler from scikit-learn.

Is the problem associated with the optimizer or the learning rate?


texts: about predict the class for a new datapoint

texts: I have a new data point and want to classify it into the existing classes.I can calculate pairwise distance for the new point to all existing points(in the existing classes). I know using KNN would be a straightforward to classify this point. Is there a way I could randomly sampling existing classes and then correlated the new point to a potential classes without calculating all pairwise distances?


texts: Intuitive explanation of how Latent SVM works?

texts:
Can anyone explain (or refer to a great explanation of) the intuition of how Latent SVM works?

I think Latent SVM should have some resemblance to CRF (Conditional Random Fields) and EM (expectation maximization) and of course standard SVM (max-margins, kernel trick) and I'll appreciate very much answers using these 'tools' to explain how Latent SVM works.


texts: Why are variables of train and test data defined using the capital letter (in Python)?

texts:
I hope this question is the most suitable in this site...

In Python, usually the class name is defined using the capital letter as its first character, for example

class Vehicle:
    ...

However, in machine learning field, often times train and test data are defined as X and Y - not x and y. For example, I'm now reading this tutorial on Keras, but it uses the X and Y as its variables:

from sklearn import datasets

mnist = datasets.load_digits()
X = mnist.data
Y = mnist.target

Why are these defined as capital letters? Is there any convention (at least in Python) among machine learning field that it is better to use the capital letter to define these variables?

Or maybe do people distinguish the upper vs lower case variables in machine learning?

In fact the same tutorial later distinguish these variables like the following:

from sklearn.cross_validation import train_test_split

train_X, test_X, train_y, test_y = train_test_split(X, Y, train_size=0.7, random_state=0)


texts:
I installed the packages for Tensorflow and Keras by using the Anaconda Packege installer (I have a separate environment for that). Unfortunately I can't use Tensorflow or Keras. When I use the following code, I get an error message:
    import tensorflow as tf
from tensorflow import keras

zip_path = tf.keras.utils.get_file(
    origin='https://storage.googleapis.com/tensorflow/tf-keras-datasets/jena_climate_2009_2016.csv.zip',
    fname='jena_climate_2009_2016.csv.zip',
    extract=True)
csv_path, _ = os.path.splitext(zip_path)

--> Error message: AttributeError: module 'tensorflow' has no attribute 'keras'
I tried to check the version of tensorflow by using the following code:
import tensorflow as tf
print(tf.__version__)

--> Error message: AttributeError: module 'tensorflow' has no attribute 'version'
I checked in the package manager of Anaconda and there the version of Tensorflow is 2.1.0 (see Screenshot)
From this website I can see that there is in fact a version 2.2 (https://docs.floydhub.com/guides/environments/)
However, I tried to update tensorflow (such that I can use it with keras) by using
Conda update tensorflow

But this did not change anything. I get the message: "# All requested packages already installed." and problem still persists and the version number remained unchanged. Also trying
Conda update keras

yields the same message "# All requested packages already installed."
What am I doing wrong? I'd appreciate every comment and would be quite thankful for your help.


texts:
I have the following problem:
I have created a customized Dictionary for getting used in some NLP tasks. I want to enhance my dictionary by finding phrases similar to the phrases I have in my dictionary.
For example:

Lets say I have a phrase:
- take responsibility
then, I should be able to find phrases like:
- hold accountable for
-responsible for

I have just started learning NLP and I tried Word2Vec model over my dictionary but it only gives similar words that are already present in the dictionary.
Is there any other algorithm that helps me find similar phrases using multiple sources.


texts: How to achieve SHAP values for a CatBoost model in R?

texts:
I'm asked to create a SHAP analysis in R but I cannot find it how to obtain it for a CatBoost model. I can get the SHAP values of an XGBoost model with
shap_values <- shap.values(xgb_model = model, X_train = train_X)

but not for CatBoost.
Here is the reproducible code for my CatBoost model:
library(data.table)
library(catboost)

train_example <- data.table(categorical_feature = c("a", "b", "a", "a", "b"), payment = c(244, 52352, 4235, 3422, 535), age = c(34, 27, 19, 40, 92), target = c(0,0,1,0,1))
label_values_train <- train_example$target
train_X <-train_example[, - "target"]
train_X$categorical_feature <- as.factor(train_X$categorical_feature)
# datapool for the train dataset:
pool <- catboost.load_pool(train_X, label = label_values_train, cat_features = 1)
fit_params_report <- list(iterations = 1000,
                          loss_function = 'Logloss',
                          eval_metric='F1',
                          border_count = 42,
                          depth = 8,
                          learning_rate = 0.04,
                          l2_leaf_reg = 0.001,
                          task_type = 'CPU')

model <- catboost.train(pool, params = fit_params_report)

How can I proceed to get the SHAP values here?


texts: logistic regression - why exponent (log ratio) is linear

texts:
I am new to ds and stats in general. I read numerous article to understand logistic regression. I got some idea why it works and how it fits to scattered plot of 1s and 0s when target variable is binary. however one piece of puzzle I still don't understand is how come someone derived that

ln(p/1-p) = B0 + B1X1 + ..

I see all articles assume that this is the link function and then go on talking about how it solves our regression problem for binary variable. but how this link function came about.


texts: How to handle "unknown" category in machine learning classification problems?

texts: Tutorial problems come in the form of binary or mult-class classification where data are all properly labelled. In real-life applications, there are incoming data that do not belong to any category and cannot be classified. How can we handle these data which fall into "unknown" category? The universe of "unknown" can be far more than "known". So, the data for "unknown" can be too much and lead to class imbalance. How do we train the model to deal with "unknown" data? Or do we ignore it?


texts:
Problem
I want to train HyperGAN with a set of 400+ images of people, but they aren't the specified size (32x32 pixels) for training.
Question
Is there any way/program to help cropping/resizing them so as to not do it 100% manually?


texts: Time duration weighted recurrent neural network

texts:
Suppose the input time-series feature is $\vec{X}=[\mathbf{x_0},\mathbf{x_1},...\mathbf{x_T}]$, where at each time step $t\in[0,...,T]$, feature $\mathbf{x_t}$ is a vector with dimension $n$.
Typical RNN, such as LSTM/GRU, may assume that time-series are sampled with equal interval.
However, in my problem, $\mathbf{x_t}$ is not collected from time interval with equal duration. For example, $\mathbf{x_0}$ is calculated from a time interval with 1min duration. $\mathbf{x_1}$ is calculated from a time interval with 5min duration. It is resonable to expect that $\mathbf{x_1}$ may carray more information than $\mathbf{x_0}$ since it comes from a longer time interval. Thus we want to modified traditional RNN cell such that it can naturally consider this situation.
One of the choice, from my own opinion, is to add an extra deterministc operation in orginal RNN cell. We use GRU as an example here. We modifed the GRU cell as follow:
$$
z = \sigma(\tau_t\odot(U_zx_t) + W_zs_{t-1}) \\
r = \sigma(\tau_t\odot(U_rx_t) + W_rs_{t-1}) \\
h = tanh(\tau_t\odot(U_hx_t) + W_h(s_{t-1}\odot r)) \\
s_t = (1-z)\odot h + z\odot s_{t-1}
$$
here $\odot$ is element-wise multiplication and $\tau_t$ is the new added operation. We define $\tau_t = d_t/T$, where $d_t$ is the time duration of feature $\mathbf{x_t}$ at time step $t$ and $T$ is the chosen time scale. Suppose we choose $T=5$min, then $\tau_0=0.2$ and $\tau_1=1.0$ as we described before. Then $\mathbf{x_1}$ will naturally contribute more than $\mathbf{x_0}$ when passing the cell.
Is there any other ideas, expecially published paper that can accomplish the problem given above ? Comments are also wellcome here.


texts: ValueError from statsmodels ExponentialSmoothing

texts:
I've been having a frustrating issue with the ExponentialSmoothing module from statsmodels.

My data is a pandas series with 74 weekly data points that looks like this:

2017-12-31    6069
2018-01-07    8143
2018-01-14    6740
2018-01-21    6433
2018-01-28    6631
2018-02-04    6308
2018-02-11    5536
2018-02-18    6025
2018-02-25    5171
...           ...

When I call the following functions:

model = ExponentialSmoothing(data, trend='add',damped=True,seasonal='mul',seasonal_periods=52)
model_fit = model.fit()

I get:

Traceback (most recent call last):
  File ".\smoothingjuly.py", line 24, in <module>
    model_fit = model.fit()
  File "C:\Users\lhughes\AppData\Local\Programs\Python\Python37\lib\site-packages\statsmodels\tsa\holtwinters.py", line 641, in fit
    l0, b0, s0 = self.initial_values()
  File "C:\Users\lhughes\AppData\Local\Programs\Python\Python37\lib\site-packages\statsmodels\tsa\holtwinters.py", line 773, in initial_values
    b0 = ((lead - lag) / m).mean()
ValueError: operands could not be broadcast together with shapes (22,) (52,)

Why is this? It works if I decrease the number of seasonal periods, but that makes my model useless. Is 74 data points not enough for the model? If so what is the minimum?

Thanks


texts: Mixing Textual Data and Numerical Data (Neural Network)

texts:
I have 2 "nature" of data (more actually if I count images data) :

Textual (that I treat with special tokenization and a TfIdfVectorizer) ~ 5000 features
Non textual (like length of sentences, # of characters in my sentence, etc.) : only numerical features
Image data (presence of special label in the image in a document) : very rare so no need here

With all this kind of data I should be able to distinguish spam from non spam (different languages here).

For now I do have a good performance on my model but I was thinking of a way to mix more "properly" this kind of data in order to improve the performance.

For now I simply deal with all those features at the same level (with a RandomForest Classifier) but I was wondering if it could be a good idea to work with Neural Network and specifically neural network with multi input & multi output : my idea here is to deal with each nature of data first then merge all to see interactions between the different kind of features. I am using the MultiInput MultiOutput architecture present in Keras :

In fact for the Main_Input part (for me it is for textual features) I am just using a simple NN with only one hidden layer (only 64 nodes) and then an output layer with a $sigmoid$ activation function.

My Aux_Input are my "numerical" features and after merging all of these I just add one Dense Layer with again 64 nodes and then a output layer with a $sigmoid$ activation function.

The implementation in Keras is pretty good by the performance are worst than when I simply deal with all the features at the same time and throw it in an output layer (so just a Logistic Regression) ... and It is a little bit confusing to me: to me this kind of architecture is more suitable for my phenomena (I have two kind of data and I deal first separately then merge to get interaction).

Is it because I am loosing information between my initial textual features (TFIDF) and the "numerical" features by passing through a hidden layer only for the textual features ?
How to deal with "high dimensional" data (here the TFIDF features) and maybe perform a reduction dimension without losing performance ?
When I deal only with textual features I get better performance and when I deal with only numerical features, again, I get better performance but for both little bit worst than doing a Logistic Regression with all features together ... any clue here ?
Do you have suggestions or remarks for my architecture ?

In fact, just by writing down this question I realize that maybe I can do multiple classifier (one for each kind) and then "average them with a new one" ... but I don't know how ^^ (I am thinking of something like the output of a ML model would be an input of my final model). I am pretty sure that separately I can improve performance (for example try different embedding for the textual part).

What do you think ?

Thank you in advance for any comments, clues or whatever that can clear my mind or that can refer to a good article/book/tutorial/example :)


texts: What does "exaggeration" mean in the context of Boosting?

texts:
I am learning boosting, the machine learning ensemble meta-algorithm. The professor is grouping 3 weak classifiers into an ensemble and said that before this time point it is easy to understand. Take a dataset, train a simple model, find the smallest error rate, something like this. This idea is easy to implement, for instance, gradient descent would take logistic regression to the smallest error rate. Then, the professor talked about the data with an exaggeration of classifier errors.

My question: What does that mean? Can anyone give a sample for this operation based on an open dataset in Python or R?


texts: How to select the split point for Continuous Attribute Age

texts:
For the above table, midpoints for possible split points are 22.5 and 35. I have calculated the entropy and gain for each value and 35 had the minimum Entropy and highest gain. Is it correct ?
Given High -> (-), and Low -> (+)
D<22.5 => [0+, 2-],
Entropy (D<22.5) = 0, since all the values are of the same class High.
D>22.5 => [2+, 2-],
Entropy (D>22.5) = 1, since the values are distributed equally among Low and High classes.
D<35 => [2+, 3-],
Entropy (D<35) = -[2/6 x $log_2$(2/6) + 3/6 x $log_2$(3/6)]= 0.5
D>35 => [0+, 1-],
Entropy (D>35) = 0, since all the values are of the same class High
Gain (D, Age>22.5) = 0.918 - 2/6 (0) - 4/6 (1) = 0.2513
Gain (D, Age>35) = 0.918 - 5/6 (0.5) - 1/6 (0) = 0.5103
Is that right?


texts: how to make moving moving graphs that vary in time like a video?


texts: Is there any other probability distribution model than gaussian for multivariate data

texts: Whenever we talk about the probability distribution of data having more than one feature, we have only one option i-e multivariate normal distribution. Is there any other probability distribution model exist for multivariate data? If yes, then how can we find its parameters using MLE in MATLAB


texts: sklearn stratifying dataset with more than one label

texts:
I have come across sklearn.model_selection.train_test_split as a method to split up the train and testing dataset.

Furthermore they have a stratify which you can set to the labels to ensure your train and test data set have approximately the same proportion of labels of each type.

My question is what if you have more than one type of label for your data, so that your labels are not a size len(dataset) x 1 but a size of len(dataset) x m where m is the number of your label types. What tool can you use to stratify (and split) your dataset in the multilabel case.


texts: What does it mean if performance of two different iterations of the same network (CNN model) varies a lot?

texts:
So I trained CNN model for people detection on caltech-pedestrian dataset:
Then I was curious and evaluated the model in every 1000th iteration on Evaluation toolbox(I guarantee, there is no bug in evaluation).

However, plot of the performance does not look so good. The miss rate spikes between 20K(20,000) and 30K iterations.

I am confused what does that mean. I mean usually we would expect the miss rate to decrease as we train the model more.

I am using yolo object detection network https://github.com/AlexeyAB/darknet/blob/master/cfg/tiny-yolo-voc.cfg

So learning options are the same as https://github.com/AlexeyAB/darknet/blob/master/src

I also tried 10x smaller learning rate.
However, no luck:


texts:
i have got a question. i don't understand how to develop my DLModel.
I'm working on DGA Detection with this type of dataset:

1,d o m a i n 1
0,d o m a i n 2
1,d o m a i n 3
....

1 for dga and 0 for legit domain.

so i trained fasttext that gave me vector rapresentation for every char i have got in my dataset (39 char).

Than i would like to train this model

model.add(layers.LSTM(128))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(1))
model.add(layers.Activation('sigmoid'))
model.compile(loss="binary_crossentropy",optimizer="adam",metrics=['accuracy'])
model.summary()

but when i try to fit my model with:

model.fit(xtrain,y_train,batch_size=BATCH_SIZE,epochs=EPOCHS,callbacks=callbacks_list,verbose=1)

i had always error:

indices[7,1] = 39 is not [0,39]

i think that depends from shape of my input and output but i can't figure out where is the problem.
For example my
xtrain shape is (79998, 143) --> every line is a domain, ad 143 is the max input.
weight matrix is (39, 100)

i'm new to this studies and i'm really glad if someone could help me!

thanks!


texts: Why Deep Reinforcement Learning fails to learn how to play Asteroids?

texts:
Deep Q-learning, A3C, policies evolved with genetic algorithms, they all fail to learn Asteroids, or at least perform way worse than human. From the hardest Atari games according to RL most of the focus is on Montezuma's revenge, which clearly suffers from sparse rewards. However I don't think this is the case of Asteroids (video), since for every asteroid shot a reward is provided. Why DRL performs that bad then?

Here are some papers that report a bad result on Asteroids (some articles refer to each other):

Human-level control through deep reinforcement
learning
Massively Parallel Methods for Deep Reinforcement Learning
Deep Reinforcement Learning with Double Q-learning
Dueling Network Architectures for Deep Reinforcement Learning
Prioritized Experience Replay
Rainbow: Combining Improvements in Deep Reinforcement Learning
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
A Neuroevolution Approach to General Atari Game Playing


texts: What's the difference between transfer learning and feature extraction in CNN?

texts:
So from what i understand, transfer learning is the fact of training a model on a dataset where you have a lot of data, then keeping most of trained coefficients, and only re-training the last layer on a dataset for a problem where you have a little amount of data
And feature extraction for CNN means that you have a generic model which extracts features from the image, and then you only need to train the last layer to solve a particular problem on images.
So is feature extraction just a particular case, or is there some subtlety i did not understand ?


texts: How to map ground truth to prediction for UNet architecture

texts:
I've gone through the paper describing the UNet convolutional neural network a number of times, but am still having trouble figuring out how to connect the output of the network to the ground truth targets.  Below is an image depicting the architecture of the network.

(source: uni-freiburg.de)
As can be seen from the figure, the output is a 388 x 388 x k matrix (where k is the number of classes).  The target segmentation in this case should be 572 x 572 (same spatial dimensions as the input image).  How do we match these up?  Are we suppose to perform some kind of interpolation on the output feature map to get it to match the dimensionality of the input?


texts: What are some easy to learn machine-learning applications?

texts:
Being new to machine-learning in general, I'd like to start playing around and see what the possibilities are.

I'm curious as to what applications you might recommend that would offer the fastest time from installation to producing a meaningful result.

Also, any recommendations for good getting-started materials on the subject of machine-learning in general would be appreciated.


texts: Is it ethical to claim experience with big data when the data isn't a part of the new advertising/social media/retail fad?

texts:
Obviously most employers, when hiring a data scientist, would prefer experience with big data and/or data science.  But what can one safely assume they will acknowledge as experience?

Let's say someone often launches software on a computing cluster that typically generates an amount of data.  I'm not sure what the best measure of this data is for data science.  I'll call it one or two thousand rows, 200k or 300k points per row...certainly under 500k.  Then for each point, let's call it 25 or 30 values.  This amounts to 30 or 40 gig of data.  300 or 400 times of this and you can call it a study - maybe one or two studies per year.  I'm under the impression that this is much smaller than a data scientist at Google or Facebook would be used to, but it's certainly too big for my home computing systems.

If someone's been working with this for years (some people at this company have been doing it since before data science was coined/before social media existed), is it fair for them to claim big data experience?  According to this answer, it's not the amount of data but what needs to be done with the data that matters - is that a universally accepted opinion?

For what it's worth, working with this data entails manipulation/cleaning of the data with some proprietary languages, shell scripts, and a lot of Python.  A little bit of R but that's a more recent thing.  It involves tons of data visualization, drawing conclusions, and presenting to management/convincing decision makers.  Some of it involves trend determination, extrapolation, and comparisons made between data sets that aren't related in a straightforward way, so it sounds data science-ish to me.  But I will be the first to admit that I have a limited understanding of what data science currently is.

...bonus points if you can tell me whether or not this is an Easter egg or is the actual answer total for this site at the moment:

Edit:

I'll try to clarify.  What do data science employers acknowledge as experience with big data/data science?  Does the size of the data above qualify experience with it?  Or is it universally accepted among those in the field that it's not at all the size of the data, but what you need to do with the data?


texts: NGBoost and overfit - which model is used?

texts:
While training an NGBoost model I got:
[iter 0] loss=-2.2911 val_loss=-2.3309 scale=2.0000 norm=1.0976

[iter 100] loss=-3.3288 val_loss=-2.8532 scale=2.0000 norm=0.7841

[iter 200] loss=-4.0889 val_loss=-1.5779 scale=2.0000 norm=0.7544

[iter 300] loss=-4.8400 val_loss=8.8107 scale=2.0000 norm=0.6710

[iter 400] loss=-5.4463 val_loss=51.7171 scale=2.0000 norm=0.5999

It looks like overfit occurred between iterations 100 and 200.
Is the best (val_loss wise) model saved, or did I get the last one reported (with a massive overfit, -5.4463 in train loss vs 51.7171 in validation loss)?
If I really do get the overfitted model, how can I introduce early stopping (or model saving) based on the validation score?


texts: what are effects of working with categorical dataset

texts:
I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over sample and created a model. I also tried similar approach with undersampling. Both the method with logistic regression performs mediocre. I want to know how having too many categorical variables impacts the model and possible efficient way to approach the problem

feature1:1-3
feature2:0-1
feature3:0-3
feature4: 1-4
feature5: 0-2
feature6: 0-5
feature7: 1-4
feature8: continuous( max 10)
feature9 continuous( max 10)
class: 0-1


texts: Basics: What is the correct sequence for preparing simple data for ML?

texts:
I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one).

I was just wondering what would be the best way to organise the data to avoid redundancy with the following steps:

Feature Selection
Account for gaps in data(Imputing)
One hot encoding to allow non numerical features
Split data into training and validation sets
Then proceed to fit and predict with the model.

My main query is whether it is better to split the data before encoding it, or to only do the split after completing the encoding?


texts: Question Answering without BERT, transformers

texts:
I have a NLP problem, in which I need to find the start and end index of a known block. It is similar to Question answering. So, I am looking for a working code which carries out question answering or text block extraction.
Most of solutions available are using Transformers or BERT. I do not want to use BERT, as it is difficult to train. It needs a lot of data to finetune the pretrained model.
Can someone help me with a vanilla version of tensorflow code or something else, with good accuracy?


texts: Are there pitfalls in using the output of machine learning model, such as a neural net as the input to a traditional GLM or similar?

texts: To be more specific, loss reserving models in actuarial science, such as the chain ladder method, can be expressed as GLMs. I have developed a predictive model using neural nets which takes into account some aspects of the insured (it is an individual risk model). Can the output of this model be safely used as an input to the insurance company's existing loss reserve model?


texts:
I have a problem during upsampling operation in PySpark. My dataframe is:

df_upsampled.show()
+-------------------+------------------+
|                 et|           average|
+-------------------+------------------+
|2018-08-15 00:10:00| 4.165999948978424|
|2018-08-15 00:15:00|              null|
|2018-08-15 00:20:00|3.6580000072717667|
|2018-08-15 00:25:00|              null|
|2018-08-15 00:30:00|0.9999999925494194|

What I want to do is that by using Spark functions, replace the nulls in the "sum" column with the mean value of the previous and next variable in the "sum" column.

Wherever there is a null in column "sum", it should be replaced with the mean of the previous and next value in the same column "sum".

In this case, first null should be replaced by (4.16599 + 3.658)/2 = 3.91 and so on for the rest nulls..

What would be a good way to do this?


texts: Is there any method to determine which clustering algorithm to use on a particular dataset?

texts:
I'm having a hard time getting kmeans to cluster data effectively. It fails to segment data well even for a simple attribute with 5 categories. I'm aware of DBSCAN, Hierarchical Clustering and GMM. However, just wanted to know if there's any way (visual or otherwise) to narrow down the clustering algorithm which might work on the dataset in question, before I start to write the code for each of these algorithms.

Thanks in advance.


texts: Why tensorflow saver meta file is getting large?

texts:
Far as I know, a meta file for saver is for the information of model like structure and operation. So the file size should be same, but the first meta file size is much smaller than the last meta file. Why does this happen?

In addition, if I set 'write_meta_graph'=false, there is no meta file at all. Then how to restore the model later?


texts: Can we use one hot encoding instead of for loops?

texts:
I have an anomaly detection model, that I run per store with a bunch of features.
I intend to run this code, everyday, per store. Now, lets say I have 8000 stores, I would imagine, I should write a for loop and iterate and create the model per store:

final is the dataframe that consists of all the stores

Something like:

for store in final['StoreNbr'].unique():
   run the model.

My input features include: Store number, Cashier, etcc (many other features). Can I use one hot encoding for the StoreNbr column and then run the model once as opposed to a for loop and running the model per storenbr?


texts: Why is the exploding/vanishing gradient problem not solved by line search?

texts:
The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don't we just not take our step size to be proportional to the gradient, and instead do a line search?
The intuition for why gradient descent ought to work is just that if we move in the direction of steepest descent, it should tend to make the function smaller. However, it's not obvious why the step-size should have to be proportional to the magnitude of the gradient. In an old paper on the subject, Haskell shows that gradient descent converges so long as we do a line search, he doesn't consider the case of proportional step size. The more I think about it, the less good reason I see at all why taking the step size to be proportional to the gradient should be considered a good or natural way to do gradient descent.


texts: Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies

texts:
I'm still in the process of learning, so I'm sorry if this doesn't make much sense.

I'm doing Kaggle learns micro courses, and to work with missing tabular data we learned about using pipelines with imputers for numerical data and OneHotEncoding for categorical data. On the other hand, we also learned about using just pd.get_dummies which from my understanding just works as OneHotEncoding (please, correct me if I'm wrong about this).
However, using just pd.get_dummies seems to give me better results than the other method but in the other method I'm even going as far as to impute the numerical values where as with pd.get_dummies I'm not even sure what happens to missing numerical data.

So, what's going on here?

Thanks!

PS: After we use pd.get_dummies we then use "align". I read the docs but I'm still not entirely sure the purpose of doing this

X_train = pd.get_dummies(X_train)
X_valid = pd.get_dummies(X_valid)
X_test = pd.get_dummies(X_test)
X_train, X_valid = X_train.align(X_valid, join='left', axis=1)
X_train, X_test = X_train.align(X_test, join='left', axis=1)


texts: Data driven project for a self-catering accommodation business

texts:
I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a masters in Data Science (don't slag me too much pls)
So I've started a project with this small business that offer self-catering accommodation like Airbnb and what they want is to increase their occupancy. Basically they have properties which perform well and some that doesn't and they want to know why. I've got a list with variables but no clue how to use any other software than weka and I am not sure if it will be appropriate. I am also clueless how to proceed with the analysis so I would be grateful if you have any ideas. I've added a screenshot of the variables to give you an idea.
My university supervisor is not very helpful so I've decided to reach out for help here and any tips will be much appreciated.

Would it be more helpful to provide you with the excel sheet? I am not sure how to upload it


texts:
I am trying to implement simple recommender system and I am trying to understand different approaches to achieve my goal.

My dataset consists of users and items that they bought. I have information about what items user bought and descriptions of these items in form of titles.

At first I though I could use user based collaborative filtering approach but I am stuck at this. I am not quite sure how to calculate similarity for boolean data.

When I have data like this for example

   1  2  3  4
A  0  1  0  1
B  0  1  0  1
C  1  0  1  1
D  0  1  0  0
E  0  0  1  1

And I want to recommend items for user E, so how should I calculate similarity in this case? I chosen for example cosine similarity from scikit learn module in python. But I am not quite sure what should be considered as input. From what I read it should be only vectors of items that two users for which similarity is calculated have in common.

So for example if I wanted to compute similarity between user E and C what should be my input? Because if I input only values that they have in common it does not make sense right? Beacuse input will be [1, 1] and [1, 1] and for that similarity is 1.

Then I tried to input the whole vector like this:

from sklearn.metrics.pairwise import cosine_similarity
from numpy import array, reshape

c = array([1, 0, 1, 1])
e = array([0, 0, 1, 1])

result = cosine_similarity(c.reshape(1, -1), e.reshape(1, -1))

>>> result is 0.81649658

And this approach I think makes more sense but I am not sure if it is acceptable based on what I learned about this type of recommendation.


texts: How to train more models on 2 GPUs with Keras?

texts: I got 2 GPUs of type NVIDIA GTX 1070 Ti. I would like to train more models on them in such a way that half of the models are trained on one GPU only, and half on the other, at the same time. So as training goes, one model goes to GPU1, the next model goes to GPU2, ... I don't want to train one model on the two GPUs. I use Keras - Python with TensorFlow back-end. Can you please recommend resources where I can see how to do this? Most examples/articles online cover the case only if you want to distribute one model on the two GPUs. Thank you.


texts: Designing CNN that does one column convolution across the x-axis

texts:
I am currently working on designing a certain number of CNN for extracting features from images.

The images are spectogram, and each have a shape being (276,x,3).
X is here the number of column, which incidently also the length of the feature vector that should  be created.

So the cnn somehow has to use a kernel that has to be 276 rows and 1 column wide, but is it possible in keras to make a 2d kernel and perform 1d convolution. The most important factor is the 1d convolution and the shape of the 2d kernel, as it is being used to alter the importance of some of the entries in the spectogram.


texts:
I am building a Job Recommendation System where I have Student Data for different subjects in Machine Learning(Data Viz, Python, Statistics, etc) and their skills from the resume. Need to Recommend jobs scraped from a reputed website.

Current thoughts- Using CF for finding similar students with similar skills and ranking them based on their performance in the subjects and then matching them with the scraped jobs for the relevant skills.

Based upon Job Titles- options are word2vec to find semantics from job titles, Lstms for semanctics etc.

Based upon Job's key skills- Can do direct strin matching and filter based upon Experience.

I want a good approach for my Problem.


texts: Is it right to impute Train and Test set?

texts:
I am experimenting with a dataset and I have a couple of columns with high cardinality. So, I performed mean target encoding (given that my dataset had more than 50000 observations). But, before doing this, I split my data (using Pandas) into training and test and then encoded them. After checking, I realized that my training and testing datasets had Nan's. Since you can not run an ML algorithm with Nan's, I decided to impute both of them with MissForest.

My question is: Is what I have done acceptable?


texts: Is there any paper of object detection that detect not trained class or object?

texts:
After reading some object detection papers (kinds of R-CNN, YOLO,...) I'm wondering if there is a detector that detects objects about not trained class.

For example my model is not trained to detect buildings, but is it possible to make my model detect building or some vehicles just with pixel data?


texts: How to use K.function with two inputs and a concatenate layer?

texts:
In Keras, I try to compute use the K.function between some layers. But I get an error when I use a Concatenate layer.

Here is a minimal code you can try yourself:

import numpy as np
from keras.layers import *
from keras.models import Model
def test_model(concat, conv2_dim=32, kernel=(3,3), dropout=0,
                     optimizer='adam', extra_conv=False, padding='valid',
                     width_dense=1024, batch_norm=False):
    img_shape = (64,64,1)

    image_input = Input(shape=img_shape)
    entropy_vector = Input(shape=(1,))

    conv = Conv2D(conv2_dim, kernel_size=kernel,
                     activation='relu', padding=padding,
                     input_shape=img_shape)(image_input)
    conv = MaxPooling2D(pool_size=(2, 2), padding=padding,)(conv)

    final_conv = Flatten()(conv)
    dense = Dense(width_dense, activation='relu')(final_conv)

    concat_layer = Concatenate()([dense, entropy_vector])

    if(concat): #Skipping
        flat = Dense(width_dense//4, activation='relu')(concat_layer)
    else:
        flat = Dense(width_dense//4, activation='relu')(dense)

    output = Dense(1, activation='sigmoid')(flat)

    model = Model(inputs=[image_input, entropy_vector], outputs=[output])
    model.compile(loss='binary_crossentropy',
                  optimizer=optimizer,
                  metrics=['accuracy'])
    return model

X = [np.ones((1,64,64,1)), np.ones(1,)]
m = test_model(concat=False) # you can try with False
fn = K.function(m.input, [m.output])

m.predict(X)
fn(X)

But I get this error if concat=True in the test_model function:

InvalidArgumentError: ConcatOp : Ranks of all input tensors should match: shape[0] = [1,1024] vs. shape[1] = [1]
     [[{{node concatenate_18/concat}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dense_112/Relu, _arg_input_44_0_1/_4327, concatenate_18/concat/axis)]]
     [[{{node dense_114/Sigmoid/_4329}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_68_dense_114/Sigmoid", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]


texts: I received this error message: Found input variables with inconsistent numbers of samples: [15573, 15600]. How do I resolve it please?

texts:
Data source

I received that error and when running my code, I realised that the issue lies with my X_train and y_train.

The link below is the source of my data (test.csv and train.csv)

My X_train.shape gave me (15573,) and my y_train gave me (15600,)

How do I reshape this?


texts: DecisionTree split quality using chi-squared statistic

texts:
In most of the popular packages (rpart etc.), the decision tree split is based on information gain, entropy or Gini index.

Is there a popular decision tree package ( either 'R' or python) where the node split quality can be specified using chi-squared statistic?


texts: Can ReLU replace a Sigmoid Activation Function in Neural Network

texts:
I'm new into this and I'm trying to replace the sigmoid activation function in the following simple NN with ReLU. Can I do that? I've tried replacing the sigmoid function, but it's not working. The output should be the AND gate(if input (0,0)-> output 0).

import numpy as np

 # sigmoid function
def nonlin(x, deriv=False):
   if(deriv == True):
       return x*(1-x)
   return 1/(1+np.exp(-x))

# input dataset
X = np.array([[0, 0],
          [0, 1],
          [1, 0],
          [1, 1]])

# output dataset
y = np.array([[0, 0, 0, 1]]).T

# seed random numbers to make calculation
# deterministic (just a good practice)
np.random.seed(1)

# initialize weights randomly with mean 0
syn0 = 2*np.random.random((2, 1)) - 1

for iter in xrange(10000):
    # forward propagation
    l0 = X
    l1 = nonlin(np.dot(l0,syn0))

    # how much did we miss?
    l1_error = y - l1

    l1_delta = l1_error * nonlin(l1, True)

    syn0 += np.dot(l0.T,l1_delta)


texts: Predicting yearly income with linear regression using Python


texts: Estimating forward velocity for a swimmer

texts:
With a modern IMU with 9 angles of freedom collecting accelerometer, magnetometer and gyroscope data on 3 axis, what would be the best approach on filtering the data and handling it to accurately estimate the forward velocity of the swimmer?

My approach was to:
1. Use a 3-point moving average to get rid of any vibrations caused by unneeded movements
2. Use a median average to get rid of repetitve movements such as shakes or water  resistance
3. Perform integration of accelerometer data.

At this point, I'm unsure of my data filtering approach. Is there a better approach? What methods should I use for acc. integration?


texts: How to deal with unbalanced data in pixelwise classification?

texts:
I'm trying to train a fully convolutional network to generate binary image showing roads from satellite photos. For a long time, I struggled to get the network to output anything but black - then I figured out the problem. I have plenty of training data, but of course most of the output pixels
will be black, since roads take only small percentage of space.

I've created a toy example showing the problem:

from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv2D, MaxPooling2D
import numpy as np
import random
import sklearn.model_selection

THRESHOLD = 3
BALANCED = False

if BALANCED:
    X = [[
            random.randint(THRESHOLD + 1, 255) if
                random.randint(0, 1) else
            random.randint(0, THRESHOLD)
            for _ in range(10)] for _ in range(10**4)]
    Y = [[0] if x[0] <= THRESHOLD else [1] for x in X]
else:
    X = [[random.randint(0, 255) for _ in range(10)] for _ in range(10**4)]
    Y = [[0] if x[0] <= THRESHOLD else [1] for x in X]

X, Y = np.array(X), np.array(Y)
X_train, X_test, Y_train, Y_test = sklearn.model_selection.train_test_split(
        X, Y, test_size = 0.2)

model = Sequential()
model.add(Dense(8, activation = "sigmoid", input_shape = (10, )))
model.add(Dense(1, activation = "sigmoid"))

model.compile(loss = "binary_crossentropy", optimizer = "adam")

model.summary()
model.fit(X_train, Y_train,
        batch_size = 10, epochs = 10,
        verbose = 2,
        validation_data = (X_test, Y_test))

Y_res = model.predict(X_test)
Ys0 = [y[0] for x, y, y_opt in zip(X_test, Y_res, Y_test) if y_opt[0] == 0]
Ys1 = [y[0] for x, y, y_opt in zip(X_test, Y_res, Y_test) if y_opt[0] == 1]

print(np.mean(Ys0))
print(np.mean(Ys1))

Here, I artificially create data, with X being ten random bytes, and Y being boolean thresholding first byte on the value of three. That means normally only around 1/64 of all samples will be generated with y equal to 0, similarly to my original problem. There's also a second dataset, with the same decision boundary, but more evenly (class-wise) distributed data - set by changing BALANCED toggle in the code. The results confirm my suspicions.

For balanced set, actual 0's are given, on average, value of 0.025. 1's - 0.97. That's reasonable. On the unbalanced, raw dataset, 0's have mean of 0.87, and 1's - 0.98. That means the zeros are pretty much always classified as one anyway. On my real data, the effect is even worse, and the returned value is pretty much equal to percentage of black, regardless of real classification.

I tried adding class_weights = [100, 1] to fit call, but it didn't change much. Is there a good strategy for dealing with such unbalanced data? In this toy example I could probably sample the x-y pairs (e.g. keep only 5% of 1-classified data), but I don't think this would work for my  full problem - since I'm using FCN, I'm putting a whole image into the network at once. There are no images with 50% roads, so I cannot possibly sample images to make roads take enough space.

I also tried, quite randomly, changing loss functions, including using one for categorical data and one-hot encoding expected outputs (i.e. [1, 0] and [0, 1] instead of 0 and 1). This did not fix anything either.


texts:
In the past, when trying different machine learning algorithms in order to solve a problem, I used to write drown the set of approaches on a notebook, keeping details such as features, feature preprocessing, normalization, algorithms, algorithm parameters... therefore, building a hand-written logbook.

However, currently I'm concerned about using a 'more professional' tool, so that I can keep more details and even share it with other team-members, who are also able to stamp their approaches.

It would be great an automated and collaborative tool that keep track of the work done, considering details like: features, algorithms, algorithms parameter, data pre-process, data, metrics... beyond a collaborative Google Drive Spreadsheet for instance.

How are you solving this? How are you keeping track of the work done? What's your logbook tool?

Thank you very much in advance.


texts:
I have 5 different classes in which I want to classify some data points. I'm using RNN with Echo-state networks (Reservoir computing).

Normally, a straightforward method consists of computing the outputs y and deciding the class using the argmax[k](y) with k is the dimension of the right class. Now I found another method that consists of computing the argmax[k](sum(y)), and that is what I didn't understand.

I've attached an image of the mathematical approach. The latter is understandable, but it appears not logical for me, since the sum of y over the data points timescale gives the same result for each time sequence, meaning that all data points will be classified in the same class(which corresponds to the max of the sum(y)).

Can someone what I don't understand?


texts: Can Single Node Hadoop Cluster be installed on a system with 1gb RAM

texts:
I am trying to learn hadoop, would like to know if for basic single node installation 1gb RAM system would be enough or we need more RAM. It would be helpful if someone can share what other minimum system requirements which I can setup single node setup.
I trying to check on Apache Hadoop site but there is no specific mention of minimum system requirements for installation.
Thanks


texts: Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.)

texts:
I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following:

Assumed data type (numerical vs string)

Assuming it's determined to be a numerical column:

Mean and Std Deviation
Max and min values
Number of NaN's
% of NaN's

Assuming it's determined to be a string column:

Number of distinct string values

So the tool takes the matrix and outputs some kind of report along these lines. The goal is to do some sanity checking on my input data set as well as checking data transformations along the way. For example, if I'm doing some kind of transformation and values in column A go from 1% NaN to 60% NaN, then I did something really bad.

Does such a tool exist?


texts: How can I implement deconvolution on CNN (TensorFlow)?

texts:
I'm new in tensorflow and machine learning. Could you explain me how implement deconvulation on this CNN model(http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/) for text classification?

CNN model:

class TextCNN(object):
"""
A CNN for text classification.
Uses an embedding layer, followed by a convolutional, max-pooling and softmax layer.
"""
def __init__(
  self, sequence_length, num_classes, vocab_size,
  embedding_size, filter_sizes, num_filters, l2_reg_lambda=0.0):

    # Placeholders for input, output and dropout
    self.input_x = tf.placeholder(tf.int32, [None, sequence_length], name="input_x")
    self.input_y = tf.placeholder(tf.float32, [None, num_classes], name="input_y")
    self.dropout_keep_prob = tf.placeholder(tf.float32, name="dropout_keep_prob")

    # Keeping track of l2 regularization loss (optional)
    l2_loss = tf.constant(0.0)

    # Embedding layer
    with tf.device('/cpu:0'), tf.name_scope("embedding"):
        self.W = tf.Variable(
            tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
            name="W")
        self.embedded_chars = tf.nn.embedding_lookup(self.W, self.input_x)
        self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)

    # Create a convolution + maxpool layer for each filter size
    pooled_outputs = []
    for i, filter_size in enumerate(filter_sizes):
        with tf.name_scope("conv-maxpool-%s" % filter_size):
            # Convolution Layer
            filter_shape = [filter_size, embedding_size, 1, num_filters]
            W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
            b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
            conv = tf.nn.conv2d(
                self.embedded_chars_expanded,
                W,
                strides=[1, 1, 1, 1],
                padding="VALID",
                name="conv")
            # Apply nonlinearity
            h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
            # Maxpooling over the outputs
            pooled = tf.nn.max_pool(
                h,
                ksize=[1, sequence_length - filter_size + 1, 1, 1],
                strides=[1, 1, 1, 1],
                padding='VALID',
                name="pool")
            pooled_outputs.append(pooled)

    # Combine all the pooled features
    num_filters_total = num_filters * len(filter_sizes)
    self.h_pool = tf.concat(pooled_outputs, 3)
    self.h_pool_flat = tf.reshape(self.h_pool, [-1, num_filters_total])

    # Add dropout
    with tf.name_scope("dropout"):
        self.h_drop = tf.nn.dropout(self.h_pool_flat, self.dropout_keep_prob)

    # Final (unnormalized) scores and predictions
    with tf.name_scope("output"):
        W = tf.get_variable(
            "W",
            shape=[num_filters_total, num_classes],
            initializer=tf.contrib.layers.xavier_initializer())
        b = tf.Variable(tf.constant(0.1, shape=[num_classes]), name="b")
        l2_loss += tf.nn.l2_loss(W)
        l2_loss += tf.nn.l2_loss(b)
        self.scores = tf.nn.xw_plus_b(self.h_drop, W, b, name="scores")
        self.predictions = tf.argmax(self.scores, 1, name="predictions")

    # CalculateMean cross-entropy loss
    with tf.name_scope("loss"):
        losses = tf.nn.softmax_cross_entropy_with_logits(logits=self.scores, labels=self.input_y)
        self.loss = tf.reduce_mean(losses) + l2_reg_lambda * l2_loss

    # Accuracy
    with tf.name_scope("accuracy"):
        correct_predictions = tf.equal(self.predictions, tf.argmax(self.input_y, 1))
        self.accuracy = tf.reduce_mean(tf.cast(correct_predictions, "float"), name="accuracy")

I know about tf.nn.conv2d_transpose, but I don't understand how can I get original data (not embedded_chars)


texts: Is it better to augment both training and validation sets or just training set?

texts: Is it better to augment data both training and validation sets or just the training set in order to achieve the best accuracy possible on a convolutional neural network? why?


texts: Named Entity Recognition: NLTK using Regular Expression

texts:
Many times Named Entity Recognition (NER) doesn't tag consecutive NNPs as one NE. I think editing the NER to use RegexpTagger also can improve the NER.

For example, consider the following input:

  "Barack Obama is a great person."

And the output:

Tree('S', [Tree('PERSON', [('Barack', 'NNP')]), Tree('ORGANIZATION', [('Obama', 'NNP')]),
    ('is', 'VBZ'), ('a', 'DT'), ('great', 'JJ'), ('person', 'NN'), ('.', '.')])

where as for the input:

  'Former Vice President Dick Cheney told conservative radio host Laura Ingraham that he "was honored" to be compared to Darth Vader while in office.'

the output is:

Tree('S', [('Former', 'JJ'), ('Vice', 'NNP'), ('President', 'NNP'),
    Tree('NE', [('Dick', 'NNP'), ('Cheney', 'NNP')]), ('told', 'VBD'), ('conservative', 'JJ'),
    ('radio', 'NN'), ('host', 'NN'), Tree('NE', [('Laura', 'NNP'), ('Ingraham', 'NNP')]),
    ('that', 'IN'), ('he', 'PRP'), ('``', '``'), ('was', 'VBD'), ('honored', 'VBN'),
    ("''", "''"), ('to', 'TO'), ('be', 'VB'), ('compared', 'VBN'), ('to', 'TO'),
    Tree('NE', [('Darth', 'NNP'), ('Vader', 'NNP')]), ('while', 'IN'), ('in', 'IN'),
    ('office', 'NN'), ('.', '.')])

Here Vice/NNP, President/NNP, (Dick/NNP, Cheney/NNP) is correctly extracted. So, I think if nltk.ne_chunk is used first, and then if two consecutive trees are NNP, there are higher chances that both refer to one entity.

I have been playing with NLTK toolkit, and I came across this problem a lot, but couldn't find a satisfying answer. Any suggestion will be really appreciated. I'm looking for flaws in my approach.


texts: Why isn't leaky ReLU always preferable to ReLU given the zero gradient for x<0?

texts: It looks to me like the leaky ReLU should have much better performance since the standard ReLU can’t use half of its space (x < 0 where the gradient is zero). But this doesn't happen and in practice most people use standard ReLU.


texts: Index for efficient argmax(w.x) query ~ 20d

texts:
I'm looking for a spatial index that can efficiently find the most extreme n points in a certain direction, i.e. for a given w, find x[0:n] in the dataset where x0 gives the largest value of w.x and x1 the second largest value of w.x, etc... . Is there a name for this type of query? What would be an efficient data structure to use? x might have around 20 dimensions.

Thankyou!


texts: Aspect based sentiment analysis using machine learning approach

texts:
I am very new in machine learning. I have annotated data with category, aspect, opinion word and sentiment. for example, for the bellow text

"The apple was really tasty"

I have category->food, aspect-> apple, opinion word ->tasty and sentiment->positive. I have training data like this format.

How can I train a SVM classifier using this format of training set?
How to extract features like n-gram, POS and sentiment word to train the classifier?
Could you please suggest any beginning step for this aspect based sentiment analysis using machine learning algorithms?


texts: Will stochastic gradient descent converge for multivariate linear regression

texts:
I am trying to figure out if stochastic gradient descent for a multivariate linear regression will converge (assuming there is no mini-batching, i.e., the batch size is 1).
My guess is yes, based on the fact that stochastic gradient descent will converge for most well-behaved, convex functions, but the multivariate aspect is throwing me off, as well as the batching.
What is a good way to think about this problem, from both the multidimensional aspects as well as the hyperparameters?


texts: Dropping Missing Observations under MAR Assumption

texts:
Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and non-missing data but there is no theoretical background supporting that there are unobserved factors that can determine if the data is missing or not.
I want to drop the missing observations instead of using an imputation method. However, I am not sure how to prevent bias. I can control for the factors that explain if the data is missing or not simply by adding them into regression but I am not sure if this would be enough.


texts: Classify sensor data (multivariate time series) with Python's scikit-learn decision tree

texts:
i'm trying to apply scikit learns decision tree on the following dataset with the goal of classifying the data:

sensordata:

multiple .csv files
every .csv file has multiple sensors (see here)
each .csv file has one label (0 or 1)

So far I've tried to train my model with Pandas Series. It worked, but the decision tree couldn't differate the features/sensors. Is pandas series the right approach for analyse data like this? Or does anyone have another solution for this problem?


texts: Should the weights be rotated when using SciPy full convolution?

texts:
I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume that the signal.convolve2d handles this internally.

full_conv = signal.convolve2d(global_grad, filters, boundary='fill', mode='full')

Since my gradient check did not pass I now think it must be something wrong and the filters should be rotated. Could someone help me clarify this point? thanks


texts: Transfer Deep Learning from one aerial imagery datset to many others

texts:
I am new to Deep Learning but have been able to use RasterVision successfully to predict building footprints within a set of aerial imagery.
This aerial imagery data set is for a province of New Zealand. Now that I have a model that predicts successfully in this province, I am interested in how I could use this to predict in the many other regions of New Zealand. The problem is these regions are captured with differing sensors, resolution and with different color balancing applied (I have tried using my model in another region with poor results 70% recall as opposed to 92% in the region trained for).
I am guessing I could take my model as a base to begin training in another region...
My question is, is it conceivable that I could have a model trained that would predict with acceptable accuracy in many regions with differing resolution (0.1m --> 0.7m) and different color balancing or is the approach to take a base model and retrain for every different imagery dataset (which is obviously less ideal)?
Are there examples of such an approach across such differing aerial/satellite imagery?
I note this question does answer some of this in terms of resolution. What I am just as interested in is the impact and managing of differing color balancing across datasets
The other imagery datasets I want to start predicting on include these


texts: How to select features for Text classification problem

texts:
I am working on a problem where we need to classify user query into multiple classes.

  Problem:

  Suppose we are running a website for selling products. The website has a form where the user can write any complaint or issue.

  In order to resolve users issue, we thought to classify issues into predefined classes so that we can understand what type of problems users are facing.

Issues classes can be like

Class 1: Payment Issues
Class 2: Registration issues
Class 3: Order booking issues
Class 4: Problem accessing website app
...

I am thinking to apply random forest but cannot decide what features should I select. Any suggestions would be of great help thank you


texts: Can mini-batch gradient descent outperform batch gradient descent?

texts:
As I was reading and going through the second course of Andrew Ng's deep learning course, I came across a sentence that said,

With a well-turned mini-batch size, usually it outperforms either
gradient descent or stochastic gradient descent (particularly when the
training set is large).

But how is it possible? can mini-batch gradient descend really give us a better set of weights and biases even though it's not updating them based on the whole dataset? I can only think that it's maybe not overfitting and in that way, it can give better results.


texts: How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python)

texts:
I have some data which contains anomalies as well.

I want to model data using Gaussian distribution assuming variables are dependent in Python.

How can I model this? Should I use the PDF formula as a function, figure out the probabilities and take out the accuracy?

Note: This has been given as an assignment in my college. I am trying to figure out the solution.


texts: Results of quadratic SVM in Matlab are different from the results obtained in Python

texts:
I am trying to replicate a quadratic SVM classifier from Matlab to Python, however I am having different results regarding the accuracy. In Matlab the accuracy is 0.8955 meanwhile in Python the accuracy is 0.82089. I believe a 7% difference is high enough to be something there that I am doing wrong. For instance my Python code is the following:
        # Standardize the input data
        mean = np.zeros(descriptors)
        std = np.zeros(descriptors)
        scaled_mean = np.zeros(descriptors)
        scaled_std = np.zeros(descriptors)

        for i in range (0,descriptors):
            mean[i] = x_train[i].mean()
            std[i] = x_train[i].std(ddof = 1)
            x_train[i] = (x_train[i] - mean[i])/std[i]
            scaled_mean[i] = x_train[i].mean()
            scaled_std[i] = x_train[i].std(ddof = 1)

        #If gamma = 'auto' it goes into an infinite loop, Matlab gamma therefore is used
        clf = svm.SVC(kernel = 'poly', degree = 2, gamma = 0.4955, cache_size = 1000)
        clf.fit(x_train, y_train)
        # save the model to disk
        joblib.dump(clf, filename)

As my training dataset consists of 3 features with 66 rows each I have done no validation in the classification learner app that Matlab provides, but I do have in mind that overfitting might be a problem in the future.
Aside from that I would like to point out that I have used previously the scaler that sklearn provides to standardize, but since it does not take into account the ddof parameter that Matlab does I have chosen to compute the standardization myself.
I leave the datasets that I have used here, but please note that in the the matlab train.mat file only the first 3 columns are used to train, and the last column contains the labels (0,1), the remaining ones are not used at all.
https://ufile.io/f/rrf6g


texts: How to predict the probability of an event?

texts:
I have a dataset where a set of people donated for charity along with the dates of the donation. I have to find the probability of each donor donating in the next three months.

Data is available from August 2014 - February 2016. I have to predict the probability of each person donating for March-June 2016.

Any help would be appreciated?

Below is a snapshot of the data

id  date    amount
1   13-08-14    2485
1   21-11-14    2105
1   17-09-15    1359
2   13-08-14    2542
2   20-04-15    1276
2   12-10-15    2694
3   20-11-14    3556
4   28-07-15    3383
5   13-08-14    1698
5   11-12-14    1725
5   09-06-15    1376
5   17-09-15    3230

Regards


texts: What exactly are the data augmentation experimental Keras' layers doing?

texts:
From what I gathered, data augmentation consists in increasing your number of instances in your dataset by applying some transfromations. Let's say I want to classify images. If I apply a random rotation to every image in a data set containing $n$ images, I will obtain a new dataset with $2n$ images, $n$ pairs of the original image plus it's random-rotated counterpart.
Assuming this is true, I don't understand what keras experimental layers related to data augmentation are doing.
Take tf.keras.layers.experimental.preprocessing.RandomRotation . In the image classification tutorial, it puts this layer inside the Sequential model like this:
model = Sequential([
  layers.experimental.preprocessing.RandomFlip("horizontal",
                                                 input_shape=(img_height,
                                                              img_width,
                                                              3)),
  layers.experimental.preprocessing.Rescaling(1./255),
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(0.2),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dense(num_classes)
])

This is already kind of weird because a layer produces an output from an input (obviously) but it doesn't duplicate the image. Anyway, I decided to check this in the documentation, and, in effect, this is what is happening.
Init signature: layers.experimental.preprocessing.RandomRotation(*args, **kwargs)
Docstring:
Randomly rotate each image.

By default, random rotations are only applied during training.
At inference time, the layer does nothing. If you need to apply random
rotations at inference time, set `training` to True when calling the layer.

Input shape:
  4D tensor with shape:
  `(samples, height, width, channels)`, data_format='channels_last'.

Output shape:
  4D tensor with shape:
  `(samples, height, width, channels)`, data_format='channels_last'.

Therefore, I understand that I'm just randomly rotating the image, that is, changing each image in the dataset, but I'm not doing any data augmentation. However, I find this would make no sense, otherwise they wouldn't mention this as a data augmentation procedure. So what am I missing?


texts: Character-level seq2seq with LSTM in Keras for language declension

texts:
I am trying to create sequence to sequence mapping of words on Croatian language, to automatize declension  (https://en.wikipedia.org/wiki/Declension). English language is fairly simple in that regard, but most world languages do have extensive declension rules. Analogy in English language would be car -> cars, or bus -> buses. Since Croatian language has 14 forms for each noun (7 for singular, and 7 for plural), and there are a lot specific rules with exceptions, deep learning seems like a good strategy to try out. I have created a training dataset with about 2000 entries, and I would like to train a model that is capable of generating 13 other forms from the initial word. I followed several tutorials and examples, but I can not make it work, since model is returning floating-point values, instead of 0 and 1, which I need to revert one-hot encoded vectors for each word. I am guessing that some of the parameters in model are not correct, but after a couple of days trying, I was not able to make it work. My code is here:

https://github.com/matkosoric/deklinacije

Initial step would be to create mapping for just one form, and then others, and maybe eventually combine outputs.

What I am doing in my Jupyter Notebook is chartacter-level encoding, to get vectors like this:

[[1, 17, 14, 3, 15, 6, 7],
 [1, 17, 3, 10, 2, 19, 2, 8, 1],
 [1, 17, 3, 7, 6, 7, 4],
 [1, 17, 3, 5, 4, 12, 11],
 [1, 22, 5, 2, 9, 1, 7, 9, 1],
 [1, 14, 1],
...

Then I add zero-padding, and then I do the one-hot encoding of the characters. Finnaly, this is the code for model:

model = Sequential()
model.add(LSTM(28, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences=True, activation='softmax'))  #, return_sequences=True
# model.add(Dropout(0.2))
model.add(LSTM(28, activation='softmax', return_sequences=True, recurrent_activation="sigmoid"))          #, return_sequences=False
model.add(Dense(28, activation='sigmoid'))
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])    # binary_crossentropy | categorical_crossentropy
history = model.fit(X_train, Y_train,
          epochs=20,
          batch_size=128,
          validation_data = (X_val,Y_val)
         )

After training and using model for prediction for test set, I get something like this, which can not be converted back to meaningful character sequence:

[0.5826852  0.42405558 0.42048743 0.41398    0.425908   0.4192267
 0.4133946  0.41803166 0.4172448  0.42309627 0.4156795  0.419433
 0.416399   0.41344917 0.4158144  0.4196655  0.42249662 0.42083743
 0.42371705 0.41620597 0.41855162 0.42147878 0.41726097 0.41544187
 0.42190933 0.42433727 0.42017445 0.409134  ][0.5842657  0.4228464  0.41903406 0.41235432 0.42476133 0.41782877
 0.41161612 0.4165     0.415777   0.42182064 0.41404086 0.41801238

I am guessing that Dense layer at the end should have number 2 as a parameter, since I am expecting zero or one values, but it is not compiling because of return_sequences=True in the LSTM layer before.


texts: Why does removal of some features improve the performance of random forests on some occasions?

texts: I completed feature importance of a random forest model. I removed the bottom 4 features out of 17 features. The model performance actually improved. Shouldn't the performance degrade after removal of some features, given that some data has been lost? What are some reasons to explain the performance improvement?


texts: Is it a good idea to use tensorflow instead of numpy for numerical approximations?

texts:
I intend on perfroming some numerical approximations for a problem in physics.

The main gist of the program will be to perform a svd on large sparse matrices and also calculating the trace of a large matrix.

I used numpy/scipy and multiprocessing modules in python to do this, but it is not fast enough.

I also implemented the same code using tensorflow on some gpus. Tensorflow does the calulations much faster, but it takes a long time to send the data into the gpus and bring it out. Each instance of data is sent individually as float values.  Is there any way to send all the data at once into the gpu?

Is it a good or bad idea to use tensorflow for numerical calculations?

Below is a snippet of my code....

def Expectation_Value(density,N):
   expec=0
   for i in np.arange(1,N+1):
       a=tf.linalg.trace(tf.matmul(Sigma(i,N,Sz),density))
       expec+=a
return expec/N

def main():
   df=pd.read_csv('.....')
   sess=tf.Session()

   for l in range(len(df)):
   Delta=df['Delta'].iloc[l]
   Omega=df['Omega'].iloc[l]
   Gamma=df['Gamma'].iloc[l]
   J=df['J'].iloc[l]

    Sz_tens=tf.zeros(len(df))
    Expec=Expectation_Value(DMT,N)
    Sz_tens[l]=Expec
Sz_arr=Sz_tens.eval(session=sess)
df['Sz']=Sz_arr
sess.close()
df.head()


texts: Manually changing coefficients of a model

texts:
Is there a way to manually change parameters of a model in Orange 3?

I have a dataset and trained a logistic regression model on it. Now, I can alter the coefficients (beta values) of the logistic regression model and visualize how it changes the predictions.

Is there a simple way to do that in Orange? I understand the model parameters will not be optimal if I am manually change them, but still is there a way to do it?

Thanks.


texts: Why is the cosine distance used to measure the similatiry between word embeddings?

texts:
While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean distance suitable for this task.
Let us consider 2 vectors a and b. Where, a = [-1,2,-3] and b = [-3,6,-9], here b = 3*a, i.e, both the vectors have same direction but different magnitude. The cosine similarity between a and b is 1, indicating they are identical. While the euclidean distance between a and b is 7.48.
Does this mean the magnitude of the vectors is irrelevant for computing the similarity in the word vectors?


texts:
We know that the formula of Fisher Linear Discriminant regarding the weight vector is :

My question is does this formula put any constraint on the magnitude of the weight vector w? or the magnitude can take any value?


texts: Deviation towards the higher number side

texts: Let's say I have 10000 different observations. I want to calculate the mean deviation only towards the higher side of the observation. Do we have any term in statics for such a parameter?


texts: Product Recommendation based on purchase history

texts: I am dealing with problem where i have to increase the sales by product recommendation.I only have customer data and product that they have purchased.No ratings,reviews or feedback is present.What approach fit best for my problem.


texts: How to use prediction model after onehot encoding?

texts:
I have created a prediction model for this dataset
>>df.head()

    Service    Tasks Difficulty     Hours
0   ABC         24     1           0.833333
1   CDE         77     1           1.750000
2   SDE         90     3           3.166667
3   QWE         47     1           1.083333
4   ASD         26     3           1.000000

>>df.shape
(998,4)

>>X = df.iloc[:,:-1]
>>y = df.iloc[:,-1].values
>>from sklearn.compose import ColumnTransformer
>>ct = ColumnTransformer([("cat", OneHotEncoder(),[0])], remainder="passthrough")
>>X = ct.fit_transform(X)
>>x = X.toarray()
>>x = x[:,1:]

>>x.shape
(998,339)

>>from sklearn.ensemble import RandomForestRegressor
>>rf_model = RandomForestRegressor(random_state = 1)
>>rf_model.fit(x,y)

How can I use this model to predict Hours for user input in this format [["SDE", 90, 3]]
I tried
>>test_input = [["SDE", 90, 3]]
>>test_input = ct.fit_transform(test_input)
>>test_input = test_input[[:,1:]

>>test_input[0]
array([24, 1], dtype=object)

>>predict_hours = rf_model.predict(test_input)
ValueError

Since my dataset has many categorical values its not possible enter the encoded value of "SDE" as input, I need to convert "SDE" to onehot encoded format after receiving the input [["SDE", 90, 3]]
I don't know how to do it can anyone help?


texts:
When I was running the text mining I did not have an issue for 57 different searches. I was able to retrieve all of the records regardless of how many there were.
Until these 2 errors popped up.

I also went back and tried previous searches that worked fine and this issues poped up.

I tried running the same search on a different computer and did not run into this issue.

I have already uninstalled and reinstalled the program twice with no change.


texts: "Understanding Machine Learning: From Theory to Algorithms" the universal approximation theorem

texts:
I'm readying on "Understanding Machine Learning: From Theory to Algorithms" the Universal approximation theorem:

  ..."Networks are universal approximators. That is, for every fixed precision parameter, $\epsilon >0 $, and every Lipschitz function $f : [−1; 1]^{n} \rightarrow [−1; 1]$, it is possible to
  construct a network such that for every input $\textbf{x} \in [−1; 1]^{n}$, the network outputs
  a number between $f(x) − \epsilon$ and $f(x) + \epsilon$".

It seems to me that the function $f : [−1; 1]^{n} \rightarrow [−1; 1]$ is Boolean.

I think that the n-dimensional unit hypercube ${\displaystyle [-1,1]^{n}}$ can be replaced with a compact set of $ \mathbb{R} ^ {n} $ but the codomain makes me puzzled.

I was expecting a function:
$f : \mathbb { R } ^ { n } \rightarrow \mathbb { R }$


texts: How to check if the dataset contains sufficient information to predict a continuous variable

texts:
Is there a way to check if a dataset generally contains enough information to predict a target variable?  In other words, can you calculate something like a correlation-coefficient between all features X and the target y, instead of training models?

Just to get an estimate of how good you could probably get, or if there is a constellation of features in X that correlate with y?


texts:
I have trained a sequential model with keras for MNIST dataset and this is the code I've used.

# Create the model
model = Sequential()

# Add the first hidden layer
model.add(Dense(50, activation='relu', input_shape = (X.shape[1],)))

# Add the second hidden layer
model.add(Dense(50, activation='relu'))

# Add the output layer
model.add(Dense(10, activation = 'softmax'))

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics = ['accuracy'])

# Fit the model
model.fit(X, y, validation_split=.3)

Output:

Train on 1750 samples, validate on 750 samples
1750/1750 [==============================] - 0s - loss: 0.1002 - acc: 0.9811 - val_loss: 0.3777 - val_acc: 0.8800

Can you explain what is loss, acc , val_loss, val_acc? How can I know my model performance from these metrics in output. Please explain if possible.


texts: Best way to classify datasets with mixed types of attributes

texts: I would like to know what is the best way to classify a data set composed of mixed types of attributes, for example, textual and numerical. I know I can convert textual to boolean, but the vocabulary is diverse and data become too sparse. I also tried to classify the types of attributes separately and combine the results through meta-learning techniques, but it did not work well.


texts: How to implement co-occurance matrix in Python which only takes into account next word relationships

texts:
I've been trying to create a dictionary version of a co-occurence matrix in Python, such that it only counts instances where one word follows another in a set of sentences. I found this implementation here, and it's been very helpful, however it measures the "complete" co-occurence, i.e. both the "forward" and "backward" relationships between words. Does anybody have any idea how I might achieve this?
Below are the original question, and the accepted solution:
Original Question (asked by Harika M):

To implement co-occurence matrix in sucha a way that number of times word1 occured in context of word2 in neighbourhood of given value, lets say 5. There are 100 words and a list with 1000 sentences. So how can i calculate co-occurence matrix of size (100* 100) using python?

Orginal Solution (by Kasra Manshaei):
from nltk.tokenize import word_tokenize
from itertools import combinations
from collections import Counter

sentences = ['i go to london', 'you do not go to london','but london goes to you']
vocab = set(word_tokenize(' '.join(sentences)))
print('Vocabulary:\n',vocab,'\n')
token_sent_list = [word_tokenize(sen) for sen in sentences]
print('Each sentence in token form:\n',token_sent_list,'\n')

co_occ = {ii:Counter({jj:0 for jj in vocab if jj!=ii}) for ii in vocab}
k=2

for sen in token_sent_list:
    for ii in range(len(sen)):
        if ii < k:
            c = Counter(sen[0:ii+k+1])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c
        elif ii > len(sen)-(k+1):
            c = Counter(sen[ii-k::])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c
        else:
            c = Counter(sen[ii-k:ii+k+1])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c

# Having final matrix in dict form lets you convert it to different python data structures
co_occ = {ii:dict(co_occ[ii]) for ii in vocab}
display(co_occ)

Output:
Vocabulary:
 {'london', 'but', 'goes', 'i', 'do', 'you', 'go', 'not', 'to'}

Each sentence in token form:
 [['i', 'go', 'to', 'london'], ['you', 'do', 'not', 'go', 'to', 'london'], ['but', 'london', 'goes', 'to', 'you']]

 {'london': {'go': 2, 'to': 3, 'but': 1, 'goes': 1},
 'but': {'london': 1, 'goes': 1},
 'goes': {'london': 1, 'but': 1, 'you': 1, 'to': 1},
 'i': {'go': 1, 'to': 1},
 'do': {'you': 1, 'go': 1, 'not': 1},
 'you': {'do': 1, 'not': 1, 'goes': 1, 'to': 1},
 'go': {'london': 2, 'i': 1, 'to': 2, 'do': 1, 'not': 1},
 'not': {'do': 1, 'you': 1, 'go': 1, 'to': 1},
 'to': {'london': 3, 'i': 1, 'go': 2, 'not': 1, 'goes': 1, 'you': 1}}

Essentially, I'm trying to recreate the dictionary output format, but instead of getting the context of each work, all I'm looking to get are - for example - how many times the word "london" follows the word "to" (and not how many times the word "to" precedes the word "london" in the example). Stated another way, I'm only looking to see how many times word n follows word n-1.
Thanks for any help!


texts: AttributeError: 'DataFrame' object has no attribute 'as_matrix' is the error


texts:
My problem is to cluster some time series together. But due to a huge length I was interested in using some methods to reduce the dimensionality. I was thinking of Discrete Wavelet Transform since the data are discrete. And I was wondering how many levels should I use to extract coefficients? Is there any best practise for that?
Also for clustering I was thinking to use Coeffs of Approximation only OR extract statistics such as mean, kurtosis , max etc.. from each coefficient (Approximation and all the Details)
Also, I've seen some methods using Fourier and extracting something like "m" components. But that does not seem very promising.
I'd like to hear your opinion on this. Thanks a lot.


texts: Best approach for A/B testing two different recommendation systems

texts:
I have two recommendation systems for musical preference which make a list of predictions for a particular user based on the songs they have saved in their library. The user then rates how good each recommendation was out of 6. I will be evaluating the performance of the recommendation systems based on the average rating given to songs recommended by system A and system B.
Let's use A to denote a song recommended by system A, and B to denote a song recommended by system B. For a particular user, should the recommendations be (AAAAAA or BBBBBB)  or should they be (ABABABAB)? I have implemented the first for now, being (AAAAAA or BBBBBB). Thus, in the current system, each respondent will be randomly assigned A or be, and only get the recommendations from that system. Is this the right approach, or does only recommending 1 system to each respondent bias them against what the other system could have recommended ?
Let's assume that B is far better than A. If we only recommend the same system to each user, and a user listens to songs which are all system A, they would never had heard system B, and the ratings of A would probably have been different (lower) if they had listened to the better system too. Is the ABABAB approach the best one ? Which is the best method to evaluate the performance of each system while reducing bias ?
Thanks


texts: convert dataframe without header to dictionary with a row of number

texts: I am trying to put a dataframe into dictionary  ,with the first column as the key ,the numbers in a row would be the value . how should I do it ? I tried to_dict but did not work , my df has no header or rownumber.


texts: How to optimize input parameters given target and scoring parameters

texts:
I'm new to machine learning/optimization, so I apologize in advance if this has been answered before. I don't know which search terms to use.

I have a large dataset where I have a number of input parameters $I_1$, ... $I_n$ ($n$ up to 10), a target parameter $T$ and a scoring parameter $S$. There are roughly a million rows of real-life data, where each row has the input parameters, the target parameter and the scoring parameter. The data is operational data from a piece of machinery where $T$ is output power, $S$ is fuel consumption and $I_1$, ... $I_n$ are tuning parameters for the machinery. The relationships of the tuning parameters to $S$ and $T$ are unknown but likely nonlinear. So I basically want to understand how to tune the machinery to produce the desired output power as fuel-efficiently as possible given the parameters I can tune.

What I need is some way of getting to a function where I input $T$ and get out the optimal combination of $I_1$, ... $I_n$ that minimizes $S$ for that value of $T$. I work in Python, and I assume that it's some combination of sklearn and scipy, but I haven't been able to figure out the steps to take for this type of problem. Thanks in advance.


texts: What are common ways to identify subject and object from a question?

texts:
I am looking for a way to extract the potential  subject and object from a question in French.

For the moment, I am building some handmade rules.
Alternatively, I started to think of using an already trained model for the task.
I wanted to use StanfordNLP parser but unfortunately it is not free for commercial use.

So, here comes the questions:

Are they ML models used for this task?
Are they labeled data sets (in french) which can help create ML models used for this task?

[Edit] I am looking mainly for open source tools/ libraries.


texts: Why does reducing polynomial regression to linear regression work?

texts:
Getting into machine learning, have a reasonable background in statistics and understand the basic principles of linear algebra (matrix multiplication etc.) - but am having a damn hard time figuring out why reducing a polynomial regression works.

For example, say we have this function:

$y =$ $\beta_0$ + $\beta_1$$x_1$ + $\beta_2$$x_2$$^2$

From what I've seen on 4 videos and 6 articles, we can use the following substitutions:

$x_2$ = 1
$x_3$ = $x$
$x_4$ = $x^2$

To create the following model:
$y =$ $\beta_0$ + $\beta_1$$x_1$ + ($\beta_2$$x_2$ + $\beta_3$$x_3$ + $\beta_4$$x_4$)

And then, fine - we can solve that as a normal multiple linear regression, and all is well.

But why, why does this work? I really cannot find an explanation for this.

Thanks!


texts: Features for blink detection in real-time single channel EEG

texts: I am looking to detect blink events in real-time single channel EEG. Classification of a moving window of samples to determine whether a blink artifact exists requires feature extraction (except when using deep learning, I am not experienced enough for this). What features would be useful to extract from a window of approx. 50-200 samples of time series data for detecting a blink event. The blink event can be easily seen in the below picture:


texts: How to perform Logistic Regression with a large number of features?

texts:
I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression.

According to the "rule if ten" I need at least 10 events for each feature to be included. Though, I have an imbalanced dataset, with 20% o positive class and 80% of negative class.

That gives me only 70 events, allowing approximately only 7/8 features to be included in the Logistic model.

I'd like to evaluate all the features as predictors, I don't want to hand pick any features.

So what would you suggest? Should I make all possible 7 features combinations? Should I evaluate each feature alone with an association model and then pick only the best ones for a final model?

I'm also curious about the handling of categorical and continuous features, can I mix them? If I have a categorical [0-1] and a continuous [0-100], should I normalize?

I'm currently working with Python.

Thanks a lot for your help!


texts:
I am interested in the field of named entity disambiguation and want to learn more about it. I have heard that there are contests organised by various associations on these kind of research topics. These contests are very helpful as they give a practical experience in these fields.

I found one such contest organised by Microsoft research here though the dates have already passed. Can anyone point me to any other such contests ? Also, is there a site which catalogues these contests so that one can just go there and know about all upcoming contests ?

Thanks in advance.


texts: Human activity recognition using smartphone data set problem

texts:
I'm new to this community and hopefully my question will well fit in here.
As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using smartphone data sets. As far as I'm concern this topic relates to Machine Learning and Support Vector Machines. I'm not well familiar with this technologies yet so I will need some help.
I have decided to follow this project idea (first project on the top)
The project goal is determine what activity a person is engaging in (e.g., WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, LAYING) from data recorded by a smartphone (Samsung Galaxy S II) on the subject's waist. Using its embedded accelerometer and gyroscope, the data includes 3-axial linear acceleration and 3-axial angular velocity at a constant rate of 50Hz.
All the data set is given in one folder with some description and feature labels. The data is divided for 'test' and 'train' files in which data is represented in this format:
  2.5717778e-001 -2.3285230e-002 -1.4653762e-002 -9.3840400e-001 -9.2009078e-001 -6.6768331e-001 -9.5250112e-001 -9.2524867e-001 -6.7430222e-001 -8.9408755e-001 -5.5457721e-001 -4.6622295e-001  7.1720847e-001  6.3550240e-001  7.8949666e-001 -8.7776423e-001 -9.9776606e-001 -9.9841381e-001 -9.3434525e-001 -9.7566897e-001 -9.4982365e-001 -8.3047780e-001 -1.6808416e-001 -3.7899553e-001  2.4621698e-001  5.2120364e-001 -4.8779311e-001  4.8228047e-001 -4.5462113e-002  2.1195505e-001 -1.3489443e-001  1.3085848e-001 -1.4176313e-002 -1.0597085e-001  7.3544013e-002 -1.7151642e-001  4.0062978e-002  7.6988933e-002 -4.9054573e-001 -7.0900265e-001

And that's only a very small sample of what the file contain.
I don't really know what this data represents and how can be interpreted. Also for analyzing, classification and clustering of the data, what tools will I need to use?
Is there any way I can put this data into excel with labels included and for example use R or python to extract sample data and work on this?
Any hints/tips would be much appreciated.


texts: For a multi-class classification problem, how to transform the target variable to a form that is usable by sklearn algorithms?

texts: I recently tried to create a model for predicting what class a sample belongs to out of 160 possible classes. These classes of the target variable are just simple strings describing workouts like "Push-ups", "Sit-ups" etc. I used sklearn's method LabelEncoder to transform the target variable, and it assigned every class a number between 0 and 159. I know that the reason why we are supposed to do one-hot encoding for categorical features is to avoid telling an algorithm something nonsensical like "blue is greater than red". My question then becomes, doesn't LabelEncoder tell the sklearn algorithm that "push-ups" is less than or greater than "sit-ups" if both of them are assigned distinct numbers between 0-159? Why or why not? This feels like a very stupid question, but I had real trouble finding an answer anywhere online.


texts:
I need to extract some "valuable" information from document scan. For example, document's number, incoming date, organizations, persons, etc.

Example document:

I'm trying to extract highlighted segment of the document. Original scan doesn't have that highlighting. And value can be handwritten or typewritten.

I tried U-Net and Mask RCNN for my dataset (~100 examples). Without any success.

Any ideas?


texts:
I was thinking of training a neural network that would be able to classify twitter users according to their followers. For example, I would like to know if a user is "gamer" or not by the people he follows. Have a dataset of people who are gamers or not and train them based on that.

The problem that I find is that the number of followers of each person will vary ... One person could follow thousands of users while another could only follow hundreds, so how can I determine my feature there?

Any suggestion? How could I do this?

This is in the way I am representing the data, so you can understand the problem of the features I was referring to

I did not know very well if asking this on StackOverflow, I have finally come here, it is the first time I post here, I hope it is the right place. Thank you.


texts: I have time series for which I used autocorrelation property and made autoregression model for prediction, with different time delays 0-30. I got diagram which is dependency MSE on time delay 0-30 http://imgur.com/a/7l2je and I don't know how to explain this discontinuity with the first dot for which delay is 0 and MSE is equal 0?


texts: Predicting evolution of an unknown mathematical function with machine learning?

texts: Is it possible to predict how a mathematical function evolves using ML? I am studying radioactive decay of polonium 210 and I have gathered data for its decay throughout a couple of weeks. I was wondering if I could use Machine Learning to train on the data available and predict how the Radioactive Activity would evolve after the timeframe where I gathered data. Can I predict (with ML) how the radioactive activity would evolve based on available experimental data?


texts: You can view a neural network as a tool that tries to learn "basis functions" (i.e. last hidden units) and then recombines then into a function approximation of the "true" function. I want to visualize this process during learning, and in order to do that, I would need to plot the behaviour of the hidden units at (say) every epoch during training and visualize how they change. How can one do this using Keras? Ideally, I do not want to have to fully rewrite the "fit" procedure but I am willing to do this if needed.


texts:
I'm hoping you have some research or experience with determining the completeness of a data set. I'm trying to use a twitter dataset I scraped myself and want to have an indication on the completeness.
Obviously, I will miss some data but I am wondering if there is a formula or method to calculate a probability on the completeness.


texts: Why do relative positional embeddings instead of absolute positional encoding improve the Transformer?

texts: I've been researching the Music Transformer, the paper for which introduced an efficient algorithm to compute Relative Positional Embeddings in a Transformer. I know that Relative Positional Embeddings slightly improve the performance of the Transformer on various tasks. But why? Apart from adding the consideration of distances between elements in a sequence to the Attention calculation, what exactly causes Relative Positional Embeddings to be more advantageous than Absolute Positional Encoding in a Transformer?