texts: unimportant features impact on model's performance

texts:
Using XGBoost and RandomForests, do unimportant features (according to the feature_importances_ attribute) hurt the model's performance?

Do I need to carefully select highly correlated and import features? Or do I throw everything in and hope that it can correctly add some information on the target variable?


texts: Can you explain the difference between SVC and LinearSVC in scikit-learn?

texts:
I've recently started learning to work with sklearn and have just come across this peculiar result.

I used the digits dataset available in sklearn to try different models and estimation methods.

When I tested a Support Vector Machine model on the data, I found out there are two different classes in sklearn for SVM classification: SVC and LinearSVC, where the former uses one-against-one approach and the other uses one-against-rest approach.

I didn't know what effect that could have on the results, so I tried both. I did a Monte Carlo-style estimation where I ran both models 500 times, each time splitting the sample randomly into 60% training and 40% test and calculating the error of the prediction on the test set.

The regular SVC estimator produced the following histogram of errors:

While the linear SVC estimator produced the following histogram:

What could account for such a stark difference? Why does the linear model have such higher accuracy most of the time?

And, relatedly, what could be causing the stark polarization in the results? Either an accuracy close to 1 or an accuracy close to 0, nothing in between.

For comparison, a decision tree classification produced a much more normally distributed error rate with an accuracy of around .85.


texts: How can I use mean normalization. Should I use it for numerical columns or categorical columns as well?

texts: Should we normalize the categorical columns in our dataset? Or just the numerical columns?


texts: Extracting table data from document image and export to excel

texts:
I need to extract data from document image which is in table format. We need to extract it into CSV/Excel format.
I was thinking to use opencv and pytessract to detect border line but there is some document which no table border. So now I can not get general solution for it. Please anyone have a general solution to achieve this?


texts: Why is my Count (distinct) not working the way I would expect it to be? PowerBI

texts:
My total is set as Count (distinct). However, the total itself doesn't make any sense, as you can see in my attachment. Any ideas what I'm doing wrong? By not making any sense, I mean that the numbers above are not being added.

Any suggestions would be greatly appreciated.


texts: Is scraping job posts and gathering skills tags illegal?

texts: I want to scrape a freelancer site (freelancer.com) and gather their public projects skill tags. Is this illegal to do? And is publishing it illegal?


texts: Strategy for improving performance of 3D convolutional GAN

texts:
Others working with neural nets and GAN's might find this question interesting.
Background:
I've been working with data from Berkeleys PEER Ground Motion Database to generate new novel seismic traces. (Real traces are rendered above.) Coming from a background in engineering my first attempt involved decomposing the traces into their {X,Y,Z} components, however the results were less than satisfying. The repeated mode collapse output can be seen below. There might be ways to fix this with more time and resources but I thought I would try another approach.

Forward:
I still have a bit of time to work on the data, and was looking to solicit methods for turning this {X,Y,Z} point data into something more digestible by a 3D convolutional network. I have given a small sample of one of the traces of 3 features each over time. Each trace is scaled {-1,1) across all axis and interpolated to 4000 steps. (Nearly all of the 788 traces in the training data are over 4000 steps and were down sampled.)
{{0.095746,-0.301555,-0.407207},{0.0955857,-0.301693,-0.407324},{0.0953887,-0.301861,-0.407461},{0.095148,-0.302067,-0.407609},{0.0948748,-0.302296,-0.407768},{0.0945636,-0.302547,-0.407985},{0.0942212,-0.3028,-0.40827},{0.0938457,-0.30301,-0.408664},{0.0934454,-0.303154,-0.409237},{0.0929839,-0.303259,-0.410011},{0.0924333,-0.303451,-0.410692},{0.0917118,-0.3039,-0.41102},{0.0907978,-0.304671,-0.411107},{0.0897068,-0.306049,-0.410927},{0.0886644,-0.308036,-0.410422},{0.0878034,-0.310481,-0.409685},{0.087091,-0.313055,-0.409298},{0.0863677,-0.315387,-0.409476},{0.0854884,-0.317162,-0.409922},{0.0845426,-0.31811,-0.410777}}

My knowledge of the subject matter suggests I need to transform this data into some type of array with places a 1 if there is a trace point there and 0 if void. Is that correct?
Before jumping in and creating another big branch of code, I wanted to confirm some of my assumptions moving forward. It seems like a rather computationally expensive method to divide these real valued 2x2x2 volumes and essentially count how many points are in each volume to recreate the geometry. Are there any precedence for this that I can follow? Is there a way to do this directly with a 3D convolution?


texts:
Given a few specific words, which techniques of Natural Language Processing can I use to achieve creating a meaningful sentence from those words?

eg.

Words:
jackets, highest sale, sweaters, lowest sale

Sentence:
Jackets exhibited the highest sales while sweaters sold the least.

If my question is too broad, let me know so I can ask more specific questions.


texts:
I have a dataset with 5000 observations with class 0 and 300 observations with class 1. I would like to run some statistical analysis, for example on the average length of strings, the number of words, ...
I was calculating the average. Do you think it makes sense to calculate and compare these values with a similar difference In classes?


texts: Group-by correlation vs All observations correlation

texts:
Let's suppose that I have a dataset with datapoints about footballers.

The data are about footballers' performance and information (e.g goals, assists, injuries, age, weight etc) on a monthly basis for the last 2 years.

My goal is to see how the performance and status of a footballer at a particular month is related to his performance and status of the next month.

At a first stage, I just want to run some correlation to detect some of these relationships.

In this case, does it make sense to run a separate correlation at each footballer's data of the last 2 years and then average the correlation results across players or directly run a correlation across all footballer's data at any month?


texts: How to transform stock data for LSTM-based neural network

texts:
I am trying to classify stock returns using an LSTM-based neural network.

I would like to use closing price and volume as features (see below), but am unsure of whether I need to transform these (e.g., by differencing) before feeding them into the network?

If anybody has done this sort of thing before and could give me some advice, or could refer me to any papers, I'd be very grateful.

Thanks,

Jack


texts: Higher frequency of time series benefits

texts: We are setting up an experiment for a model that is able to predict the evolution of a time series in different horizons. One of the parameters to decide is the granularity of frequency of our samples (every 1/5/10 mins). My question is: does higher frequency sampling provide better result as a general rule?


texts: data processing, correlation calculation

texts:
I have product purchase count data which looks likes this:

user item1 item2
   a     2     4
   b     1     3
   c     5     6
   ...   ...   ...

These data are imported into python using numpy.genfromtxt. Now I want to process it to get the correlation between item1 purchase amount and item2 purchase amount -- basically for each value x of item1 I want to find all the users who bought item1 in x quantity then average the item2 over the same users. What is the best way to do this? I can do this by using for loops but I thought there might be something more efficient than that. Thanks!


texts: Advice on transitioning from Andrew Ng's Stanford Coursera Machine Learning (in MATLAB) into Python?

texts:
I'm currently finishing up Andrew Ng's Coursera course, taught in MATLAB/Octave, but I'm looking to code in Python. The course is an introduction into how some algorithms work from scratch. These include: Linear/Logistic Regression, gradient descent, Support Vector Machines, Neural Networks, K-means clustering, anomaly detection, Gaussian distribution. (Decision trees/random forests aren't covered so I have to learn those).

Any advice on implementing this theoretical knowledge into Python? Is it necessary/recommended to recode the MATLAB algorithms from scratch in Python, or instead should I jump straight into learning Python libraries (sk-learn for now with pandas, matplotlib, etc.) and start projects?
Any project recommendations for my level? Certain concepts I should focus on now? Where to next? I know Python relatively well and have completed tutorials on numPy, Pandas, matplotlib, and sk-learn (very basic).

Next I plan on studying Harvard's CS109 data science course and taking Andrew Ng's Deep Learning Specialization on Coursera.

My ideal goal (a long journey) is to become a machine learning engineer and get into AI and deep learning.

Any advice is welcome! Thanks a lot.


texts: Does Call Detail Record(CDR) data record different cell-site/tower on incoming and outgoing call/text?

texts:
I am working on CDR data to find important locations of user. The data I have does not have coordinates so I will be using cell-site location as user's location. But after doing some working on data I have observed something weird in data i.e CDR of a user is generated from two different locations/towers on the exact same time, those towers are spatially very far, around 8.5km, and there are also other towers in between. You can see the snaps below.

Is it possible that this CDR records, the tower_id from which the user was called i.e it records only the id of only outgoing tower??


texts:
I am really interested in the geometric interpretation of perceptron outputs, mainly as a way to better understand what the network is really doing, but I can't seem to find much information on this topic.

I know a perceptron with no hidden layers can be seen as defining a hyperplane in between two classes, which works fine if they are linearly separable.

However, I find more difficult to understand what exactly does a multi layer perceptron do, and how it can classify an input that is initially not linearly separable.

I found this page http://www.borgelt.net/doc/mlpd/mlpd.html that says the hidden layers can be interpreted as making a coordinate transformation on the input. I would like to read more about this, but I didn't find much online, so I would really appreciate if you could elaborate on this or point me to some paper or book that covers this.

I am also puzzled by the visualisation that appears here http://tiny.cc/2l7o7y , a 3 neuron hidden layer manages to separate the input, which makes perfect sense since separating that input in a 3 dimensional space is trivial.
However, adding another 2 neuron layer after that creates in them shapes that are obviously non-linear. My guess is that it represents the separation created in the original 2-dimensional space. Is that right? If not, what are we seeing here, exactly?


texts: Fundamental difference between Tokenizer and TextVectorization in Keras

texts: Is there any fundamental difference between Tokenizer at the word level and the TextVectorization in Keras?


texts: Can we use machine learning to generate a text output based on the input strings

texts:
Problem : Generate a text output based on input strings which will be combined using a number of rules.

Example :

    Feature1               Feature2                    O/P

Rule 1  Enum_Domain           Priority          /Enum_Domain/Priority

Rule 2 Enum_Domain.EnumData   Name              /Enum_Domain/EnumData/Name

Rule 1  Trunkgroup            Gateway            /Trunkgroup/Gateway

Rule 2  GatewayGrp.Gateway    IP                /GatewayGrp/Gateway/IP

This is a simple programming problem, but is there any machine learning algorithm that can learn these rules and generate the output based on the two inputs.


texts: Which one is better for handling spatio-temporal data: 3D CNN vs 2D Recurrent CNN?

texts:
Please forgive my ignorance and lack of experience: I am asking this question seeking answer from the experts/experienced persons in the field.

I have a training dataset where each sample is a 3D cube (x,y,t) with grid points of (Nx = 256, Ny = 256, Nt = 200). The input consists of spatio-temporal evolution of a specific wave propagation. However, the input is also corrupted by other unwanted wave modes and noise components. I need to build a neural network (possibly convolutional auto-encoder or something similar) that can provide 3D output cube where the target wave propagation is mapped in space and time, effectively denoised of the unwanted wave modes and noises.

My question is: should I consider a 3D convolutional neural network for this task? Or, since each (x,y) frames are stored in a collection of a time series from t(1) to t(n) in the 3D cube, should I try a Recurrent 2D convolutional neural network for this job? Is there any advantage of trying one over the other?

Eventually I have to do this for 4D cube (x,y,z,t) with grid points of (Nx = 256, Ny = 256, Nz = 100, Nt = 200). In that case, would a Recurrent 3D convolutional neural network be better than 4D convolutional neural network?

If you have any literature in mind regarding this, please share. Thanks in advance.


texts: Approximation of long sequence of layers by one layer

texts:
Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of the model.
Layers from i up to j, effectively perform some nonlinear transformation:
$$
\mathbb{R}^{d_i} \rightarrow \mathbb{R}^{d_j}
$$
Where $d_i$ and $d_j$ is the dimensionality of the input at $i_{th}$ and $j_{th}$ layers. Has there been any reseach on approximation of this nonlinear transform by some linear one - search of convolution layers with the activation function, which approximate the action of multiple layers as close as possible.
The loss function for this approximation can be simply an MSE error, where the inputs are the activations of $\mathbb{R}^{d_i} $ and outputs - activations $\mathbb{R}^{d_j}$. I see the possible problem, that mapping from one high dimensional space to another requires a lot od training data for reliable estimate, but, nevertheless I wonder, whether such an approach has been pursued in the literature?


texts:
So I have finally gotten a hold of ggplot2 and ggvis packages and tried out shiny too to have interactive graphics and visualizations. but i have been using Power BI before and came to R as a an good alternative for creating effective visualizations. But there is one thing that has been missing in R which is to be able to filter out all visualizations on a page on the basis of a mouse click in one of the visualizations on the page. I understand how we can create slicers and other ways to filter graphics using shiny or better with ggvis.

So can anyone suggest a package or something i have missed in the above packages to help me achieve this.


texts: How is BCELoss counted in PyTorch? [different result comparing to mathematical implementation]

texts:
I am trying to understand how Binary Cross Entropy is counted in PyTorch.
I've tried the same code from the PyTorch documentation here, but I get a different result comparing to mathematical implementation of this function.
Code (I made little changes comparing to documentation):
import torch
import torch.nn as nn
m = nn.Sigmoid()
loss = nn.BCELoss()
input = torch.randn(1, requires_grad=True)
print(input)
print(m(input))
target = torch.empty(1).random_(2)
print(target)
output = loss(m(input), target)
print(output)
output.backward()
print(output)

Output for the code above is:

The mathematical formula for BCELoss is:
output = -[target*log(our_value) + (1-target)*log(1-our_value)]

From the mathematical formula above I should get 'output'=0.3215 for 'our_value'=0.4770 and 'target'=1. But PyTorch shows that the 'output'=0.7403.
I've also found a C code here of this function but the formula is almost the same (only very small epsila should be added and it gives no difference in output). So, it's still unclear for me.
How does it come to? Any hints are welcome.


texts: What more does TensorFlow offer to keras?

texts:
I'm aware that keras serves as a high-level interface to TensorFlow.

But it seems to me that keras can do many functionalities on its own (data input, model creation, training, evaluation).

Furthermore, some of TensorFlow's functionality can be ported directly to keras (e.g. it is possible to use a tf metric or loss function in keras).

My question is, what does TensorFlow offer that can't be reproduced in keras?


texts: Why normalize function has a different result on a matrix vs single value?

texts:
I have a matrix like:

B=[ 1.5035; 1.5728; 1.6485;  1.5369;  1.5467;  1.572; 1.5374;  1.787;  1.5825; 1.6905];

Using normalize function like  normalize(B,'range') has this result:

ans =  0
  0.24444
  0.51146
  0.11781
  0.15238
  0.24162
  0.11958
        1
  0.27866
  0.65961

But when I use it for a single value like:

normalize(B(2,:),'range')

the result is 0 , but the result for row number 2 in ans is  0.24444.
Why its different and how can I fix it?


texts: Why are there over 600 open pull-requests in Scikit-learn github repo?

texts:
I considered submitting a function that I deem missing to scikit-learn repo, but as of June 28th 2019 there are over 600 pending Pull Requests: https://github.com/scikit-learn/scikit-learn/pulls.

Can you explain the situation with maintainers? I mean, reviewing a pull-request is not that big of a deal compared to writing commits and also this normally has higher priority than writing code, so why I see new commits in the project, but PRs are still not reviewed?

By the way, I also found a scikit-learn-contrib project that seems to be contain many additional functions: https://github.com/scikit-learn-contrib.


texts: Finding relationships in datasets and predicting data

texts:
I'm a .NET developer trying to learn more about AI/Machine Learning.
I have a dataset with few variables and known results. I'm looking for a language/framework/tool/approach to analyze the data and identify which variable(s) contributed the most towards the results and eventually predict what the result will be given only the variables.
I live in a Microsoft eco-system. Could someone suggest a starting point to research? Does this sound like a common problem?


texts:
I have a dataset with labels and usernames:
Labels   Usernames
1         Londonderry
1         Londoncalling
1          Steveonder43
0         Maryclare_re
1         Patent107391
0         Anonymous
1         _24londonqr
...

It seems that the usernames containing the word London are very frequent in having assigned label=1.
Do you have any idea on how I could proof it?


texts:
I am trying to create a kNN search from scratch for a project. I think I have the concept of it and how it works, but I can't understand what exactly is a class.

I have a matrix of $X*Y$ where $X$ is the number of elements and $Y$ their vector. So if the values are completely random, is every element/row a different class, or do I specify the classes some other way?


texts: How to clone Python working environment on another machine?

texts:
I developed a machine learning model with Python (Anaconda + Flask) on my workstation and all goes well. Later, I tried to ship this program onto another machine where of course I tried to set up the same environment, but the program fails to run. I copied the program to other machines where it also runs smoothly.

I cannot figure out what the problem is in the failed case (both the program code and the error message are copious so I am not able to present them here) but I'm almost certain that it is something with the different versions of the dependencies.

So, my question is that given an environment where a certain program runs well, how can I clone it to another where it should run well also? Of course, without the cloning of the full system ;)


texts: How do I deal with the fact that I have images which are not consistent with the class they belong in an image classification problem with CNN?

texts:
I am really new to Neural Networks and to Machine Learning in general, and I have been given a dataset composed by images for performing multi-class image classification with a CNN.
The images were already divided into classes, and looking at the images I have noticed that some of them are complitely different from the class they belong, for example If I have a class Fruits, with images of fruits, in the folder of this class I have some pictures of cars, people,..., which of course are not fruits and neither belong to any other class in the classification problem.
The problem is that this creates some problems when I train my CNN, and this results in a low accuracy, infact I cannot go above 0.5.
How do I deal with the fact that I have images which are not consistent with the class they belong?


texts: the error occurred while selecting feature using recursive feature elimination in sklearn

texts:
I tried to rank the feature using recursive feature elimination in sklearn. However, I got this error when using RFE. here are the error and code information.
  from sklearn import svm
  x_vals = data['all_data'][:,0:320]

 y_vals_new = np.array([0 if each=='Neg'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,320]])

 clf = svm.SVC(decision_function_shape='ovo',kernel='rbf')

rfe = RFE(clf, 200)
rfe = rfe.fit(x_vals,y_vals_new)

print(rfe.support_)
print(rfe.ranking_)

clf.fit(DEAP_x_train, DEAP_y_train)

print("###Valence###")
print("when the kernel function is rbf")
print('The mean square error %10.9f ' % np.mean((clf.predict(DEAP_x_test)-DEAP_y_test)**2))  # The mean square error

print('the mean accuracy on the given test data and labels %10.9f'% clf.score(DEAP_x_test, DEAP_y_test))

Did anyone know the reason of this error？ some opinion showed that RFE（recursive feature elimination) only works with SVC when the kernel is chosen to be linear. Is that correct？ Thanks a lot！


texts: Predict numerical value when each variable is a target and a predictor

texts:
Imagine I have created a tabular dataset when each row is a company and each variable is the company's rating between 0 through 100 given by a rating agency.  Each of the ratings agencies can decide who they cover, so some might not cover a company leaving it blank.  Please look at example below:
C = Company
R = Rating agency

   | R1 | R2 | R3 |
---------------
C1 | 80 | 51 |
C2 |    | 35 | 23
C3 | 60 |    | 45

I want to know what the score would have been had the rating agency covered that company using the relationship between the rating agency. In data science terms, How can I predict the missing scores for each company when the target variable is also the predictor variable. how should I approach this?

UPDATE

Seems like this is a standard Recommender systems question, however, I have added an extra question about using a system of models, one model for each of the rating agencies.  Could that possibly work better?  Look in comment section


texts: GridSearchCV with MLPRegressor with Scikit learn

texts:
I'm trying to apply automatic fine tuning to a MLPRegressor with Scikit learn. After reading around, I decided to use GridSearchCV to choose the most suitable hyperparameters. Before that, I've applied a MinMaxScaler preprocessing. The dataset is a list of 105 integers (monthly Champagne sales).

The problem is that for some reason the GridSearchCV isn't operating (at least correctly, I think). When I print the parameters used by the model, appear some values out of the range defined in param_list.

Furthermore, i know that the dataset is too small for a MLP, the idea is program the model now and use it later in a larger dataset. Although, the final dataset is not very large, so I'll be very thankful of hear any idea to improve the accuracy of the model in a small datasets!

Thanks!

Code:

from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import TimeSeriesSplit
from sklearn.model_selection import GridSearchCV
from matplotlib import pyplot
from sklearn.neural_network import MLPRegressor
from sklearn.metrics import mean_squared_error
import pandas as pd

dataset = pd.read_csv('champagne.csv', header=None)

scaler = MinMaxScaler()
scaled_dataset = scaler.fit_transform(dataset)

mlpr = MLPRegressor(max_iter=7000)

param_list = {"hidden_layer_sizes": [1,50], "activation": ["identity", "logistic", "tanh", "relu"], "solver": ["lbfgs", "sgd", "adam"], "alpha": [0.00005,0.0005]}
gridCV = GridSearchCV(estimator=mlpr, param_grid=param_list)

splits = TimeSeriesSplit(n_splits=3)

pyplot.figure(1)
index = 1

for train_index, test_index in splits.split(scaled_dataset):

    training_set = scaled_dataset[train_index]
    testing_set = scaled_dataset[test_index]

    train_index_array = train_index.reshape(-1,1)
    test_index_array = test_index.reshape(-1,1)

    gridCV.fit(train_index_array, training_set)
    predicted = gridCV.predict(test_index_array)
    parameters = mlpr.get_params()

    test_mse = mean_squared_error(testing_set, predicted)

    pyplot.subplot(310 + index)
    pyplot.plot(predicted)
    pyplot.plot([None for i in training_set] + [x for x in testing_set])
    index += 1

    train_index.flatten()
    test_index.flatten()


texts: Identifying the role of an entity (e.g. a company) in a sentence

texts:
I have a general question about how to distinguish between the following two types of sentences:
Type 1:
Goldman Sachs also pointed to a rise in predicted losses in the healthcare sector in 2020 due to rising external debt.
Sovereign default risks are on course to rise further in 2021, say Goldman Sachs analysts.
Type 2:
Goldman has financed or invested in companies in China for more than 25 years.
Goldman Sachs will pay the U.S. government about $1.3 billion and admit wrongdoing in a bribery scandal.
In Type 1, Goldman Sachs is only reporting on the activities in the market, whereas in Type 2 Goldman is the main agent and the sentence actually conveys/reports something about what is happening with Goldman as a company (excuse me for my lack of the linguistic vocabulary here).
Is there a family of NLP models that is well suited to tackle this task, i.e. distinguish between these two types of text? I started looking a bit into the Semantic Role Labeling, could this be the right approach to take here?


texts: Loading a Model with weights and optimizers without creating an instance in PyTorch

texts:
I recently downloaded Camembert Model to fine-tune it for my purpose.
Upon unzipping the file the contents are:

Upon loading the model.pt file using pytorch:
import torch
model = torch.load(model_saved_at)

I saw that model was in OrderedDict format containing the following keys:
args
model
optimizer_history
extra_state
last_optimizer_state

As the name suggests most of them are OrderedKeys themselves with the exception of args which belongs to a class argsparse.Namespace. Using vars() we can see args only contains some hyperparameters and values which are to be passed from the command-line.
model["model"] contains the weights which I want to load and use as my base model.
A small part of it is as shown below:
for ans in model["model"].keys():
    try:
        print(ans, "\t" ,model["model"][ans].size())
    except:
        print(ans, type(ans))

decoder.sentence_encoder.embed_tokens.weight     torch.Size([32005, 768])
decoder.sentence_encoder.embed_positions.weight      torch.Size([514, 768])
decoder.sentence_encoder.layers.0.self_attn.in_proj_weight   torch.Size([2304, 768])
decoder.sentence_encoder.layers.0.self_attn.in_proj_bias     torch.Size([2304])
decoder.sentence_encoder.layers.0.self_attn.out_proj.weight      torch.Size([768, 768])
decoder.sentence_encoder.layers.0.self_attn.out_proj.bias    torch.Size([768])
decoder.sentence_encoder.layers.0.self_attn_layer_norm.weight    torch.Size([768])
decoder.sentence_encoder.layers.0.self_attn_layer_norm.bias      torch.Size([768])
decoder.sentence_encoder.layers.0.fc1.weight     torch.Size([3072, 768])
decoder.sentence_encoder.layers.0.fc1.bias   torch.Size([3072])
decoder.sentence_encoder.layers.0.fc2.weight     torch.Size([768, 3072])
decoder.sentence_encoder.layers.0.fc2.bias   torch.Size([768])

However, I cannot use load_state_dict() since I have no instance of this class. How am I suppose to load the weights and optimization parameters without creating an instance? I thought of using sentence.bpe.model but they are for tokenization purposes.


texts: proper activation function at output and loss function to optimize for OCR?

texts:
I am trying to make a CNN model on IAM handwritten words data(which has images of words handwritten by multiple people and targets are text in the images). So, I can encode words to numbers(A=0, B=1 and so on for capital, small and punctuation). But, I couldn't quite decide on what loss to use for this problem, in tensforflow.keras.
I have found out about something called CTC loss on some repositories on GitHub. But, ctc loss was used with softmax as activation in final dense layer. And since softmax outputs probabilities for each class that sum up to 1; So, I presume softmax can't output probabilties for two "o"s at the same time.
So, what are the proper activation and loss functions that I can use for a problem like this one?


texts: How to interpret my logistic regression result with statsmodels

texts:
so I'am doing a logistic regression with statsmodels and sklearn.
My result confuses me a bit. I used a feature selection algorithm in my previous step, which tells me to only use feature1 for my regression.
The results are the following:

So the model predicts everything with a 1 and my P-value is < 0.05 which means its a pretty good indicator to me. But the accuracy score is < 0.6 what means it doesn't say anything basically.
Can you give me a hint how to interpret this?
It's my first data science project with difficult data.
My code:
X = df_n_4["feat1"]
y = df_n_4['Survival']

# use train/test split with different random_state values
# we can change the random_state values that changes the accuracy scores
# the scores change a lot, this is why testing scores is a high-variance estimate
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)
print(len(y_train)," Testdaten")

# check classification scores of logistic regression
logit_model = sm.Logit(y_train, X_train).fit()
y_pred = logit_model.predict(X_test)
print('Train/Test split results:')
plt.title('Accuracy Score:{}, Variablen: feat1'.format(round((accuracy_score(y_test, y_pred.round())),3)))
cf_matrix = confusion_matrix(y_test, y_pred.round())
sns.heatmap(cf_matrix, annot=True)
plt.ylabel('Actual Szenario');
plt.xlabel('Predicted Szenario');
plt.show()
print(logit_model.summary2())


texts: How to derive association from a regression model?

texts:
I understand how to make predictions with a trained neural network model that uses loss=binary_crossentropy and a 1-node activation=sigmoid output layer to make binary classifications.

But how can I determine the strength of association between a feature and a label? I'm trying to make a neural network that competes with generalized linear models that show p values for each feature.

The predictions are not the important part. I need to provide insight about the features so that we can learn about the biology of a disease (DNA variant is feature and disease is label). I know that there is such a thing as feature importance, but isn't that just a rank ordered list of features?

def create_baseline():

    model = Sequential()
    model.add(Dense(60, input_dim=60, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model


texts: Negative impact of "important" features on model performance

texts:
I have a random forest regressor with a set of base features, fit & optimised with sklearn random search algorithm. When I add a set of additional features and retrain (again with random search optimisation) the model performance becomes worse.

Afterwards, I use multiple algorithms to determine the feature importance of all features - recursive feature elimination (RFE), built-in RF feature importance, permutation importance. Some of the newly added features actually show a high importance (with all algorithms).

So how can it be that features, which show high importance actually reduce the model performance (significantly)? I could not yet find any satisfying answer to this problem (or misunderstanding of mine).


texts: Convert TFIDF Values to Vector Space Model

texts:
I'm working on a project using tf-idf values and cosine similarity for clustering. As my database (elasticsearch) provides tfidf values out of the box (term_freq & doc_freq), my code involves calculating the tfidf vectors manually using this data, and then performing cosine similarity.
However, I'm unable to use the sklearn cosine similarity on these values as they are 1-dimensional rather than 2-dimensional -- this indicates to me that I need to input the values into a matrix or create a vector space model in order to do so? How would I achieve this?
Here's some of my code for illustration :)
# cycle through the term vectors provided by elasticsearch
# and append them to their corresponding term in dataframe

for x in v1:
    tf = v1[x]['doc_freq'] / len1
    idf = math.log(num_docs / v1[x]['term_freq'])
    df.at[0, x] = tf*idf

for x in v2:
    tf = v2[x]['doc_freq'] / len2
    idf = math.log(num_docs / v2[x]['term_freq'])
    df.at[1, x] = tf*idf

df = df.fillna(0.00)

# -----------------------------------------

# create numpy matrix using dataframe

# approach 1
matrix = np.zeros((2, number_unique_terms))
    # populate this with dataframe row 0 and row 1

# approach 2
matrix = np.matrix(df.iloc[0], df.iloc[1])

Both of the approaches outlined above still give me the 1D array error when attempting to perform cosine similarity. Am I making a simple mistake when it comes to numpy? Or am I attempting to create a vector space model (or matrix) in the wrong way?
Thanks in advance! :)


texts:
I am fitting ARIMA model (from statsmodels) on 20 000 elements dataset on a 24 CPU 200+GB RAM cloud server for over 24 hours now. It loads all the CPU's. But It takes so long... Is it how it works or is here something wrong?
p,d,q are just numbers
from statsmodels.tsa.arima_model import ARIMA

for t in range(len(test)):
    model = ARIMA(history, order=(p, d, q))
    model_fit = model.fit(disp=1)
    output = model_fit.forecast()
    yhat = output[0]
    predictions.append(yhat)obs = test[t]
    history.append(obs)


texts: How do I pick "K" for precision at K and recall at K?

texts:
I understand precision at k and recall at k.  It is a more useful metric for evaluating the success of a binary classifier when the positive class is overwhelmingly out-weighed by the negative class.

I'm wondering how to choose an appropriate "k" value.  According to resources like this the recall at "k" is bounded by the number of positive examples, so it is not a useful metric to use when evaluating the success of severely imbalanced classes.

It also seem to me like this:  Precision at K is also limited by the number of positive examples at K.  Saw we have 100 examples total, and only 3 are positive examples.  Say we rank these

Scenario 1: we choose k=10.  Then, the precision at K can at most be 3/10 = 0.3. And, the recall at k will be 0.03 because there are 3 in the entire dataset of 100.

Scenario 2: we choose k=3.  Then, the precision at K would be 3/3 = 1.0 !!!And, the recall at k will STILL be 3/100 = 0.03.

Even though our binary classifier is performing perfectly, it's perfect performance only is reflected when we choose k=3?

So, my question how do I choose K correctly?


texts:
I'm starting at Data Science and, to get something going, I just ran the code from Siraj Raval's Intro to Data Science video. He implements a simple Decision Tree Classifier but I couldn't help but notice that, given the same training set, the classifier doesn't always yield the same prediction (nor the same fit apparently); which I happen to find terribly weird, since, from what I've learned, a Decision Tree is supposed to be deterministic.

The only thing I can think of that could be causing the randomness would be that the branches are being chosen at random at some point because 2 options might be identically valued. I would say this could be corrected with a little bit more training data, but even if I add 5 more people, nothing changes. Does anybody have an explanation for what's going on?

Following is the code (in Python) from the video in a for loop to count how many predictions for male and female the Decision Tree has yielded.

from sklearn import tree
from sklearn.svm import SVC

n_male_pred_tree = 0
n_female_pred_tree = 0

n_male_pred_svm = 0
n_female_pred_svm = 0

for i in range (1,1000):
    # This loop tests the consistency of the CLF
    # The Decision Tree is not very consistent (It's 50-50)
    X = [[181,80,44],[177,70,43],[160,60,38],[154,54,37],
         [166,65,40],[190,90,47],[175,64,39],[177,70,40],
         [159,55,37],[171,75,42],[181,85,43]]

    Y = ['male','female','female','female',
         'male','male','male','female',
         'male','female','male']

    tree_clf = tree.DecisionTreeClassifier()
    svm_clf = SVC()

    tree_clf.fit(X,Y)
    svm_clf.fit(X,Y)

    tree_prediction = tree_clf.predict([[190,70,43]])
    svm_prediction = svm_clf.predict([[190,70,43]])

    if tree_prediction == 'male':
        n_male_pred_tree += 1
    else:
        n_female_pred_tree += 1

    if svm_prediction == 'male':
        n_male_pred_svm += 1
    else:
        n_female_pred_svm += 1

print(f"MALE pred Tree: {n_male_pred_tree}")
print(f"FEMALE pred for Tree: {n_female_pred_tree}")
print("\n")
print(f"MALE pred for SVM: {n_male_pred_svm}")
print(f"FEMALE pred for SVM: {n_female_pred_svm}")


texts:
I have 2 independent data sets (1. 300 rows and 2.3000 rows) with 6 months trades observations for 50 traders. In both datasets I have:
trader id,
stock title,
buy/sell volume,
date of trade,
sector of stock

My goal is to detect possible outliers (suspicious trades) in this two datasets.

What algorithm you would recommend for this 2 tasks and why?
What can we use for 1 task when we have average only 5-6 trades per trader?


texts: How to select multiple columns in a RDD with Spark (pySpark)?

texts:
Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week.

So for i.e. [8,7,6,7,8,8,5]

How can I manipulate the RDD so it only has Monday, Wednesday, Friday values? There are no column names by the way. But the PySpark platform seems to have _co1,_co2,...,_coN as columns.


texts: I recently  learnt about the anamoly detection using  autoencoders(specifically denoisinng autoencoders).To train the autoencoders do we need  a data  having some pattern? or is there some way to train in abnormal data ?Also how we decide that the data is suitable for training autoencoder model.


texts: What is cross validation good for, exactly?

texts:
I keep seeing that cross validation is a good way to reduce overfitting, but, in my case, I don't see how it helps much. Let me explain:
I'm interested in running a Multinomial Logit for prediction, with 3 classes on the dependent variable.
I saw that cross validation is good to tune hyperparameters, which makes sense, but in my case I don't have any hyperparameters to test. What good is it for then? Just checking that the accuracy on "different" datasets, with the same features, doesn't vary much?
Let's say after a Stratified k-folds approach I get a mean accuracy and precision on the test set of 90%. Is this enough to say my model, with those features I selected, is not overfitted?


texts: How should business rules be used to add to small, existing training dataset? (Semi Supervised Learning)

texts:
Problem Statement:
Take 1200 labeled users and project them to the rest of our userbase (3.6M)
What I've done:

Leveraged business rules to add to existing dataset. After doing so, I'm left with the following dataset by class.
 2    51119

 1    33695

 0    15538

Instead of creating a multi-class model, I've decided to go with three separate binary classification models.

After doing so, my model is doing a very poor job of projecting to the rest of the userbase.
For Class 0: 28 have been classified as positive. (Expected around 15%)
For Class 1: <1% have been classified as positive. (Expected around 15%)
For Class 2: 16% have been classified as positive (Expected around 40%)

I'm wondering if my business rules may be too strict? Also, should be I be worried if the classes have too much separation?


texts: Professionals appear to interpret sample correlation (e.g. Karl Pearson) as if it represents linear correlation. Is it the correct interpretation?

texts: I am stressed following  the wrong interpretation. What is the correct way of understanding a correlation coefficient.


texts: Is it always better to use the whole dataset to train the final model?

texts:
A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, e.g. a product.

  My question is: Is it always for the best to do so? What if the
  performance actually deteriorates?

For example, let us assume a case where the model scores around 65% in classifying the testing subset. This could mean that either the model is trained insufficiently OR that the testing subset consists of outliers. In the latter case, training the final model with them would decrease its performance and you find out only after deploying it.

Re-phrasing my initial question:

  If you had a one-time demonstration of a model, such as deploying it
  on embedded electronics on-board an expensive rocket experiment, would you trust a
  model that has been re-trained with the test subset in the final step without being
  re-tested on its new performance?


texts: Batch processing with variable length sequences

texts:
I have a lot of time series with different lengths. I would like to know what are the best practices to fit them to a Bidirectional LSTM model. The problem is a Binary Classification of Sequence to Sequence. So for every time step, I want to predict the binary class.

Currently, I create a tensor for each data frame with the shape of (1, None, #Features). Then I fit every tensor separately to the model.

Is it better to combine every data frame to a tensor of the shape (#Time Series, None, #Features) and fit them all at once? Does this make even a difference?

Or could it be better to go with the sliding window approach and split one time-series into smaller windows?

I can't specify a max-length of the time-series so I think I cannot use pad-sequence from Keras.


texts: what should be the order of class names in sklearn tree export function (Beginner question on python sklearn)

texts:
I am trying a simple example with sklearn decision tree. I am giving "number,is_power2,is_even" as features and the class is "is_even" (of course this is stupid)

Here is the code

from sklearn import tree

features =[[1,0,0] , [2,1,1] , [3,0,0] , [4,1,1] , [5,0,0] , [6,0,1] ,  [900,0,1] , [1001,0,0]] #val,pow2,even
labels =  ['o'     ,  'e'    , 'o'     ,  'e'    ,  'o'    ,  'e'    ,   'e'      ,  'o'] #is even

clf = tree.DecisionTreeClassifier()
clf = clf.fit(features,labels)

print (clf.predict([[203,0,0]]))

import pydot

import pydotplus
from IPython.display import Image
dot_data = tree.export_graphviz(clf, out_file=None,
                     feature_names=['number','pow2','even'],
                     class_names=['o','e'],
                     filled=True, rounded=True,
                     special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data)
# Image(graph.create_png())
graph.write_pdf("1.pdf")

The decision tree correctly identifies even and odd numbers and the predictions are working properly.

The decision tree is basically like this (in pdf)

     is_even<=0.5
        /\
       /  \
   label1  label2

The problem is this. The label1 is marked "o" and not "e". However if I put class_names in export function as

class_names=['e','o']

then, the result is correct. I thought the output should be independent of class_names order.

Am I doing something wrong, or does the class_names order matter. If the latter is true, what is the right order (for an arbitrary problem)


texts: Adding bias in deconvolution (transposed convolution) layer

texts:
My question is regarding the transposed convolution operation (also commonly called deconvolution or upconvolution). In TensorFlow, for instance, I refer to this layer.

My question is, how / when do we add the bias (intercept) term when applying this layer?

When working with 'regular' convolution, we do this:

conv_output = tf.nn.conv2d(input, kernel, strides, padding='VALID')
conv_output = tf.nn.bias_add(conv_output, bias)

How do we do this when applying the deconvolution layer? My confusion arises because my advisor told me to visualise upconvolution as a pseudo-inverse convolutional layer (inverse in the sense that convolution downsamples the input, while transposed convolution upsamples it. I know they are not mathematically inverse.)

According to him:

Regular convolution: conv = x.w + b

Transposed convolution: x = (conv - b).W (where w and W are not the same).

Is the above equation even right? Something about it makes me feel uneasy.

In this scenario, since we are "going backwards", should we do something like this:

deconv_output = tf.nn.bias_add(input, -1 * bias)
deconv_output = tf.nn.conv2d_transpose(deconv_output, kernel, strides, padding='VALID')

Or should we add the bias after applying the transpose convolution, as we do in 'regular' convolution?


texts:
I am working on a project to detect buildings from satellite imagery in Tanzania using convolutional neural nets. I use a pre-trained model which I further train on a selected area in Tanzania.
The model performs well (f1 score ≈ 0.85) when tested on an area near where it was trained but much worse (f1 score ≈ 0.3) when tested in a region further away. To tackle this issue, I thought I could use few-shot learning - I would first train the network on a large dataset in one region and when I want to use the model in a different region, I would fine-tune it with a few examples from that region.
However, when I read about few-shot learning, for example in this article or in this paper, they describe it as a method for learning new classes/categories from few examples. But I don't want the model to learn new classes. I want it to learn to recognize the same class, only on images looking slightly different than those used for training.
Question: Is few-shot learning suitable for what I want to achieve? If not, what could I use instead?


texts: Forecasting revenue/ROI based on different advertising spend scenarios

texts:
Example: Let’s say a company spends 1,000,000 USD a year on online advertising and the sales directly attributed (via tracking, etc) to that spend is 2,000,000 USD (100% ROI). How can that company generate forecasts around spending 500,000 USD or 1,500,000 USD on online advertising?
Of course, efficiency will change. ROI should increase in the 500,000 USD scenario and decrease in the 1,500,000 USD scenario. The company is able to use metrics, such as avg. cost-per-click, impression share (and several others), from their hundreds of advertising campaigns to predict the changes in ROI (the metrics correlate with ROI).
Initial high-level idea:

Prep the historical data, perform feature engineering, use a model
such as Facebook Prophet (or multiple models) to generate forecasts
of spend, revenue based on historical data. Validate against test
data.
Create regression models: One including advertising spend,
avg. cost-per-click, impression share, etc. With revenue as the
dependent variable predicted off of advertising spend. Another
including advertising spend, avg. cost-per-click, impression share,
etc. With advertising spend as the dependent variable predicted off
of ROI (let’s say the client wants to know the most they can spend
while achieving 125% ROI).
Take the forecasts from step 1 and plug
them into the appropriate regression model from step 2.

Thoughts? Understand I may be missing a much better process, so let me know if you recommend a different approach.
Thanks in advance.


texts: Structure the dataset for financial machine learning

texts:
I am trying to construct a dataset to apply MLP in forecasting financial returns. The main idea is that I want to predict future equity returns (1 month ahead, but the horizon can vary, just to give an idea) using fundamental data.

My dataset is made of n features for N companies. The main issue is that the features are not recorded at the same point in time, so that some of them would be a monthly time series and some other a daily time series. I am trying to understand the general way to approach this problem because it's my first ML application.

What should I do if my features are time series of data that differs in this sense? I have red a lot of publications, but no one gets in deeper about how the dataset is composed (before train/test splitting). Can someone recommend some useful source?

EDIT

Suppose I have a variable amount of companies over 10 years of monthly data, mainly because some of them doesn't exist anymore, but it is important for the problem of survivorship bias.

For each of these N companies I have n features that represents fundamentals of these companies. I would be able to regress the monthly return, so I am trying to solve a supervised learning problem.

What I cannot understand is the feasibility of the problem using a simple Multilayer perceptron instead a more complex structure like RNN or even LSTM. I wouldn't want to switch to one of these more complex architectures for some specific reason related to the project I am building.

Probably I am missing the reasoning behind the training of the network. How should I provide input data to the architectures in order to perform backpropagation and GD? The only thing I am sure is that I should shift the monthly return series of a time period to recast the problem as supervised.


texts: Can a recommendation system be used as a binary classifier?

texts:
I have a computer-generated music project, and I'd like to classify short passages of music as "good" or "bad" via machine learning.

I won't have a large training set. I'll start by generating 500 examples each of good and bad music, manually. These examples can be transposed and mirror-imaged to produce 12,000 examples of each good and bad.

I have a way of extracting features from the music in an intelligent way that mimics the way a perceptive listener would do. The trouble is that it involves thousands of features on a single passage. Each feature is a number from 0 to 1.

I'm pretty new to data science, and my understanding is that binary classifiers like decision trees wouldn't work well with that many features. But note that I'll have a lot of clumping and dependence of features. I just don't know how to predict that structure. Also, good examples will probably clump around a few categories, while bad examples will tend to be all over the place and "far" from good examples.

It occurred to me that maybe a recommendation system could work.

Taking the example of movie recommenders, the score of each of my features  would be like the rating a viewer gives a movie. Each example will rate thousands of features, i.e. a viewer rating thousands of movies.

Then we have one movie, called the "Good Movie." Each training example gives that either 5 stars or 0 stars depending on whether it's good or bad.

We then take a test example and predict whether it likes the "Good Movie."

Can this sort of thing work? Is there another approach to working with thousands of features?

EDIT: adding a comment on dependence or "clumping" of features: I intuitively think that "good" music will probably clump around several categories and within those clumps there will be strong correlation between features. Let's say inside clump #1, features A and B are correlated. But inside clump #2, A and B might be uncorrelated with each other, although each correlated to other features. I mention this in case it's relevant to doing feature reduction with PCA or other techniques.


texts: MAD vs RMSE vs MAE vs MSLE vs R²: When to use which?

texts:
In regression problems, you can use various different metrics to check how well your model is doing:

Mean Absolute Deviation (MAD): In $[0, \infty)$, the smaller the better
Root Mean Squared Error (RMSE): In $[0, \infty)$, the smaller the better
Median Absolute Error (MAE): In $[0, \infty)$, the smaller the better
Mean Squared Log Error (MSLE): In $[0, \infty)$, the smaller the better
R², coefficient of determination: In $(-\infty, 1]$ not necessarily the bigger the better

Are there any strong reasons not to use one or the other?


texts: Scikit-learn: train/test split to include have same representation of two different types of values in a column

texts:
I have a dataset of online purchase orders that contains two types of customers:

Customers who have an account and thus are known customers with unique customer number.
Customers who do not have an account and do their purchases as a guest; thus, these are unknown customers and are given a default number(out of a list of 50 pre-defined numbers). Meaning it is not a unique number and the only way to distinguish transactions here is by the unique order number.

To avoid distortion and make sure that my train/test dataset will have the same distribution when it comes to known and unknown customers, I want to select a Train/Test set as follows:
The part of the dataset that includes the known customers to be split 80/20.
The part of the dataset that includes the unknown customers to be split 80/20 ( the selection of the sub dataset here will depend on the rule that orders are not linked to a customer (= linked to a default customer number).
In Scikit-learn I need to set up the X(features) and Y(target) in order to do the train_test_split. My question is would it be logical and doable to split my dataframe into:

df1 which contains all known customers.
df2 which contains all unknown customers.

Then set up X, Y and consequently x_train, x_test, y_train , y_test for each of df1 and df2, then merge the result into one and continue to build the rest of the model?
Is this a good approach or is there another solution?


texts: NLP algorithms for categorizing a list of words with specific topics

texts:
Currently I am using LDA to apply topic modeling to a corpus. Since LDA is unsupervised, it returns a set of words for a given 'topic' but doesn't necessarily specify the topic itself. I was wondering if there are any suggestions for algorithms that take a list of words and sees what topics it can be categorized to?

For example [cat, dog, fish] can be categorized to animals or pets.

One output for my model:

['game', 'week', 'fantasy', 'sportsline', 'play', 'players', 'league', 'random', 'sunday', 'season', 'agent', 'elink', 'exercise', 'start', 'yards', 'free', 'injury', 'expected', 'practice', 'getbad', 'weekly', 'year', 'reports', 'starting', 'luck', 'nat', 'nfl', 'weeks', 'smith', 'fast']

Could be categorized to football or sports.

Any suggestions, specifically with Python models/packages would be much appreciated.


texts: The Bayes error rate is a theoretical bound that determines the lowest possible error rate for a classification problem, given some data. I was wondering whether an equivalent concept exists for the case of regression algorithms. My aim is to determine how far my regression algorithm's error is from that theoretical bound, as a way to assess how far am I from the best possible solution. Is there any way to obtain a bound of the lowest regression error for a given dataset?


texts: Using AutoEncoder for Greedy Layer-Wise Pretraining [Convolutional Neural Networks]

texts:
I am trying to implement greedy layer-wise pretraining for Convolutional Neural Network binary classifier using AutoEncoders. However, I am a little bit confused regarding the logic of implementation. If I understood correctly, I need to:

Build a CNN architecture and flag all layers as NOT TRAINABLE (i.e. model.layer[idx].trainable = False)
Iterate through each layer, make layer in the current iteration trainable, while others remain NOT trainable, and fit the model for couple of epochs using MSE loss.
Repeat step 2 until all layers, but the one meant for classification, are trained.
Fine-tune network for its original task.

Is this correct? If so, can anyone tell me how is greedy layer-wise pretraining different from training a complete AutoEncoder architecture and then using Encoder weights to initialize CNN architecture which is going to be fine-tuned latter.


texts: How does deep learning helps in detecting multiple objects in single image?

texts: Let's say there are two cars in an image. How can it detect these cars, given that it can detect single car in an image?


texts: How are the Google Cloud Natural Language API Sentiment Analysis outputs interpreted?

texts:
I am trying to get a better understanding of the outputs given by Google's sentiment analysis API. It takes in a sentence and gives out two values - magnitude and score. I am trying to interpret the magnitude value better. Magnitude is defined in the documentation as -

A non-negative number in the [0, +inf) range, which represents the absolute magnitude of sentiment regardless of score (positive or negative).

Initially, I thought it is a confidence score or weight but I am not sure how the value would change since it can be ANY number. Does anybody know how it is calculated or what it means apart from the definition provided in the documentation?


texts: Multivariate supervised LSTM for for regression only (not for prediction)

texts:
I am a beginner with machine learning. I want to get a multivariate regression model with LSTM. I followed this tutorial by Jason Brownlee:
https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/

In that tutorial the output feature (var1) is also part of the input features, just with another time lag (t and t-1). But what if I don't have the output feature (var1) when I use the model on new data?

I want to get a regression model based on 10 input features and one output feature. I want to train the model in a supervised manner (with 10 input and one output feature). The model should then deliver an output sequence (e.g. of 100 values) based on input features. I always have input features, so I don't want to predict without input features.

Do you understand what I mean? I want to get a LSTM model in a way of a multiple linear regression model where I can get with some input values an output value. How should I prepare the LSTM model?


texts: A Deep CNN model delivering better results with standardization, when compared with normalization

texts:
I developed a deep CNN model, based on the architecture discussed in this paper, to generate predictions for time series data. My training data is shown in the figure below:

In order to train the model, I tested two approaches. The first one where I normalize the data between 0 and 1, and in the second one I standardize the data based on the approach discussed here.

I noticed that in my case, standardization performs far far better than normalization (which gave me a practically flat line as predictions) for generating the predictions. I was wondering if there is any usually suspected general reason for why this could be the case, or is it just that one method works well for a particular type of problem?


texts: Terminology: "flat geometry" in the context of clustering

texts:
Sklearn's documentation refers to "flat" or "non-flat" geometry of clusters to describe the use-cases of their implemented clustering algorithms. Those terms are not directly defined. However, the following quote states:

  Non-flat geometry clustering is useful when the clusters have a specific shape, i.e. a non-flat manifold, and the standard euclidean distance is not the right metric.

So far, I do understand that a manifold type of shape is poorly represented by a cluster centroïd and that clustering such dataset should rather rely on local density, nearest neighbors, or connectivity constraint. However, I do not understand how it relates to the concept of flatness according to Wikipedia's definition:

  In geometry, a flat is a subset of a Euclidean space that is congruent to a Euclidean space of lower dimension.

It seems contradictory to me. For instance, a hypersphere is not geometrically flat but would be flat using sklearn's terminology.

My questions are then:

Are flat and non-flat geometry a legit terminology in machine learning and statistics?
If yes, what is the mathematical definition?
If no, what is the most appropriate alternative? (e.g. manifold vs convex ???)


texts: What are the advantages/disadvantages of neural net packages in R?

texts:
I've seen that there are four neural net packages in R:

neural
neuralnet
nnet
RSNNS
H2O

What are the advantages/disadvantages of those compared with each other?

As I found out NeuralNetTools only provides additional tools.


texts: Similarity measure for multivariate time series with heterogeous length and content

texts:
I am interested in clustering multivariate N time series of T'values' each(different lengths) using python. Each variable have many trends and values which are simultaneously numeric and nominal.

A sample $T_{i}$ in the dataset has the following format:

TimeStamp       | Sensor0 | Sensor1| Sensor2
2015-02-05 11:30|<Min     | On     | off
2015-02-05 11:31|<Min     | on     | off
2015-02-05 11:32| Action2 | 10     | 0.0001
2015-02-07 11:33| Action2 | 10     | 0.00012
2015-02-07 11:34| Action2 | 10     | 0.00012
2015-02-07 11:35| Action2 | 20     | 0.00015

Another sample $T_{j}$ in the dataset has the following format:

TimeStamp       | Sensor0 | Sensor1| Sensor2
2015-10-05 11:30| Action2 | 11     | off
2015-10-05 11:31| Action1 | 11     | off
2015-10-05 11:32| Action2 | NAN    | 0.0001
2015-10-07 11:33| Action3 | NAN    | 0.00012
2015-10-07 11:34| <Min    | 10     | 0.00012
2015-10-07 11:35| <Min    | 15     | on

For the missing values (not numeric), they were not collected by the sensors so my idea was to replace them by minimum values., given that all values are strictly positive.
Otherwise, they would be considered as missing values. In which case the problem would be of finding a similiraty measure that can compare missing values (off,on..) and numeric values.

I am wondering if there is a similarity / distance measure already exist in the litterature to compare such multivariate timeseries, with hetergonuos lengths, and whether this kind of problem has already been formulated in the papers, books or else for R and python.

Thanks for your advice.


texts: Why Gradient methods work in finding the parameters in Neural Networks?

texts:
After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things.

Let us focus on the supervised learnings (for example).
Given a set of data $\mathcal{D}_{train}=\{(x_i^{train},y_i^{train})\}$ and $\mathcal{D}_{test}=\{(x_i^{test},y_i^{test})\}$ where we assume $y_i^{test}$ are unknown,
the goal is to find a function
$$
f_\theta(x), \qquad \text{such that} \quad f_\theta(x_i^{test}) \approx y_i^{test}.
$$
To do this, we need a model for $f$. Typically, neural networks are frequently employed. Thus we have
$$
f_\theta(x) = W^{(L+1)}\sigma(W^{(L)}\sigma(\cdots \sigma(W^{(1)}\sigma(W^{(0)}x+b^{(0)})+b^{(1)})\cdots )+b^{(L)})+b^{(L+1)}
$$
where $\theta = \{W^{(i)},b^{(i)}\}_{i=0}^{L+1}$.
Then $f_\theta$ is a neural network of $L$ hidden layers.
In order to find $\theta$, typically, one define a loss function $\mathcal{L}$.
One popular choice is
$$
\mathcal{L}(\mathcal{D}_{train}):= \sum_{(x_i^{train},y_i^{train})\in \mathcal{D}_{train}} \left(f_\theta(x_i^{train}) - y_i^{train} \right)^2.
$$
In order to find $\theta^*$ which minimizes the loss function $\mathcal{L}$,
a typical (or it seems the only approach) is to apply the gradient method.

As far as I know, the gradient method does not guarantee the convergence to the minimizer.

However, it seems that a lot of research papers simply mention something like

  We apply the standard gradient method (e.g., Adam, Adadelta, Adagrad, etc.) to find the parameters.

It seems that we don't know those methods can return the minimizer. This makes me think that it could be possible that all the papers rely on this argument (utilizing the parameters found by gradient methods) might be wrong. Typically, their justifications are heavily on their examples saying it works well.

In addition to that, sometimes, they mentioned that they tuned some parameters to run gradient methods. What does that mean ``tune"? The gradient method high depends on the initialization of the parameter $\theta^{(0)}$.
If the initial choice were already close enought to the minimizer, i.e., $\theta^{(0)} \approx \theta^*$, it is not surprising that it works well.
But it seems that a lot of trials and errors are necessary to find a proper (good and working well) initialization. It sounds to me that they already found the good solution via trials and errors, not gradient methods. Thus tuning sounds to me that they already found a parameter which already closes to $\theta^*$.

I start to think that there may be something I am not aware of as the volume of such researches is huge. Did I miss something? Or can we just do research in this manner? I am so confused...
I am not trying to attack or critize a specific paper or research. I am trying to understand.

Any comments/answers will be very appreciated.


texts: Inception Score (IS) and Fréchet Inception Distance (FID), which one is better for GAN evaluation?

texts:
IS uses two criteria in measuring the performance of GAN: The quality of the generated images, and their diversity based on the entropy of the distribution of synthetic data.

On the other hand, FID uses the Inception network to extract features from an intermediate layer.

But how can I know that what to be used in a given situation. Any comparisons between them or recommendation for usage?


texts: What is the convergence criteria of a semi-supervised learning algorithm?

texts: I would like to know when to stop doing semi supervision? For example, if I learn a classifier from a small dataset and then use it to label a pool of unlabelled dataset. I then use the newly labelled data to train my classifier again. How long should this process be continued?


texts: Python : How to use Multinomial Logistic Regression using SKlearn

texts:
I have a test dataset and train dataset as below. I have provided sample data with min records, but my data has more than 1000's of record. Here if you see E is my target variable which I need to predict using an algorithm. It has only four categories like 1, 2, 3, 4. It can take only any of these values.

Training Dataset:

A    B    C    D    E
1    20   30   1    1
2    22   12   33   2
3    45   65   77   3
12   43   55   65   4
11   25   30   1    1
22   23   19   31   2
31   41   11   70   3
1    48   23   60   4

Test Dataset:

A    B    C    D    E
11   21   12   11
1    2    3    4
5    6    7    8
99   87   65   34
11   21   24   12

Since E has only 4 categories, I thought of predicting this using Multinomial Logistic Regression (1 vs Rest Logic). I am trying to implement it using Python.

I know the logic that we need to set these targets in a variable and use an algorithm to predict any of these values:

output = [1,2,3,4]

But I am stuck at a point on how to use it using python (sklearn) to loop through these values and what algorithm should I use to predict the output values? Any help would be greatly appreciated.


texts: Initial values of memory and previous block output in LSTM?

texts:
I am trying to understand LSTM and reading colah blog , As LSTM structure looks like this :

So LSTM takes three inputs:

Input vector
Memory from previous block
Output from previous block

and return :

memory from current block
output from current block

which will become previous memory and previous block output for next state ,

My confusion is what is the initial values of ct-1 ( memory from previous block) and ht-1 ( hidden output from previous block) , since there is no previous block then what are initial values ?


texts: How are Hyperplane Heatmaps created and how should they be interpreted?

texts:
For nonlinear data, when we are using Support Vector Machines, we can use kernels such as Gaussian RBF, Polynomial, etc to achieve linearity in a different (potentially unknown to us) feature space and the algorithm learns a maximal separating hyperplane in that feature space.

My question is how do we create heatmaps such as the one seen in the image below to show this max. separating hyperplane in our original space and how should it be interpreted?


texts: What could be the value of having a business analyst in a data science team?

texts:
Correct me if I am wrong, but while business analysts are often an important role in software delivery teams, but it seems not the case in the data science field.

Commonly available data scientist skill sets state also importance of story-telling skills and understanding customer's business, but is that enough?


texts: How to relate the regularized regression coefficients with their number in the plot?

texts:
I have a model with 13 independent variables (all of them are significant and 2 of them are categorical variables) and 678 observations. I ran the ridge regression and lasso on the data set using glmnet package. Here is the output.

I was wondering how I can relate the variables to their number and what L1 norm mean.
Thanks.


texts: Early stopping and final Loss or weights of models

texts:
In a deep model, I used the Early stopping technique as below in Keras:

from keras.callbacks import EarlyStopping

early_stopping = [EarlyStopping(monitor='val_loss',
                          min_delta=0,
                          patience=2,
                          verbose=2, mode='auto')]

model.fit(train_x, train_y, batch_size=batch_size,
                epochs=epochs, verbose=1,
                callbacks=early_stopping,
                validation_data=(val_x, val_y))

model.fit(train_x, train_y, batch_size=batch_size,
                epochs=epochs, verbose=2,
                callbacks=early_stopping,
                validation_data=(val_x, val_y))

Now, when I run this code, in the output it prints the loss value for training and validation of each epoch.

I set the patience=2 in the early stopping. So, it continues the training process two times after when the validation loss increased instead of decreased.

Some things like this:

Epoch 1/10
- 198s - loss: 99.7160 - val_loss: 123.0397
Epoch 2/10
- 204s - loss: 78.7000 - val_loss: 109.0344
Epoch 3/10
- 208s - loss: 65.4412 - val_loss: 78.0097
Epoch 4/10
- 268s - loss: 61.9812 - val_loss: 79.0312
Epoch 5/10
- 298s - loss: 59.1124 - val_loss: 79.3397
Epoch 6/10
- 308s - loss: 57.2200 - val_loss: 218.0397
Epoch 00007: early stopping

In the end, what will be the final weights of the model and the Loss values? The final time of training or two times before it?

If it considers the final epoch, so should it be better if I set the patience as little as possible to overcome the overfitting?

Thank you


texts: getting error while scrapping Amazon using Selenium and bs4

texts:
I'm working on a class project using BeautifulSoup and webdriver to scrap
Disposable Diapers on
amazon for the name of the item, price, reviews, rating.

My goal is to have something like this where I will split this info in
different column:

 Diapers Size 4, 150 Count - Pampers Swaddlers Disposable Baby Diapers, One
 Month Supply
   4.0 out of 5 stars
   1,982
   $43.98
  ($0.29/Count)

Unfortunately, I get this message after the 50 data appears: message: no such
element: unable to locate element: {"method":"css selector","selector":".a-last"}

Here is my code:

URL = "https://www.amazon.com/s?
k=baby+disposable&rh=n%3A166772011&ref=nb_sb_noss"
driver = ('C:/Users/Desktop/chromedriver_win32/chromedriver.exe')
driver.get(URL) html = driver.page_source soup = BeautifulSoup(html, "html.parser")
df = pd.DataFrame(columns = ["Product Name","Rating","Number of
Reviews","Price","Price Count"])

while True:
for i in soup.find_all(class_= "sg-col-4-of-24 sg-col-4-of-12 sg-col-4-of-36
s-result-item sg-col-
4-of-28 sg-col-4-of-16 sg-col sg-col-4-of-20 sg-col-4-of-32"):
ProductName = i.find(class_= "a-size-base-plus a-color-base a-text- normal").text#.span.get_text
print(ProductName)
 try:
Rating = i.find(class_= "a-icon-alt").text#.span.get_text()
  except:
 Rating = "Null"
print(Rating)
try:
NumberOfReviews = i.find(class_= "a-size-base").text#.span.get_text()
 except:
 NumberOfReviews = "Null"
 print(NumberOfReviews)
 try:
Price = i.find(class_= "a-offscreen").text#.span.get_text()
except:
Price = "Null"
print(Price)
try:
PriceCount = i.find(class_= "a-size-base a-color-secondary").text#.span.get_text()
except:
PriceCount = "Null"
print(PriceCount)
df = df.append({"Product Name":ProductName, "Rating":Rating, "Number of
Reviews":NumberOfReviews,
"Price":Price, "Price Count":PriceCount}, ignore_index = True)
nextlink = soup.find(class_= "a-disabled a-last")
if nextlink:
print ("This is the last page. ")
break
else:
progress = driver.find_element_by_class_name('a-last').click()
subhtml = driver.page_source
soup = BeautifulSoup(subhtml, "html.parser")

Unfortunately, I hit a block road trying to figure out why it is not taking a_last.


texts: Cluster images labels in some given categories using word embeddings

texts:
Given:
set of images Labels in string format each one. Also I've given a set of Categories, also in string. ($Images \neq Categories $)
Goal: I need to map given labels to given categories to "squeeze" our labels set onto categories set.
Toy example: given two sets: Labels = ['apples', 'juice', 'sun', 'volleyball player', 'birds', 'trees'] and Categories = ['fruits', 'summer']. So result will be simple dictionary dict with elements from Categories set:
dict['fruits'] = ['apples', 'juice', 'trees']; dict['summer'] = ['sun', 'volleyball player', 'birds']
Question: is there is a way to do that? Maybe there are many approaches with growing complexity --- it would be good to find out it all.
My approaches:

First idea is very simple and intuitive: we can clusterize our Labels, using K-means, or Agglomerative Clustering. Then take clusters centroids and assigning to each centroid closest in cosine similarity vector from Categories. Mapped.
Second idea is a bit complex --- to use Latent Dirichlet Allocation (LDA). First part very similar with first idea: we need to form documents, so we can clusterize Labels and then just assume  that every cluster is a document. Then we got a set of topics and can also assign to each topic vector the vector from Categories. Mapped.

The problems in this approaches is a word representation. I want the algorithm to be most accurate, but understand also, that some words connected with another with number of contexts and have a number of meanings. So we need embeddings. Since BERT now is most powerful and interesting idea in word representation I can use BERT and hope it will be the most powerful approach (instead of word2vec).
Few more questions(optional answering, but appreciaеted):

What approach seems more reasionable?
Is it good assumption to make documents (in 2 approach) from labels, i.e. from not-necessarily related to each other?

Since I'm not a pro in DS yet, any help/comments/hints are appreciated.


texts: Understanding the algebra behind a specific partial derivative equation

texts:
I am following this article about neural networks.
Given:

Until here I understand everything, but then he continues to:

I don't understand how he got to that conclusion. I think he skipped some algebra steps that would have made it easier for me to understand.


texts: Clarifying Probability Mass Function (PMF)

texts:
I am currently reading Deep Learning book, and I want to get better understanding of probability theory. In chapter 3.3.1 of Deep Learning book it states that:

  Often we associate each random variable with a diﬀerent probability
  mass function and the reader must infer which PMF to use based on the
  identity the random variable, rather than on the name of the
  function;P(x) is usually not the same as P (y).

And not many paragraphs latter, it saids the following:

  Probability mass functions can act on many variables at the same time.
  Such a probability distribution over many variables is known as a joint
  probability distribution.P(x=x, y=y) denotes the probability
  that x=x and y=y simultaneously. We may also write P (x, y) for brevity

I am having hard time grasping these two paragraphs. What do they mean when they say that P(x) is USUALLY not the same as P(y). As I understood, random variable is basically a random phenomenon from a real world that we wish to model. And each random phenomenon has its own Probability Mass function. Does this mean that the first paragraph indicates that random variable y represents a different phenomenon, and in the second paragraph random variable y represents the same type of phenomenon as x and that is why we use the same probability mass function?

Thanks in advance!


texts: Dataset for Named Entity Recognition on Informal Text

texts:
I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the documents in my dataset, I'm looking for out of domain data that's a bit more "informal" than the news article and journal entries that many of today's state of the art named entity recognition systems are trained on.

Any recommendations? So far I've only been able to locate 50k tokens from twitter published here.


texts: Purpose of test data in binary classification

texts:
I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as Accuracy, F1Score, and etc.
UserId,UserName,AppId,Label
1,Lin,1,1
1,Lin,2,1
1,Lin,3,1
1,Lin,4,1
1,Lin,5,1
1,Lin,6,0
1,Lin,7,0
1,Lin,8,0
1,Lin,9,0
1,Lin,10,0
2,Tim,1,1
2,Tim,2,1
2,Tim,3,1
2,Tim,4,1
2,Tim,5,1
2,Tim,6,0
2,Tim,7,0
2,Tim,8,0
2,Tim,9,0
2,Tim,10,0
3,Kirstin,1,1
3,Kirstin,2,1
3,Kirstin,3,1
3,Kirstin,4,1
3,Kirstin,5,1
3,Kirstin,6,0
3,Kirstin,7,0
3,Kirstin,8,0
3,Kirstin,9,0
3,Kirstin,10,0
4,Michael,1,1
4,Michael,2,1
4,Michael,3,1
4,Michael,4,1
4,Michael,5,1
4,Michael,6,0
4,Michael,7,0
4,Michael,8,0
4,Michael,9,0
4,Michael,10,1
5,Timmy,1,0
5,Timmy,2,0
5,Timmy,3,0
5,Timmy,4,1
5,Timmy,5,1
5,Timmy,6,0
5,Timmy,7,1
5,Timmy,8,1
5,Timmy,9,1
5,Timmy,10,0
6,Lick,1,1
6,Lick,2,0
6,Lick,3,0
6,Lick,4,0
6,Lick,5,1
6,Lick,6,1
6,Lick,7,0
6,Lick,8,0
6,Lick,9,1
6,Lick,10,0
7,Shawn,1,1
7,Shawn,2,0
7,Shawn,3,1
7,Shawn,4,0
7,Shawn,5,1
7,Shawn,6,0
7,Shawn,7,0
7,Shawn,8,0
7,Shawn,9,1
7,Shawn,10,1

To test the trained model, I used this test set.
UserId,UserName,AppId,Label
8,Lianne,1,1
8,Lianne,2,0
8,Lianne,3,1
8,Lianne,4,0
8,Lianne,5,0
8,Lianne,6,0
8,Lianne,7,0
8,Lianne,8,0
8,Lianne,9,1
8,Lianne,10,0

Following is the prediction results.
8,Lianne,1,1  => Predicted true
8,Lianne,2,0  => Predicted false
8,Lianne,3,1  => Predicted true
8,Lianne,4,0  => Predicted true
8,Lianne,5,0  => Predicted true
8,Lianne,6,0  => Predicted false
8,Lianne,7,0  => Predicted false
8,Lianne,8,0  => Predicted false
8,Lianne,9,1  => Predicted false
8,Lianne,10,0 => Predicted false

// Sum this up...
TP = 2
TN = 5
FP = 2
FN = 1

Accuracy = 2 + 5 / 2 + 5 + 2 + 1 => 7 / 10 => 0.7
Recall = 2 / 2 + 1 => 0.66
Precision = 2 / 2 + 2 => 0.5
F1Score = 0.5 * 0.66 / 0.5 + 0.66 * 2 =  0.33 / 1.16 * 2 = 0.5689

The calculation was proven alright and as I expected, the model is right on almost all cases except the app id 2. While I'm doing this, this idea comes into my mind.

" What test data is for? "
" Is it for checking out the desired model metrics like above? "
" Why bother with some proportion of data to perform a test? when you just need a certain, small group of dataset to evalute the model? "
" Even if I have a thousands of million rows of training dataset, wouldn't it be enough to have a complete small, unique set of dataset that covers the whole variations? do I still need some proportion of testdata from the training set? "

Well, if I already know that the app id range is from 1 to 10, isn't it fairly enough to just prepare only 10 rows of test set as long as it has app id 1 to 10? even when the training data set is gigantic?
Why do I need to care about N% of test data set from the training data set?


texts: Convert Atypical Date Format for Time Series in Python

texts:
I have an atypical time format that I need to convert into a datetime index for time series analysis.  I'm working in Python / Pandas.

The column is 'BC_DT', and the format is "27-MAR-18".  Example is below.

BC_DT
27-MAR-18
28-MAR-18
29-MAR-18

I tried this method, but I'm getting an error: ValueError: time data '27-MAR-18' does not match format '%d-%b-%Y'

df['Converted_Date'] = df['BC_DT'].apply(lambda x: dt.datetime.strptime(x, '%d-%b-%Y'))


texts: Denoising Autoenoders with Variable Length Input

texts:
I'm working on a problem with data from a continuous real-valued signal. The goal is to use ML to smooth the signal based off of past data. To accomplish this, the signal is windowed into a period that's meaningful within the domain. The problem is that this period is highly variable in length.

I've reviewed this question and this question and neither solve the problem, they are more about how to deal with missing values.

Seeing as denoising autoencoders are based off of matrix multiplication, this presents a serious problem. What is the standard approach in such a situation? Should I define an arbitrary (large) window size, and expand windows that are too small (and vice versa)? Or is there a better approach for variable length inputs?


texts:
I went through this comparison of analytic disciplines and this perspective of machine learning, but I am not finding any answers on the following:

How is Data Science related to Machine learning?
How is it not related to Machine Learning?


texts: Dataset recommendation, ordinal target variable

texts:
Does anyone have any recommendations for datasets with an ordinal target?
I'm currently using a rounded target from the Boston dataset.
from sklearn import  datasets
boston = datasets.load_boston()
X, y = boston.data, boston.target

y -= y.min()
y_ord = np.round(y)


texts:
I have training data in form of pair of documents with an associated label - {doc1, doc2, label}. Label is defined as function of pair of documents.

Now I want to build a model which can predict the label given two new documents.

I want to try different representation of document (instead of common ones say TF-IDF). Can I use vectors (topic distribution) from LDA as features for a classifier?


texts:
I'm very new to data science in general, and have been tasked with a big challenge.

My organization has a lot of documents that are all sorted on document type (not binary format, but a subjectively assigned type based on content, e.g. "Contract", "Receipt", "Statement", etc...).

Generally speaking assignment of these types is done upon receipt of the documents, and is not a challenge, though we would like to remove the human element of this categorization.  Similarly, there are times when there are special attributes that we would like to identify, like "Statement showing use."  Thus far, this is entirely done by human intervention.

I am a python programmer, and have been looking at tools to extract the text from these docs (all PDFs, all OCR'ed and searchable) and run analysis.  Research has led me to look at standard libraries like NLTK, scikit-learn and gensim.  But I'm struggling to identify what would be the best methodology for categorizing newly received documents.

My research is leading me down a few paths...one is creating a Tf-iDf vector model based on a sampling of current corpa and then creating a model for an incoming document's corpus and doing a naive bayes analysis against existing models to discern which category the incoming document belongs to based on highest probability.  Question 1: is this right?  If so question 2 becomes what is the right programmatic methodology for accomplishing this?

The reason I bring this up at all is because most tutorials I find seem to lean toward a binary discernment of text corpa (positive vs negative, spam vs ham).  I did see scikit-learn has information on multi-label classification, but I'm not sure I'm going down the right road with it.  The word "classification" seems to have different meaning in document analysis than what I would want it to mean.

If this question is too vague let me know and I can edit it to be more specific.


texts: Depending samples in ad ranking and click rate prediction

texts:
I am struggling with the following problem:
Suppose we fit a machine learning model to model advertisers click rates. I used a Logistic Regression approach using a one-hot/dummy encoding.

We have two advertisers A and B with a click rate of 10% and 9% respectively. Hence, we would rank A before B and would show users ads from A more often, say 90% of all times. Hence our overall click rate would be close to 10%.

Now we introduce a new feature which says a user either clicked on the advertiser before or not. For advertiser A, we observe a click rate of 8% for users that have not clicked before and a rate of 30% for users that have at least clicked once on an ad of the advertiser. For advertiser B we see no difference in the click rate for this feature.

So when it now comes to ranking the ads and we get a "fresh" user that has not clicked on both advertisers, we would rank B above A. Since fresh users are obviously more often, we would mostly show users ads from B, say 90%.

As a consequence, our overall click rate would drop from almost 10% to a little over 9%. The machine learning model does everything correct since we did not introduce any information that the two states of this new features are timely dependent.

From an intuitive approach I would still rank advertiser A over B for these fresh users since it's not clear if they're going to click.

Any ideas how to tackle this problem to get a ranking that in the end maximizes the click rate?


texts: When using Scikit Learn Grid Search, why are my train and cv scores high, but my test score is a lot lower?

texts:
I'm using scikit learn to run some models, and am very confused as to why my test score is so much lower than my cv score and my train score.
At the start, I do a 80-20 train-test split. On the train set, I run a gridsearch with 5-fold cross validation to choose hyperparameters. refit is set to true, so after picking hyperparameters the model is refit onto the whole training set, and used to predict the test set.
When I look into cv_results_ I find that my mean_train_score (what I'm interpreting to be the train score for each k-fold cross validation loop) is really high. When I look at the mean_test_score (what I'm calling cv score), it is also really high. But then when I use my external test score, the scores are really low. This is true for all models I'm using (I'm testing 10 models). The numbers can be seen in the following picture.
Note: I'm using F1 Macro Score as measure of model performance.

(LR) Logistic Regression, (QDA) Quadratic Discriminant Analysis, (NN) Nearest Neighbors, (LSVM) Linear Support Vector Machine, (RBFSVM) Radial Basis Function Support Vector Machine, (NB) Naive Bayes, (ANN) Artificial Neural Network, (RF) Random Forests, (AB) AdaBoost Random Forests, (GB) Gradient Boosted Random Forests
So since my test set performance is a lot lower than my training score, I'm sure I'm overfitting. But I don't know why my CV score would do so well then? If my setup is prone to overfitting, wouldn't I see overfitting with the 4/5 of my train set when I did the 5-fold cross validation, meaning my CV score would be low too? I don't see why I wouldn't overfit leading to high CV scores in the 5-fold CV step, but overfit for low performance in the testing set.


texts: Model for Differing Number of Rows per Observation

texts:
Looking to build a response model (click or no click) on marketing data which displays varying number of offers to a person. I don't want to model which offer they click but do they click any of the offers presented to them. My issue is how to deal with the differing number and types of offers?

Example data could be one table of id's:

id   clicked
001       1
002       0
003       1

And varying number of offers per id:

id  discount_rate  on_amt
001     0.05       100
001     0.10       500
002     0.03        50
003     0.05       100
003     0.10       300
003     0.15       500

Do I create features from the offer data set such as average discount_rate, max on_amt etc.? Or create a very large binary sparse matrix of binned offer types such as rate_5-10_amt_0-50 1/0 and rate_5-10_amt_50-100 1/0 ...?

Or is there a good model that handles variable data like this?


texts: how to find classification accuracy in autoencoders?

texts: how can we find the accuracy autoencoders for classification of images? because we will get the reconstruction of the image and when we will plug in the test data it will spit out the image but how would we able to calculate the accuracy? or we should take difference between the prediction and input of the image .


texts: I want to know which machine learning algorithms can be used for trajectory classifications?

texts:
I am working on project for clustering of air objects based on their trajectories.
Like I want to train a model on dataset of different flying object's trajectories so later I can predict what type of object is based on trajectory data.
Now trajectory data include 4 things (Altitude, Longitude, Latitude, Time).
So based on set of such dataset we may be able to classify objects like plane, rocket, missile etc.
What I cant figure out is which algorithms can be used?
I first thought about SVM. Later I thought "Long Short Term Memory" can be used. But I am not very sure.
And I am new to machine learning. SO any help is appreciated.


texts: How to understand equations in research papers?

texts:
I was searching for latent class logit model for conjoint analysis. i found a paper which has equations for this model.

I have co-workers who knows how to decipher the meaning of these equations and write these algorithms in any language, from scratch. One of my co-worker told me that he understood what is written in this paper and can write the algorithm in R. My co-workers mostly have masters degree in engineering. I have done graduation in accounting and fortunately landed in this job of analytics. I am very keen to learn this art/science of deciphering equations in research paper.  Can anyone suggest which discipline is it or lets say any recommendation of online course or books that can help me learn this? I will be indebted for life.

Thanks!


texts:
I was reading Video-to-Video Synthesis (link) paper, in related works for future video prediction it is mentioned that existing methods fail because of regress-to-the-mean problem.
What exactly is the regress-to-mean problem and how does it affect the image and video quality.


texts:
I'm hitting an unusual roadblock in my quest to represent a set of data for the layman, and thought I'd ask for advice on how best to accomplish this task.

My data points are represented by a 4-float tuple (a, b, c, d), the sum of which is constant. These represent test conditions, to which an outcome z is assigned. This outcome is also a float, but that's largely irrelevant.

The fact that there are four effective x-axes really is throwing me off regarding the choice of visualization, and that's where I'd really like some help. If you were in my shoes, what would you render this as?

Just for completeness sake, here is a (small) extract of one of the sets:

0,3000,2500,750,101788
250,3000,2500,500,100458
500,3000,2500,250,99439
750,3000,2500,0,98573
0,3000,2750,500,101993
250,3000,2750,250,100834
500,3000,2750,0,99813
0,3000,3000,250,102370
250,3000,3000,0,101150


texts:
Please find attached a part of the code which explains what I'm trying to do. Essentially I'm trying to predict the sales of supermarket stores. Im using RandomForestRegressor for this and have predicted the results on the test set. The cross validation is done on the training set with a mean accuracy of 0.52 and I have tried to calculate RMSE which comes to 1180. Now, I want to know how do I make sense of all this and know how well it performed on the test set and how to evaluate my model. It'll be extremely helpful if you could make me understand. Thanks

Ignore the np.exp(rmse) part

Best,

Pranveer


texts: increase performances in neural networks

texts:
I am starting being interested in neural networks, and I am writing some code about it. But, differently from methods like support vector machines, random forests,..etc., to me it seems more like a black box which I can't control.

So my question is:

What are methods to increase accuracy and other performances in a neural network?

Thanks in advance.


texts: Autoencoder or layer-based dimensionality reduction?

texts:
I have a few TB of wide data. I want to reduce the number of features in my dataset before feeding my dataset into a classification model... or should I not?

Obviously, I will want to try both methods if my cluster budget allows, but is there any obvious theory as to why either of the following approaches would be better than the other?

Autoencoder (tanh, tanh, tanh layers)
Or: drastically reduce the number of inputs, perhaps via dropout, to the second and third layers of the classification model itself. Also, droput


texts: What do all the column names mean in the SentimentAnalysis package's analyzeSentiment() function in R?

texts:
If you used the analyzeSentiment function from SentimentAnalysis in R:

library(tm)
library(SentimentAnalysis)
cor <- VCorpus(VectorSource(c('testing sentiment', 'this is great'))
sent <- analyzeSentiment(cor)
names(sent)

The names of the columns are:

"WordCount", "SentimentGI", "NegativityGI", "PositivityGI", "SentimentHE", "NegativityHE", "PositivityHE", "SentimentLM", "NegativityLM", "PositivityLM", "RatioUncertaintyLM" "SentimentQDAP", "NegativityQDAP", "PositivityQDAP"

Other than wordcount, what do all these labels mean?  I didn't see any definitions in the vignette or docs.


texts: How exactly the hidden state works in an RNN ? How to decide on how many past instances to consider?

texts:
I am unable to grasp the working of RNN because in different tutorials, it is explained differently.

Please correct me as I have considered that:

In a Many to one Model: if a sentence is like My name is Shady becomes Mon nom Shady then how are the inputs were fed to model? IMO it was that My produced Mon by multiplying with w0 and produced some hidden state as h1 by multiplying with a0. Then (h1 * a1)+ (name * w1) produced the nom. I want to ask that how is the state h changing? is h_i at the ith time a single float that the i+1_th input will use? If so,then why do they say that it uses  previous N states when it is using a cumulative function of all the previous states that have already occured???

And if the sentence is paragraph of 10000 words, will the last input use cumulative function state which is a state defined by all the previous 9999 inputs??


texts:
How can I vectorize this code in R?

data <- data.frame(A = rep(5, 5), B = rep(0, 5))
data$abstract <- c("no abstract available", "A", "A", "B", "no abstract available")

for (row in (1:nrow(data))){
  data [row,"abstract"] <- gsub("no abstract available"," ",data[row,"abstract"])
}


texts:
Given a Pre-trained CNN model, I extract feature vectors for 3450 Reference (Winter) and 3450 Query images (Spring) and compare features with euclidean distance to plot the distance matrix besides ground truth.The ground truth matrix indicates the one-to-one image correspondences as follows:

img_i in Query --> img_i in Reference database

I'd like to validate my result with Precision-Recall curve and here's a code I use:

def draw_PR(cm_gt, cm_pred, query_season, ref_season):
    pr_curve = []
    for th in np.arange(0, .99, 5e-4): # apply threshold
        tp = (cm_pred < th) & (cm_gt == 0)
        tp_plus_fp = (cm_pred < th)
        tp_plus_fn = (cm_gt == 0)
        try:
            p = float(np.sum(tp))/np.sum(tp_plus_fp)
            r = float(np.sum(tp))/np.sum(tp_plus_fn)
            pr_curve.append([th, p, r])
        except:
            pass
    pr_curve = np.array(pr_curve)
    if pr_curve.size > 0:
        plt.figure()
        plt.plot(pr_curve[:, 2], pr_curve[:, 1], 'b')
        plt.ylabel('Precision')
        plt.xlabel('Recall')
        plt.axis([0, 1.01, 0, 1.01])
        plt.title('PR')
        plt.tight_layout()
        plt.grid()
        plt.savefig('R_'+ref_season+'_Q_'+query_season+'_PR' + '.pdf', bbox_inches='tight', dpi=1000)

But my Precision-Recall curve looks wrong:

Does anybody kindly help me figure out how to plot precision-recall given distance matrices of prediction and ground truth?

Cheers,


texts: Is it possible to rebuild model made by autokeras from scratch

texts:
I want to ask, is it possible to rebuild model from autokeras from scratch and get the same results.

The model looks like this, and I tried rebuilding it but I got results that are not even close to the autokeras model. I am using this to binary classification problem. Also I don't know how to extract an optimzer, learning rate etc.