texts:
When I design a document classifier using traditional feature engineering, I would prefer (to Boolean model) tf-idf model to represent a document into a vector because intuitively Boolean model loses information of how important each word is for classifying a document into certain class.

I mean using Boolean model for representing a document as a vector is to give it a less meaningful position in n-dimensional vector space than tf-idf-based feature extraction when each dimension represents a term, by using discrete value rather than a continuous value, since discrete(0 or 1) value is made to ignore the difference of weight of each term although parameter tuning process may optimize coefficient of each term when using linear combination for document classification.

Am I justified in my thinking that using Boolean feature for bag-of-words model to extract feature vector from a document is not a good choice for the above-mentioned reason?

I already know the recent approach like representation learning and dimensional reduction like word embedding or BERT language model. My question is limited to some traditional feature extraction from document data.


texts: Customer Targeting for CRM Marketing Campaign

texts: I need help or ideas to solve the below business challenge. Sample questions has been provided. A snapshot of the sample data has been attached below:


texts:
[If this is the wrong place to ask this question, please let me know where I can receive better answers:)]

So I'm building an internal tool for an digital media organization. The tool aims to track the number of hours (or any other measure of effort) spent on each brand by different people-roles (Eg: Designers, Content Writers, etc). Using the data collected for the first few months, the organization aims to achieve a better estimate for future quotes given to new clients.

I have thought about the following roles and their metrics:

Designers: Hours
Content Writers: Hours
SEO Guys: Hours
Client-Facing people. (Who actually talk to all the clients): Effort on a scale of 1-10

Whenever somebody starts work on a client (brand), they will start their timer and when done, they'll stop the timer. The tool will log the amount of time spent. Also, the current quotes to the different brands will be entered into the system initially.

I can think about this as a regression problem with the hours put in by different roles as the features and the 'quote' as the output. But I don't think there will be enough data collected in the first 2 months to give a realistic estimate for quotes to new clients.

What can my cost function be?

This will be my first project of this kind. Please give me some ideas. Even new approaches are welcome.

Thanks :)


texts:
I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques.

I am building a function to calculate a social welfare score/rank of the countries in the world. While doing so, I come across multiple outliers which essentially skew the results.

I have a few questions:

Which normalization function to use and when? I am currently using the z score.
If i come across an outlier and I know the reason for the outlier and want to avoid it affecting the result, how should I modify the value?eg . replace with mean/median. Which technique should I use and when?
After calculating the z-scores, how to devise a function of the z scores? Is it trial and error based or I can apply some technique to find the best results. Since I do not have a strict result that I am expecting, how do I calculate the coefficients and operations for the z-scores?


texts: Machine Learning Text Categorization with Data in multiple Languages

texts:
I want to label natural text Dokuments in various different categories (arround 400 different categories) using Neural Networks. I have around 50.000 documents already labeled. The problem is around half of them are english only, the others are either chinese or a mix of chinese and english.

Would it be possible to train a network with all of the documents and get a network with an good quality or is it more useful to separate the languages and train the network with english documents only (arround 20.000 docs then)?


texts: How to compare cosine distances across two groups of words?

texts:
I am using a word2vec model based on Wikipedia corpus. I was looking for a way to quantify if two sets of words - s1= {a1, a2...} and s2= {b1, b2, ...} - form distinct clusters in the vector space.
I also want to estimate if the words in one set are more tightly aligned in the vector space than the other - i.e., whether the average cosine distance among every pair of words in set A is statistically significantly higher/lower than the average cosine distance among every pair in set B.
Is there a way to make such relative comparisons using some kind of a graph metric or feature extraction techniques?


texts: summarizing time series dataset: extract time window sliding, change points, pattern seasonality in time series

texts:
I need to detect list of change points in time series dataset (temperature), and I need to split dataset into set of classes (patterns) and detect seasonality of each class (pattern). for example suppose that we create 2-classes named called and hot, each class contains readings that represent each state, I need to know seasonality of these state over the whole time period so that I can summarize dataset as hot state occurs from the time X to Y over month Z and from time ... to ... over month ...
- it is preferable if there is a solution written in python/C++/C#


texts: Cloud computing with country-specific region for Switzerland

texts:
Are there any cloud computing services that allow for processing and storing data exclusively in Switzerland? Are there any that have machine-learning-specific functionality?

Most do not seem to have this capability:

https://cloud.google.com/compute/docs/regions-zones/

https://aws.amazon.com/about-aws/global-infrastructure/

https://azure.microsoft.com/en-us/regions/

In some industries, such as finance, this is a strict requirement for regulatory and compliance reasons.


texts: predict a specific word(supplier) in documents(invoice)

texts:
I have several invoices that are passed in an ocr, then I get every invoice that is not an image but a document, I would use models of learning machines to predict the supplier of each invoice.

at the entrance I will have my bills and a supplier

I would like to ask you using the RNN or a word2vec or other model
if someone has an idea I am a taker. thank you
I tried to use the NLP but my bill is not a natural language, to tag my words.


texts: Problem upgrading pip command in Azure Jupyter notebook

texts:
Trying to install CNTK, I received a message saying that I should upgrade pip command here
This one is another question related to the upgrading of pip.
After the installation with the command
pip install cntk

I received another error message as shown in the attached image.


texts: time series forecasting of time to leave for multiple customers using one model

texts:
I am a beginner in the domain of forecasting and I was wondering if such a problem could be solved with time series analysis :

  given customer historical data of taxi pickups,along with the weather
  condition feature, is it possible to analyze their behavior on rainy days and
  predict their time to leave?  (means at what time might a customer x
  picks up a taxi next time knowing that he went out at 6:42 am
  yesterday, 7:00 am two days ago, 6:00 am three days ago ...and so on
  )?

here's how the data looks like globally :

what I want is to create a model trained on all the customer's historical data, but in the inference phase, it will be used to forecast for one customer at a time.

any guidance on how I can do this? is it a problem that can be tackled with time series? given that the problem here consists of predicting the time to leave, unlike their classic usage like sale forecasting and so on.

I would extremely appreciate any advice on methodologies implementing this.


texts: Why should the initialization of weights and bias be chosen around 0?

texts:
I read this:

  To train our neural network, we will initialize each parameter W(l)ijWij(l) and each b(l)ibi(l) to a small random value near zero (say according to a Normal(0,ϵ2)Normal(0,ϵ2) distribution for some small ϵϵ, say 0.01)

from Stanford Deep learning tutorials
at the 7th paragraph in the Backpropagation Algorithm

What I don't understand is why the initialization of the weight or bias should be around 0?


texts: Methods for standardizing string data with different naming conventions

texts:
Are there conventions for standardizing text that follows different naming conventions? I have a variable of owner names that include individual names and company names. The entries follow some common rules but not consistently and I'm trying to use the levenshtein distance to identify entries that are likely duplicates but may have been entered differently. I haven't done this before and I'm uncertain if there are common practices I can look to when it comes to making decisions to standardize the data before I calculate the levenshtein distance. Here's an example of some of the entries I may have:

DOE, JOHN + JANE
DOE JOHN + JANE J
DOE JOHN DOE JANE
COMPANY LLC
DOE JOHN + JANE J % COMAPNY LLC
DOE JOHN DOE JANE % MARLENE GRIGGS
DOE JOHN TRUSTEE OF DOE LIVING TRUST


texts: I have read that the SMOTE package is implemented for binary classification. In the case of n classes, it creates additional examples for the smallest class. Can I balance all the classes by running the algorithm n-1 times?


texts:
Is there a way to quantify or characterize some upper limits for how much one can "learn" from a dataset?

I got puzzled, when the neural state machine got me thinking components of AGI and lead me to the Visual Genome Dataset and further to generalization in neural networks (although within-distribution limited).

Consider an analogy: an intelligence analyst or X-ray specialist can use their image (satellite or X-ray) and be "informed" by that much better than a layman.

An expert has a) understanding about the domain and b) experience with the images, whereas a neural network only knows b) about the images, not a). Yet, both could make the classification, albeit with different learning material (in practice). The expert can be "informed" by an image and make a judgment of the world, using his/her education, but an image classification system only makes a classification, based on an image dataset.

By the analogy it seems: what can be understood from data is rather a property of the learner / the model, not dataset.

Likewise, a self-supervised network like GPT-2 can learn a great deal about text data, but only when there is huge amounts of it. At the same time, a human can learn a great deal, by reading a single Wikipedia article..

Thus, what can be learned from data depends greatly on the learner.

But, is there a way to characterize, given the learner, what can be learned from a given dataset? I hope this is more than a dataset quality question, maybe some qualitative characterizations could be made.

How should one approach and (properly) understand this question?

(I have feeling this cannot be answered, but let's give it a try).


texts: K-Means Clustering for data points with multiple attributes

texts:
I'm very new to K-Means clustering. Every example that I have seen has a two-dimensional data set.

I am working to classify recipes of varying ingredient composition into families. Each recipe is composed of a number of ingredients, and I want to group together the recipes that are similar to each other. Each recipe on average has 8 ingredients.

Is there any way I can use K-Means clustering to group these recipes together? Or would you suggest another unsupervised machine learning method?


texts: How to split a dataset into train and test sets for time series (multiple step-multiple output forecasting)?

texts:
I am trying to use an LSTM neural net to do multiple step / multiple output forecasting (I predict multiple values in one time knowing some values in the past). But, I have realized that I must be careful when splitting my univariate series into training and test sets. In this post, I am going to share what must be done in my opinion, but I really would like to have yours.

$(X_k)_{1\leq k\leq N}$ denotes my univariate time serie.

So basically, multiple output models are complex as they are trained to learn the dependencies between inputs and outputs as well as between outputs. This can be a problem as I transform my time serie like that :

$ D = (x=(X_k, ..., X_{k+p-1}) , y=(X_{k+p},..., X_{k+p+n-1}))_k $
with $p$ my number of steps and $n$ my number of outputs ($1 \leq k\leq N + 1 - (n+p)$). I want to split it into training and test sets. So, I must get :

$ D_{\text{train}} = (x=(X_j, ..., X_{j+p-1}) , y=(X_{j+p},..., X_{j+p+n-1}))_j $ where $j \leq N  - 2(n+1) - p$

$ D_{\text{test}} = (x=(X_i, ..., X_{i+p-1}) , y=(X_{i+p},..., X_{i+p+n-1}))_i $ where $i+p+n-1=N$

In this case, I can avoid overfitting due to a naive splitting of my dataset $D = D_{\text{train}} + D_{\text{test}}$ where $+$ denotes a concatenation operator (preserving the timeline). Indeed, this would lead to over-fitting in my opinion as I would have in my train some part of my test.

For a better understanding, let's take an example :
My univariate series is denoted as $(X_1,...,X_6)$. I want to predict $n=2$ values in the future knowing $p=2$ values of the nearest past. So, I transform my dataset as:

$ D = (x=[(X_1,X_2),(X_2,X_3), (X_3,X_4)] , y=[(X_3,X_4),(X_4,X_5), (X_5,X_6)])$

I split it naively, I get:

$ D_{\text{train}} = (x=[(X_1,X_2),(X_2,X_3)] , y=[(X_3,X_4)])$

$ D_{\text{test}} = (x=[(X_3,X_4)], y=[(X_4, X_5)]) $

I am afraid that it will lead to overfitting as the data $(X_3, X_4)$ is in both datasets. Maybe I am wrong, that's I want to know :)

Would you like to tell me if I am wrong and what do you think about?
Thanks.

** Edits are written in bold **


texts: Measuring performance of different classifiers without class type in data

texts:
To measure the performance of a classification algorithms on a dataset that has an attribute for class type, I divide my dataset to training and test samples and then create a confusion matrix for False Positive, False Negative, True Positive and True Negative samples. Hence there is a class type attribute(e.gYes or No ), the confusion matrix is pretty easy to calculate.

Now suppose that I have a dataset that lacks a class type attribute and all samples are of the class Yes .

How can I measure the performance of different classification algorithms using these kind of datasets?


texts: Predictive Clustering in R with Mixed Data

texts:
Does anyone know of a predictive clustering procedure in R for mixed data types?
For example, suppose that I had conducted an unsupervised k-prototypes procedure in order to estimate clusters for health states.
Is it possible to take the clusters from the k-prototypes procedure, and use a predictive/supervised procedure to estimate clusters at time t+1, t+2 etc?


texts: How does the validation_split parameter of Keras' fit function work?

texts:
Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ :

  validation_split: Float between 0 and 1. Fraction of the training data
  to be used as validation data. The model will set apart this fraction
  of the training data, will not train on it, and will evaluate the loss
  and any model metrics on this data at the end of each epoch. The
  validation data is selected from the last samples in the x and y data
  provided, before shuffling.

Please note the last line:

  The validation data is selected from the last samples in the x and y
  data provided, before shuffling.

Does it means that validation data is always fixed and taken from bottom of main dataset?

Is there any way it can be made to randomly select given fraction of data from main dataset?


texts: Data is hard to learn as a whole, easier to learn after splitting logically

texts:
I have a 3D spaceship duel simulation (without the source code).
I need to build a model that will learn the simulation's behavior.
I can run the simulation as many times as I want, randomly feeding it inputs (velocities, ranges etc) and the simulation output is whether my spaceship won or didn't.
I've used the simulation to create 3 million samples and tried to train a model using the dataset via deep learning.
I can't seem to get accuracy higher than 0.88.
So far I've tried using MLP and tuning the epochs, batch, size and depth of the net, adding features and adding derivative features (trigonometric etc).
Another thing I tried is splitting the dataset into 2 sets of data (logically partitioning to 2 scenarios) and built 2 models to solve each set independently.
Each model predicts it's own set well (~ 0.96) but combining the 2 models doesn't work well.
These scenarios are linked to the winning or losing (the initial label I'm trying to learn) so I can't make a model to choose the scenario and then use the relevant model.
Iv'e tried several things (including stacking, using decision tree to try and predict the scenario etc)
I would appreciate any leads or advice.


texts: Implementing the Dependency Sensitive CNN (DSCNN ) in Keras

texts:
I'm currently trying to implement the Dependency Sensitive Convolutional Neural Network for Modeling Documents by Rui Zhang in Keras.
For me this is the first network to implement in Keras, so I came up with some questions.

The network looks as follows:

So I'm starting with the input and my thought was that I have to input sequence of sequences, e.g. a list of sentences, where a sentence is a list of words that are word embeddings.
The sentences should be padded, so that all sentences have equal length. Also the documents should be padded, so that each document as equal number of sentences.

We will have a batch of the following type (only symbolic -> according to `fit()´):

s_i = np.array(shape=(max_sentence_length, word_embedding_dimensions))
doc_i = [s_1, s_2, s_3, ..., s_(max_sentences_per_doc)]
batch = [doc_1, doc_2, doc_3, ...]

where we_idx_ij is the word embedding index for the j-th word in the i-th sentence of a certain document.

Question 1:
Does this make sense?

I still have to think about how I will exactly include the w2v word embeddings, but with the model in Keras I would proceed as follows:

Includes:

Include the needed methods/layers and initialize symbolic constants

import numpy as np
from keras.layers import Activation, Input, Embedding, LSTM, concatenate, AveragePooling1D, MaxPool1D, Conv1D, TimeDistributed

# constants (for testing)
max_sentences_per_document = 10
max_words_per_sentence = 100
w2v_dimensions = 300
vocab_size = 20000
batch_size = 60

Model:

# preparing some shared layers
shared_embedding = Embedding(input_dim=w2v_dimensions, output_dim=vocab_size, weights=[W])  # optional,cause no training
shared_sentence_lstm = TimeDistributed(
    LSTM(input_dim=max_words_per_sentence, return_sequences=True, activation='tanh'),
    input_shape=(max_words_per_sentence, w2v_dimensions)
)
shared_sentence_lstm_2 = LSTM(activation='tanh')

# sentence modeling
sentence_inputs = [Input(shape=(batch_size, max_words_per_sentence, )) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_embedding(sentence_inputs[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_sentence_lstm(sentence_modeling[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [AveragePooling1D()(sentence_modeling[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_sentence_lstm_2(sentence_modeling[i]) for i in range(max_sentences_per_document)]

# document modeling
doc_modeling = concatenate(sentence_modeling)
doc_modeling = Conv1D(filters=100, kernel_size=[3, 4, 5], activation='relu')(doc_modeling)
doc_modeling = MaxPool1D()(doc_modeling)
doc_modeling = Activation('softmax')
doc_modeling.compile(loss='hinge', optimizer='sgd', metrics=['accuracy'])

Question 2:

Does it make sense to have the word embedding already in the batch-data or should I provide the s_i only with an index per word that relates to an index inside the word-embedding matrix? (I guess the latter would make more sense regarding the memory)

Question 3:

If I provide the word embeddings in the batch-data, I wouldn't need the embedding-layer, correct?

Question 4:

Would this network work?

Question 5:

Does this code implement the network as it is proposed in the paper?

Question 6:

Do you have any suggestions to improve the performance?

I hope I'm not too far off the track with this implementation and looking forward to your answers :-)
Thanks in advance!


texts: error when submitting machine learning project

texts:
I am trying to make a submission of a machine learning classification problem. I have a test dataset where to try my model. To submit I have to build a csv file. The prblem is that when I go building this csv file by doing:

sub = pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class}).to_csv('sub.csv')
sub.head()

I get the following error message:

AttributeError                            Traceback (most recent call last)
<ipython-input-37-f223adaf5068> in <module>
  1 sub = pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class}).to_csv('sub.csv')
----> 2 sub.head()

AttributeError: 'NoneType' object has no attribute 'head'

Can somebody please help me? Thank's in advance.


texts: Variational Autoencoder with custom loss in Keras giving "nan" loss while training

texts:
I am trying to write a simple Variational Autoencoder for a numerical dataset as opposed to images such as MNIST etc. I am basically replicating the keras blog post on this subject (with obvious modification for my dataset) but still getting this 'nan' loss while training (model.fit()). I am running out of ideas as to why this may happen. Below is my code along with the epochs where the loss is reported.
Any help would be greatly appreciated.
Thanks once again.
Code below,
train_x,test_x = train_test_split(data, test_size=0.2,random_state=4)
train_x, val_x = train_test_split(train_x, test_size=0.2, random_state=44)

trn_x_index = train_x.index
tst_x_index = test_x.index
val_x_index = val_x.index

scaler = StandardScaler().fit(train_x)
train_x = scaler.transform(train_x)
val_x = scaler.transform(val_x)
test_x = scaler.transform(test_x)

#def VAEmodel(params):
input_dim = data.shape[1]
latent_dim = 4
intermediate_dim = 32

# encoder
inputs = Input(shape=(input_dim,),name='encoder_input')
x = Dense(intermediate_dim, activation='relu')(inputs)
z_mean = Dense(latent_dim, name='z_mean')(x)
z_log_var = Dense(latent_dim, name='z_log_var')(x)

# reparametrization trick to sample latent code (z) from Q(z|x)
z = Lambda(sampling, output_shape=(latent_dim,), name='z')([z_mean, z_log_var])

# instantiate encoder model
encoder = Model(inputs, [z_mean, z_log_var, z], name='encoder')
#encoder.summary()

# decoder
latent_inputs = Input(shape=(latent_dim,), name='z_sampling')
x = Dense(intermediate_dim, activation='relu')(latent_inputs)
outputs = Dense(input_dim, activation='sigmoid')(x)
# instantiate decoder model
decoder = Model(latent_inputs, outputs, name='decoder')
#decoder.summary()

# instantiate VAE model
outputs = decoder(encoder(inputs)[2])
vae = Model(inputs, outputs, name='vae_mlp')

#reconstruction loss
reconn_loss = mse(inputs, outputs)*input_dim

#kl loss
kl_loss = 1 + z_log_var - K.square(z_mean) - K.exp(z_log_var)
kl_loss = K.sum(kl_loss, axis=-1)
kl_loss *= -0.5

vae_loss = K.mean(reconn_loss + kl_loss)

vae.add_loss(vae_loss)

vae.compile(optimizer='adam')

    #return vae, z_mean, z_log_var

history = vae.fit(train_x, epochs= 5, batch_size=128, shuffle=True).history

Here is the sampling function.
def sampling(args):
    """Reparameterization trick by sampling from an isotropic unit Gaussian.
    # Arguments
        args (tensor): mean and log of variance of Q(z|X)
    # Returns
        z (tensor): sampled latent vector
    """

    z_mean, z_log_var = args
    batch = K.shape(z_mean)[0]
    dim = K.int_shape(z_mean)[1]
    # by default, random_normal has mean = 0 and std = 1.0
    epsilon = K.random_normal(shape=(batch, dim))

    latent_z = z_mean + K.exp(0.5 * z_log_var) * epsilon
    return latent_z

Here is the output from the fit() method:
Epoch 1/5
1884/1884 [==============================] - 1s 326us/step - loss: nan
Epoch 2/5
1884/1884 [==============================] - 1s 340us/step - loss: nan
Epoch 3/5
1884/1884 [==============================] - 1s 339us/step - loss: nan
Epoch 4/5
1884/1884 [==============================] - 1s 342us/step - loss: nan
Epoch 5/5
1884/1884 [==============================] - 1s 338us/step - loss: nan


texts: How to approach a Data Science case study question?

texts:
I recently had a phone interview with a consumer tech company for a quant position. The question was basically, "imagine a facebook style social network site. Six months ago a new feature called 'mentions' was added which allows you to tag your friends with an @ sign. How would determine whether this feature was a success?"

I was a bit taken aback by how broad the question was. I first asked if the feature was given to everyone in the network or a sample, to which the interviewer responded "you decide" - meaning I could approach the analysis either way. I talked in general terms about calculating week over week usage of the feature as well as month over month. I also discussed computing a baseline metric for product interaction and then comparing the usage of the new mentions feature relative the baseline statistic. Overall I left the interview feeling quite dumb, as I have a pretty solid command of stats, but came away looking like an idiot.

Are there specific statistical procedures to test for something like this? al la A/B testing, or some kind of hypothesis test? And secondly, is there a good framework for approaching these types of open ended case study style questions in general?


texts: Track underlying observation when using GridSearchCV and make_scorer

texts:
I'm doing a GridSearchCV, and I've defined a custom function (called custom_scorer below) to optimize for.  So the setup is like this:

gs = GridSearchCV(estimator=some_classifier,
                  param_grid=some_grid,
                  cv=5,  # for concreteness
                  scoring=make_scorer(custom_scorer))

gs.fit(training_data, training_y)

This is a binary classification.  So during the grid search, for each permutation of hyperparameters, the custom score value is computed on each of the 5 left-out folds after training on the other 4 folds.

custom_scorer is a scaler-valued function with 2 inputs: an array $y$ containing ground truths (i.e., 0's and 1's), and an array $y_{pred}$ containing predicted probabilities (of being 1, the "positive" class):

def custom_scorer(y, y_pred):
    """
    (1) y contains ground truths, but only for the left-out fold
    (2) Similarly, y_pred contains predicted probabilities, but only for the left-out fold
    (3) So y, y_pred is each of length ~len(training_y)/5
    """

    return scaler_value

But suppose the scaler_value returned by custom_scorer depends not only on $y$ and $y_{pred}$, but also knowledge of which observations were assigned to the left-out fold.  If I have only $y$ and $y_{pred}$ (again: the ground truths and predicted probabilities for the left-out fold, respectively) when the custom_scorer method is called, I don't know which rows belong to this fold.  I need a way to track which rows of training_data get assigned to the left-out fold at the point when custom_scorer is called, e.g. the indices of the rows.

Any ideas on the easiest way to do this?  Please let me know if clarification is needed.  Thank you!


texts: Improve CoreNLP POS tagger and NER tagger?

texts:
The CoreNLP parts of speech tagger and name entity recognition tagger are pretty good out of the box, but I'd like to improve the accuracy further so that the overall program runs better. To explain more about accuracy -- there are situations in which the POS/NER is wrongly tagged. For instance:

"Oversaw car manufacturing" gets tagged as NNP-NN-NN

Rather than VB* or something similar, since it's a verb-like phrase (I'm not a linguist, so take this with a grain of salt).

So what's the best way to accomplish accuracy improvement?

Are there better models out there for POS/NER that can be incorporated into CoreNLP?
Should I switch to other NLP tools?
Or create training models with exception rules?


texts:
I'm doing a course on reinforcement learning, and one of our tasks is to implement an agent on the Lunar lander continuous V2 environment from openAI gym.
In order to solve the continuous problem, I used discretization for the number of actions, where the number of actions is supposed to be at least 8 (I chose 10).
I used 3 methods: Deep Q Network, Double Deep Q Network and dueling, and I tried to solve it in two configurations: one regular and one with random noise (adding a zero-mean
Gaussian noise with mean=0 and std = 0.05 to PositionX and PositionY observation of
the location of the lander.)
The results I get:
without noise:
DQN reached avg. of 234, DDQN reached avg. of 229 and 3DQN reached avg. of 256. We can see that the 3DQN
reaches much higher values in most episodes and the lowest values are still above 40.
with noise:
Total rewards per episode for DQN, DDQN, 3DQN train with random noise. We can see convergence at
around 500 episodes for DDQN and 3DQN and around 800 episodes. The 3DQN is very stable and reaches high values faster than the DQN and DDQN.
Are these results reasonable? What are the main differences between the models that can explain these results?
We also found that a network with about 2-3 hidden layers performs better than a deeper layer network.
Does that makes sense?


texts:
Only data scientist in an organization and I could really use a sounding board here.
In Phase One of a project I deployed four models and served their average as the prediction. I used Random Forest regressor, AdaBoost regressor, Gradient Boosting regressor, and XGB regressor.
I was able to capture data on the back end for each individual model as well as the served prediction. It turned out that AdaBoost was giving the best results with respect to my metric of choice: mean absolute error.
In Phase One of this project I didn't perform any feature selection at all.
Phase two of the project is underway and I am attempting to get better predictions, and it is my presumption that performing feature selection will improve model performance.
I've got few feature selection methods I will be using: intuitive selection (using my intuition as to which features may be most important) as well as some filter methods (variance threshold and correlation measures) and feature importance from decision tree.
All in all I intend on using 4 feature selection methods.
Using the data that was used to train the model and scoring with on the data for which I already have ground truth I intend to retrain each of the models using the methods mentioned above.
So now that is 4 feature selection methods for 4 algorithms, for a total of 16 trained models.
The problem I have now is how to design the experiment to see which models perform better.
What would be a reasonable way to go about this?
My intuition says to take the model fitted by each feature selection method, compute the MAE, and do a pairwise t-test against the deployed model, testing a one-sided hypothesis that the MAE for the new model is less than the MAE in the deployed model.
Should I adjust for multiple comparisons?
Should I instead look to AIC, BIC, or Mallow' C? Are those applicable here?  It seems tree-based models don't fit using maximum likelihood !


texts: Doubt on scope of text classification problem

texts:
I have a dataset that describes the sellers who are selling various brands. I need to identify the source (where did he buy those brands he is selling from) of those sellers. (Dimension of dataset 11,29,490 rows and 2 columns: Seller and brand)

eg:

I need unique instances for each seller. That is, in each row, I have to have a seller and all the information about the brands he is selling. My idea was to create make each unique brand as a feature and create a sparse dataset. However, we have almost 2 million unique brands.
Is it possible to do some text classification and come up with some text clusters and then make each cluster as a feature? I am not sure if this is a correct approach
I am stuck without an approach now.
Can anyone help me with how should I approach this problem?

Thanks in advance


texts: How does a feature learning component work

texts:
I've been studying a paper on Recommender systems using collaborative deep learning where in a hierarchical bayesian model called collaborative deep learning is developed.

Stacked denoising autoencoders are used for the feature learning of the input which is the data from citeulike.
And CF is used for learning the content information and collaborative filtering for the ratings(feedback) matrix.

Here is a link to the paper :http://www.wanghao.in/paper/KDD15_CDL.pdf


texts: Pivoting a two-column feature table in Pandas

texts:
How can I transform the following DataFrame into one with cities as rows and each cuisine as a column, and 1 or 0 as values (1 if the city has that kind of cuisine)?

I think this turns out to be a very common problem in transforming data into features for machine learning.

I am aware of the Pandas pivot_table functionality, but it asks for a value column, and in this case we don't have any.

import pandas as pd

data = {
    'city': ['NY','NY', 'SF','SF','SF'],
    'cuisine': ['Japanese', 'Chinese', 'French', 'Japanse', 'German']
}

df = pd.DataFrame(data)


texts: Setting batch size: training requires twice as much memory as validating

texts:
I am using Keras with a Tensorflow backend to train an Image Classification model on a GPU. I have read somewhere that training uses roughly twice (both forward and back props) the GPU memory of validating, so therefore the training batch size should be the half of the validation batch size.

However, on many blogs and tutorials, I see that people use the same batch size for training and validating.

Is it true that training uses twice the GPU memory, because of the forward and backward pass, or is this false?


texts: drop columns and rows in one line in pandas

texts:
I want to drop a range of rows and columns of a dataframe, I did it as follow:

df.drop(df.columns[3:], axis=1, inplace=True)
df.drop(df.index[3:], axis=0, inplace=True)

Can I do the two processes in one method instead of two?
Or is there any more sufficient way to accomplish this?


texts: Is there an R tutorial of using LSTM for multivariate time series forecasting?

texts:
There is a great blog post about how to use keras stateful LSTM in R to forecast sunspots. I applied it to financial ts data sets, and the problem I am encountering is that some of the forecasting numbers are way off to a degree that are unreasonable.

Therefore, I am wondering if there is an R tutorial of using LSTM for multivariate times series forecasting? I'd like to include variables like opening and closing price because I think that will "normalize" the forecasting values. I found a few tutorials in Python but I have limited experience with it. Maybe it's a time to pick up Python? Any advice or suggestions will be appreciated!


texts: How to do feature selection for clustering and implement it in python?

texts:
I am trying to implement k-means clustering on 60-70 features and I came across a post for feature selection technique on quora  by Julian Ramos, but I fail to understand few steps mentioned. I am also wondering if its the right method to select the best features for clustering?

These are the steps mentioned in the post :

  Sf={∅} #Set of features selected, initially empty

  Perform k-means on each of the features individually for some k.
  For each cluster measure some clustering performance metric like the Dunn's index or silhouette.
  Take the feature which gives you the best performance and add it to Sf
  Perform k-means on Sf and each of the remaining features individually
  Take the feature which gives you the best performance and add it to Sf
  If you have reached the desired number of features stop, else go back to 4

Also, how do we implement the same in python. I wish to write function for the same that selects best k and implement all the other steps.


texts: Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?

texts:
There're three parameters in the Radial Basis Function Networks (RBFN).

Centers of RBFs
Width of RBFs
Weights of RBFs

It's a fact that Weights can be easily updated using a simple Gradient Descent. My question is: Can we optimize Centers and Widths of RBFs using Gradient Descent such that approximation will tend to be better.

Any suggestion is welcome.


texts: How to add or delete features in pre-trained LSTM model then continue training by Tensorflow Keras?

texts:
I have a model like this
model = keras.Sequential([
    layers.LSTM(units=3, input_shape=(5, 3))
])
model.compile(optimizer='adam', loss='mse', metrics=['mae'])

Samples like this:
old_samples = [
    [1,2,3], // all samples has 3 features
    [4,5,6],
    [7,8,9],
    ...
]

Then I train it by model.fit(old_samples).
Now I need add some new samples, and these samples has 5 features. The first 3 features have the same meaning as the features of the samples in old_samples.
The new samples like this:
new_samples = [
    [1,2,3,11,22], // all samples has 5 features
    [4,5,6,44,55],
    [7,8,9,77,88],
    ...
]

Now I need to train the model with new_samples. How can I make the model input shape from (5,3) to (5,5) without loss the trained weights?


texts:
This is quite a general question, perhaps somewhat opinion-based.

In most papers people use boxplots to visualize a certain distribution, yet violinplots are able to give more information. Violinplots are made by performing a kernel density estimation on your distribution.

Are there objective arguments to use one over the other? Or maybe there are specific situations in which one would prefer one over the other?

An example of both can be found here:


texts: How do I produce generative neural networks?

texts:
To my understanding, a learning method is used to train a neural network on a dataset by training it's outputs.

I'm aware that the same neural network can be run backwards to generate data.

I'd like to learn how to reverse a neural network's algorithm to generate.

Any ideas?

See this video.


texts:
I have the following code for Grid search, but it only return the accuracy result using 5 folds cross-validation. Is it possible to obtain standard deviation from the 5 folds CV. How would you do that? Thanks in advance.

for i=1:numLog2c
    log2c = log2c_list(i);
    for j=1:numLog2g
        log2g = log2g_list(j);
        cmd = ['-q -v ', int2str(nFold), ' -c ', num2str(2^log2c), ' -g ', num2str(2^log2g),' ', svmCmd];
        cv = svmtrain(trainLabel, trainData, cmd);
        cvMatrix(i,j) = cv;

        if(cv >= bestcv)
            bestcv= cv; bestLog2c = log2c; bestLog2g = log2g;
        end
        fprintf('(%g, %g) %g (best c=%g, g=%g, Rate=%g)\n', log2c, log2g, cv, bestLog2c, bestLog2g, bestcv);
    end
end

disp(['CV scale1: best log2c:',num2str(bestLog2c),' best log2g:',num2str(bestLog2g),' cv-accuracy:',num2str(bestcv),'%']);


texts: Detecting Offensive Text Content in English and German

texts:
I am working on an NLP project about classifying offensive text data in social media. By offensive I especially mean threat words that one say to another.

Some examples:

"Stop doing this or else you will pay for it."
"Just wait until you see what's coming"
"I will break your legs next time I see you."

As an initial approach, I considered semantic and syntactic keyword matching. However, doing this on this very problem seemed harder because threating is an action and it is expressed in so many different ways.

My main goal is classifying text data by offensive/non-offensive text by using Machine Learning and Deep Learning algorithms. After weeks of online searching, I could not find a ready-to-use dataset. I considered manually labelling the data. However, I don't know where I should start.
What would be the best approach for making progress in this task? I also plan to do this in both English and German languages.
Also, below is a related article for fully understanding of the problem:
Deep learning for detecting inappropriate content in text


texts: Curve fitting (or other techniques) for revenue and price optimization

texts:
My question is as follows: with historical data we can model the relationship between discounts and sales (number of sold units). So we can address the question "how are sales boosted by discounts." The figure below (see also this link for an interactive graph) shows the actual historical data. As you can see, higher discounts drive the sales. When I add a lot of quadratic terms to a linear model (where y=sold units and x=discount levels), the R2 dramatically improves a lot, but I don’t think that that is the way to go. Could you advice how I can model such a relationship? Perhaps I should do something totally different?

My larger aim is predict revenue at different discount levels (see for example this graph, for which I used a technique which is loosely adopted from this blogpost). Key for such predictions, I believe, is to understand the relationship between discounts and sales (number of sold units). Hence, my question above. Any ideas are much appreciated! In case it is relevant, I work in Python myself.


texts: I don't have a background in neural networks. But, various studies has been proved that neural networks (feed forward / Recurrent) outperformed n-gram language modeling for predicting words in a sequence. But, in an application to text messaging or any text-based conversation, where the language which is most likely used will be more informal or colloquial. Can still a neural networks perform well than n-gram LM? Considering the data to be fed are the text messages (colloquial phrases). If so, please enlighten me, thanks.


texts: regarding the inclusion of missclassified data in final training

texts:
I find it best to ask my question in terms of cross-validation. Here it goes:
Suppose a binary classification problem, for which cross-validation has been applied for a certain learning algorithm. Let's say that both the CV train error and CV test error are  at 90% accuracy, indicating a good fit. Since this performance is acceptable for our problem, we combine the training and validation set into a final full dataset, and train the final model. For the final dataset, only training error is available, which suppose for our example will be 92%.
Now, for the question: Knowing that the final model has achieved a 92% accuracy, does it serve any purpose to keep the 8% of missclasified examples in the final dataset? Since these examples can't be learned, why not remove them and retrain the final model with only the 92% of the data that can be learned?
Notes

To the best of my knowledge, the  aforementioned removal of 8% in the above example is not a standard practice in modelling. Yet, I wonder what is the value of keeping examples that are not learned.
For completion, assume there is also another independent test set, to evaluate the final model.


texts:
According to various sources on the internet, treemaps are a way to visualize hierarchical, tree-like data structures, for example the tree generated by hierarchical clustering.
I looked at a lot of treemaps and I just cannot seem to understand where/how the sense of hierarchy/nestedness is conveyed. It just seems to be tiles arranged next to each other. How do I know which tile is a parent/child of which other tile(s)?


texts:
Data set contains records of short text, typically a sentence. The goal is to find duplicated records and similar records. Currently, I have tried R package 'text2vec', the glove word vectors and the similarity APIs provided by the package.

There is a smaller subset of this data which is already tagged as duplicated. Currently, I have not factored in this as part of model training. Also, using the text2vec package, the results are not great on this test set. So now I am considering RNNs which are known to perform well in text similarity.

Now, I need help in feature engineering and preparing my input layer. Sentences S1 and S2 that need to be compared differ in length (the word representations differ in their dimensions). How to normalize this difference? Should I consider bag-of-words or glove word vectors (word vectors being much superior representations)? Any inputs in this regard will help.


texts: RNADE vs MDR vs BART vs Bayesian Linear Regression

texts: I'm looking at a collection of problems where I need to forecast the probability of a continuous variable. My dependent variables are a combination of categorical and continuous. It looks like there are four primary candidate areas I should be reviewing and/or testing. Can anyone here shed any light on the relative strengths and risks associated with each of RNADE, Mixture Density Networks, Bayesian Additive Regression Trees, and standard Bayesian Linear Regression?


texts: scikit-learn pipeline strategy with both numeric and categorical columns

texts:
We are implementing an sklearn pipeline as follows (pseudo code):

feature_num = numeric_columns
feature_cat = categorical_columns

num_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_num))
                        ,('feature_num_prep', custom_num_prep)
                       ])
cat_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_cat))
                        ,('feature_cat_prep', custom_cat_prep)
                       ])
feature_pipeline = FeatureUnion(transformer_list=[
                               ('num_pipeline', num_pipeline)
                               ,('cat_pipeline', cat_pipeline)
])

Since we have in the custom_num_prep transformer the possibility to remove outliers - when we get to the feature_pipeline there is an error:

  ValueError: [blocks[0,:] has incompatible row dimensions....

The reason is obvious since the cat_pipeline has observations that the num_pipeline has removed.
The question is:
1. is there a pythonic way to sync the two dataframes within the pipelines?
2. if not what strategy is there for such situations?


texts: Testing Multi-Arm Bandits on Historical Data

texts:
Suppose I want to test a multi-arm bandit algorithm in the contextual setting on a set of historical data. For simplicity, let's assume there are only two arms A and B and suppose the rewards are binary. Furthermore, suppose I have a data set where users were shown one of the two arms and I have a record of the rewards. What would be the best approach to simulating the scenario of running the algorithm online?

I was thinking of doing the following: If the algorithm outputs A then I want to record a reward so the algorithm can learn, to do this we sample uniformly at random from the set of data where users were shown the output to obtain this reward.

I'd like to know if this approach is suitable and if anyone knows of any better ways to simulate online learning algorithms with historical data, in particular the multi-arm bandit problem?


texts: Is it beneficial to train 1 object detector for N classes vs N object detectors each for a different class?

texts:
I always wondered if training an object detector to recognize (let's say), dogs and cats performs better or worse than training 2 object detectors, one for cats only and one for dogs only.

Would this affect accuracy in any way? Are there scenarios where this approach would make sense ?


texts: Multi-label classification with nested features

texts:
I need to perform a multi-label classification. I have three features and they are nested. I am unsure how to combine this or what kind of classification algorithm would be best. Some multi level neural network as shown here seems good, but the nested features don't seem to be taken into account there.
I present the nested features (X) and labels (Y) in the two datasets below: one subject ID can have one or more features and one or more classes. Features and classes can be 'occupied' by one or more subject.
Note: I have about 100k subjects, 1k features (at the third level) and 200 classes.
data_features
       subject_id   feature1   feature2   feature3
               1          a         aa          aaa
               2          a         aa          aab
               3          a         ab          aba
               1          a         ab          abb
               2          b         ba          baa
               3          b         ba          bac
               1          b         ba          bad
               2          b         ba          bad
               3          c         ca          caa
               4          c         ca          caa
               5          c         cb          cba
               6          c         cb          cbb

data_labels
       subject_id   label1   label2   label3   label4
               1        0        1        0        0
               2        0        1        1        1
               3        0        1        1        0
               4        1        1        0        1
               5        1        0        0        0
               6        0        1        1        1
               7        0        0        0        1
               8        1        1        1        1
               9        0        0        1        1
              10        1        0        1        0
              11        0        1        0        1
              12        1        0        0        1

I am quite unsure what algorithm would combine those the best? (I am skilled in R and SAS and decent in Python, but will learn any other language that would be needed)


texts:
I need to replace a specific value in a particular column with NaN. Here is what I tried but it didn't work.

# Names of the columns we're searching for missing values
columns = ['median', 'p25th', 'p75th']

# Take a look at the dtypes
print(recent_grads.columns.dtype)

# Find how missing values are represented
print(recent_grads["median"].unique())
print(recent_grads["p25th"].unique())
print(recent_grads["p75th"].unique())

# Replace missing values with NaN
for column in columns:
    recent_grads.loc[recent_grads.columns == 'UN', column] = np.nan


texts: Genetic neural network to satisfy variable number of inputs and outputs

texts:
I have what I propose as a solution to my problem, however I haven't ever seen it mentioned in this way, so I worry that there is a valid reason not to do things this way.

I have a dataset of > 100,000 events, where each event has a winner.
I have plenty of data points, some data on the event itself, and some data on each entrant.

The number of entrants in each event is variable, and I want to build a neural network around picking a likely winner of the events.

As the number of entrants is variable, what appears to be common advice is to have enough inputs for the maximum case scenario, and 0 them out for events where there are empty slots.

This feels somewhat inelegant, and I had a slightly different idea.

I was going to have a NN where the inputs are information about the event, and information about 1 entrant. I would then have a single output (a float between 0 and 1). I would run this through, getting 1 output for each entrant in an event, then I would be left with a number of floats, equal to the number of entrants in the event. I would then select the highest value, and use the entrant that refers to as the choice for the winner.

Is there a reason I shouldn't be doing it this way? Is there a better solution I haven't yet come across?


texts: What is a good book or resource to start learning Artificial Intelligence ?


texts: Is it possible that MLP has better accuracy than CNN?

texts:
I am working on the epilepsy classification system which consumes EEG signals and in the result says if withing the certain period is a seizure or not. I take an advantage of Keras API for the sake of network training. I am giving a try at a few different neural network configurations and now I wonder is it possible that MLP is better than CNN in 1D classification in some cases? My question is not only related to EEG or epilepsy detection, but more if there is any example when MLP behaves better than CNN in the sense of accuracy. Or maybe it is mathematically proven that there is always some CNN which will beat MLP for the given classification problem.
For CNN I use keras.layers.Conv1D, for MLP keras.layers.Dense if that matters.


texts: Is Loss not a good indication of performance?

texts:
Im trying to segment 3D volumes using a 3D uNet network. Ive reached a stage where I am getting very good validation loss using CrossEntropy and BCE

idx:  0 of  53 - Validation Loss:  0.029650183394551277
idx:  5 of  53 - Validation Loss:  0.009899887256324291
idx:  10 of  53 - Validation Loss:  0.05049080401659012
idx:  15 of  53 - Validation Loss:  0.02019292116165161
idx:  20 of  53 - Validation Loss:  0.04724293574690819
idx:  25 of  53 - Validation Loss:  0.02810296043753624
idx:  30 of  53 - Validation Loss:  0.02642594277858734
idx:  35 of  53 - Validation Loss:  0.029894422739744186
idx:  40 of  53 - Validation Loss:  0.04158024489879608
idx:  45 of  53 - Validation Loss:  0.04574814811348915
idx:  50 of  53 - Validation Loss:  0.05406259000301361

I assumed my network is performing very well so i wrote a script to visualize my network outputs against their respective targets. What I get is something very different, not something that justifies this loss.
The samples are of depth 32 and I've outputted each z-plane as a single image.
Here is the target:

And the predicted output:

All samples are like this, not one of them accurately represent the target with the reported loss.. so I ask is my loss wrong? What should I look into to fix this?

Thanks


texts:
I'm trying to create an autoencoder for the anomaly detection task, but I'm noticing that even if it performs very well on the training set, it starts to stop recreating half of the test set. I tried with more than 10 models, (LSTM, ConvAE, ConvLSTM) and all of them fails to reconstruct the time series in the same point.
These are the performances on the training set. The blue part is the original time series and the red one is the time series reconstructed by the AE.

These are the performance on the training set. I don't understand why all the models stop performing from that point. Could that means that there are anomalies in that part?

EDIT: I'm updating the question with some details about my dataset and my code: I have a dataset with 30 devices, and for each one I have about 9000 values. The dataset is structured as well:
device1   device2   device3   ....   device30
 0.20      0.35      0.12              0.56
 1.20      2.10      5.75              0.16
 3.20      9.21      1.94              5.12
 5.20      4.32      0.42              9.56
 ....      ....      ....              ....
 7.20      6.21      0.20              -9.56

Since I'm following this guide, I started creating a sequence method to prepare my data for the Conv1D layer:
TIME_STEPS = 10
# Generated training sequences for use in the model.
def create_sequences(values, time_steps=TIME_STEPS):
    output = []
    for i in range(len(values) - time_steps):
        output.append(values[i: (i + time_steps)])
    return np.stack(output)

This is where I create the sequences and normalize the dataset:
# split the train/val/test set
n_features = dataset_sequences.shape[1]
X_train = dataset_sequences[0:3000, :]
X_val = dataset_sequences[3000:6000, :]
X_test = dataset_sequences[6000:9000, :]

# normalize the data
train_mean = X_train.mean()
train_std = X_train.std()
X_train = (X_train - train_mean) / train_std
X_val = (X_val - train_mean) / train_std
X_test = (X_test - train_mean) / train_std

Then, I feed my X_train with shape (3000, 10, 30) to my Conv1D autoencoder:
model = tf.keras.Sequential(
    [
        tf.keras.layers.Input(shape=(X_train.shape[1], X_train.shape[2])),
        tf.keras.layers.Conv1D(
            filters=64, kernel_size=5, padding="same", strides=1, activation="relu"),
        tf.keras.layers.Dropout(rate=0.2),
        tf.keras.layers.Conv1D(
            filters=32, kernel_size=5, padding="same", strides=1, activation="relu"),
        tf.keras.layers.MaxPooling1D(pool_size=2),
        tf.keras.layers.Conv1DTranspose(
            filters=32, kernel_size=5, padding="same", strides=1, activation="relu"),
        tf.keras.layers.Dropout(rate=0.2),
        tf.keras.layers.Conv1DTranspose(
            filters=64, kernel_size=5, padding="same", strides=1, activation="relu"),
        tf.keras.layers.UpSampling1D(size=2),
        tf.keras.layers.Conv1DTranspose(filters=30, kernel_size=5, padding="same"),
    ]
)


texts: nltk measure the accuracy of the new features

texts:
I have been playing around the NLTK algorithm for some data prediction.

Starting from this gib, I started my understanding process. However, there are some bits that don't make sense.

If I have a set of 100 features, all classified, what's the sense to split them, take 10% and build the training set on that alone?
I thought the training set should have included all the list, and accuracy is measured against the new keywords being tested against?

Any hint would be helpful.


texts: How to compute f1_score for multiclass multilabel classification

texts:
I have used one hot encoder [1,0,0][0,1,0][0,0,1] for my functional classification model.
The predicted probabilities for test data yprob = model.predict(testX) gives me :
yprob = array([[0.18120882, 0.5803128 , 0.22847839],
       [0.0101245 , 0.12861261, 0.9612609 ],
       [0.16332535, 0.4925239 , 0.35415074],
       ...,
       [0.9931931 , 0.09328955, 0.01351734],
       [0.48841736, 0.25034943, 0.16123319],
       [0.3807928, 0.42698202, 0.27493873]], dtype=float32)

I would like to compute the Accuracy, F1 score and the confusion matrix from this.
The sequential api offers a predict_classes function to do it.
yclasses = model.predict_classes(testX) and using the f1_score function of sklearn we could compute all those values.
How could I apply it to predict probabilities of test data for multiclass multilabel classification ?
My second question is to know if the highest value of each array of yprob = model.predict(testX) corresponds to the predicted class ? for example, [0.18120882, 0.5803128 , 0.22847839] is the first element in the array. The highest value is  0.5803128. Does it mean that it corresponds to the one hot encoder [0, 1, 0], so the second label because it is the second element in the array ?


texts: How to map topic to a document after topic modeling is done with LDA

texts:
Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ?
I am interested in clustering documents using unsupervised learning and segregating it into appropriate cluster.
Any link, code example, paper will greatly be appreciated.


texts: Large action space for deep reinforcement learning

texts:
I know that in normal Deep Reinforcement Learning(DRL) scenario, we learn a deep neural network to map current states to Q values. The number of the Q values (# of outputs of the neural network) is the same as the number of possible actions so that we can choose actions according to the associated Q values.

However, in this paper "Deep Reinforcement Learning with a Combinatorial Action Space for Predicting Popular Reddit Threads", the author uses states and actions as inputs. The network only outputs a single Q value (see pictures below). The $s_t$ is the state at time $t$ and $a_t^i$ is the $i^{\text{th}}$ action at time $t$ (you can ignore this but each action is a combination of the vector $c_t^1$ to $c_t^3$ ). $Q_t$ is just the $Q$ at time $t$ for action $i$. I am wondering why only a single Q value is learned? If this is the case, how do we determine the action for the next step?


texts: Why is the Naive Bayes classifier of sklearn faster than sklearns SVM?

texts: I've used scikit-learn in Python to compare results of naive Bayes and SVM. I've found that naive Bayes is quicker than SVM. Could anyone shed some light on reasons for such finding?


texts:
I trained an infection segmentation models on a large dataset of CT scans, and want to extend it to other datasets to show the ability of the model to generalize. What I found though, is that CT scans look very different. For example,

CT scan from the first datasets,

Image from another dataset

I tried three things:

Global mean/standard deviation (subtract from each image the same vector $m$ and divide by $s$),
Local mean/standard deviation (from each image subtract its $m$ and divide by its $s$, os that every image is normally distributed with mean $m$ and std $s$,
Gamma correction ($\gamma=0.01$), see the image at the bottom.

None of which worked. My question is: is it possible to 'normalize' other CT scans, 'normalize' in a sense make them more similar to the one on which I trained the model?


texts: Is the magnitude of a word vector correlated with the frequency of the word in a text?

texts:
In order to find the similarity between words, cosine similarity seems to be the most common measure to use. In a conversation that I had about this topic, someone mentioned that words that mean more or less the same, such as 'big', 'huge' or 'gigantic' would have a high similarity (small angle), yet due to the difference in frequency of the words in the text, the magnitude of the vector might be different. It was said that the vectors associated to words that are common, would have a higher magnitude.
Is is true that the magnitude of a word vector increases as its associated word appears more frequently in a text? And if so, how can this be explained?


texts: What is the use of the np.ndarray() function and how is it used?

texts: I just started Data Science and I want to know what the np.ndarray() function's use is, and in which scenario(s) it is used in. Thanks for the help.


texts: Feature selection with many Time stamp data and Model classification

texts:
I’ve a data set with many Timestamp features like:

Employee duty report time
Employee duty release time
Employee work start time
Employee work end time
etc.

We currently have a Java application that uses this data to tell if it is legal for an employee to work on a given day or not. The work is not continuous, some days employee work only 2 hours and other days they work straight 8 hours based on various shift assignment rules.

We feed 15 days of data to a Java Rule Engine to find out if an employee is legal to work or not. The Java Rule Engine uses many rules like if an employee works 8 hours in a day then the employee must take minimum 12 hours of rest before starting to work again. From the Java Rule Engine I collected all these data into two sets one with legal to work and other is not legal.

Now I want to use logistic regression to feed the data to classify if an employee is legal or not. What is the best approach to select the various timestamp features? Can this data set can be considered as a “time series” model?


texts: What's the proper Word2vec model to get pre-trained word embedding for a classification task?

texts: I have to use a neural network to classify whether some reviews of hotels are deceptive or truthful. I also have to use pre-trained word embeddings to fed the neural networks. So I can use Word2vec to get the word vectors from a way larger dataset of hotels reviews. However, Word2vec gives the possibility to use continuous bag-of-words and continuous skip-gram models for this task. Which one would be generally better for this specific task?


texts: What do I do with an array of log probabilities? Doc2vec

texts:
I am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I run the doc2vec.score() function on a document collection per document.

According to the Gensim api, the method returns an array of log probabilities, with values ranging from zero to negative infinity per document

My goal is to sort test documents of different lengths based on their probability to the model. I am new to log probabilities, and wasn’t sure if I need to sum, average, or grab the max value for my purposes.

How do I make use of the log probabilities?


texts: I was training a Deep Neural Network using transfer learning and the loss value as of now is close to 1.7. I am using a dataset of 31,000 images and I do apply data augmentation techniques to help the model generalize better. However, I had confusion regarding epochs and loss values. Should I be training the model for say 50 epochs for good accuracy or would it be better if I trained until the loss value is as low as possible which will be very close to zero. Which one is recommended to get the best accuracy? Will more epochs cause the model to perform better than the lowest loss value? Any help would be greatly appreciated. Thanks in advance!


texts: Intuition behind understanding eigenvectors and Machine Learning

texts:
I am struggling to understand linear algebra application in machine learning, and I am not able to answer the following question.
Is the model learned in Machine Learning  the eigenvector of the features matrix?


texts: What are advantages of oversampling over changing threshold for unbalanced classes?

texts:
Let's say that I have unbalanced data set that has two classes, and I am using Random Forest to make my predictions. Random forest will be biased towards the majority class, which will cause low recall and high precision on minority class predictions that I am interested in, and it might not be desirable.

One approach is to plot recall, precision and f1 score on the y-axis and threshold on the axis, and use this plot to select the proper threshold.

Another approach is to use oversampling, let's say SMOTE, for the minority class.

My questions is: What advantages would oversampling approach offer?

So far I only see disadvantages:

1) We have to be very careful, that our minority class is in one cluster, otherwise SMOTE will put points between the clusters, which is not desired at all. To avoid that, we should create different labels for different clusters, which is much more complicated.

2) We are creating extra data, that increases run time and memory usage.

3) Even after we have done oversampling, we still might not have ideal recall, threshold balance, and it feels redundant to still have to adjust the threshold.

I know that oversampling is a commonly used technique, what am I missing?


texts:
I have a lasgane code. I want to create the same network using caffe. I could conver the network.But i need help with the hyperparameters in lasagne. The hyperparameters  look like:

lr = 1e-2
weight_decay = 1e-5

prediction = lasagne.layers.get_output(net['out'])
loss = T.mean(lasagne.objectives.squared_error(prediction, target_var))

weightsl2 = lasagne.regularization.regularize_network_params(net['out'], lasagne.regularization.l2)
loss += weight_decay * weightsl2

How do i perform the L2 regularization part in caffe? Relevant parts from my solver.prototxt is as below:

base_lr: 0.01
lr_policy: "fixed"
weight_decay: 0.00001
regularization_type: "L2"
stepsize: 300
gamma: 0.1
max_iter: 2000
momentum: 0.9


texts: Where can I find datasets suitable for an undergraduate research project?

texts:
I have tried Kaggle but couldn’t find anything suitable. Basically I want to do a machine learning or deep learning project. Ideally it would have something to do with computer forensics or security. I have found a good dataset here.
But its very large and hard on my system I.e. a MacBook with i5 processor and 8 G/bs of RAM. I am also considering doing Convolutional Neural Network (CNN) but I can’t seem to find any datasets anywhere other than what’s in the standard built in library's of Tensorflow 2 etc. Any help would be appreciated. Also with the link I gave above I can only get the datasets from them if I look up the paper on IEEE I can only get the paper not the code used is there away to view the original authors code in their paper they stated they used Machine Learning to predict values I was thinking I could use deep learning in my version but id like to review their work first.


texts: Solving the dual problem of boosting using column generation

texts:
In our book there is boosting algorithm using column generation method (Dantzig-Wolfe decomposition) to solve the dual problem.
So lets say we have want to solve the following primal linear problem based on hinge loss:
$$\min_{w,\xi} \Sigma_{m=1}^Mw_m + C\Sigma_{i=1}^N\xi_i$$
$$ y_i\Sigma_{m=1}^MH_{im}w_m+\xi_i\geq 1 \text{ ,} i\in[N]$$
$$w,\xi \geq 0,$$
where $H_{im}:=\phi(x_i),i\in[N],m\in[M]$ and $C>0$ is a weighting factor.
Now the dual is:
$$\max_z \mathbb{1}^Tz$$
$$\Sigma_{i=1}^N y_i H_{im} z_i \leq 1 \text{ }, m\in[M]$$
$$0\leq z\leq C\mathbb{1}$$
Now in our book, it stated:

The base learniners $\phi(x)$ can be obtained dynamically, by solving the dual problem with optimal solution $z^*\in\mathbb{R}_+^N$. For a given subset of base functions, we need to check whether there exists another function $\phi_m(x)$ such that the corresponding dual constraint is violated, i.e.,
$$\Sigma_i=1^N y_iH_{im}z_i^* >1$$
Thus the left hand side has to be maximized, which can be handled by calling  a base learner with weights $z^*$ trying to maximize the weighted number of points that are correctly classified. If the value is larger than 1, $\phi_m(x)$ is added to the set of base learners ant the process is repeated. Note that this process will terminate, since there are only finitely many vectors $H_{. m}\in{-1,+1}^N$.

For me it is totally unclear why we are checking for functions $\phi_m(x)$ which violate the constraint. I mean we want to solve the dual problem and if so the constraints forces us to use functions which classify the points incorrectly. Then we had $yH_m<0$ and the constraint would be active and the dual problem could be solved. I don't know how the problem is going to be solved if we only use functions which violate the constraint


texts: How to extract insights from the given data?

texts: Ok, I have this data with 3 columns, unique id, raw text, and review text. My task is play with the dataset and find meaningful insights from it. Raw text is in plain English but review text is in another language. I have no idea how to proceed with the dataset. Even after I clean the data from the raw text, what should I do with review one because it is in another language. Which text analysis should I do and how can I implement it on the dataset?


texts: How to measure the similarity between two images?

texts:
I have two group images for cat and dog. And each group contain 2000 images for cat and dog respectively.

My goal is try to cluster the images by using k-means.

Assume image1 is x, and image2 is y.Here we need to measure the similarity between any two images. what is the common way to measure between two images?


texts: What are some modern algorithms\analysis that use Taylor Series or Lagrange multipliers?

texts:
What are some modern machine learning algorithms\analysis that use Taylor Series or Lagrange multipliers?

I spent a good chunk of my weekend reading about SQP, aka SLSQP. SciPy has an implementation, but when I looked at the source code I realized it was all in Fortran. Is there is a python implementation somewhere? I found this textbook excerpt: https://www.math.uh.edu/~rohop/fall_06/Chapter4.pdf. This resource was pretty good at explaining 1) the Taylor Series approximation 2) the Lagrange equation (multiplier variables on the constraint function). I started getting lost when the book went into approximating the Hessian of the Lagrange function. I have no idea how to actually implement this algorithm in python.

I know that XGBoost uses a quadratic (Taylor Series) term in the approximation of the loss before making a split. What some other examples? Is it worth learning about or should I stick to learning LP\MIP\GA\POS?

Taylor Series approximation of the optimal point x at point k: $ f(x) = f(k) + f^{'}(k)*(x-k) + 0.5*(x-k)^Tf^{''}(k)(x-k)$

And the Lagrange equation where the gradients of the objective and the constraints are equation to zero:
$\nabla L = \nabla f(x) + \lambda * \nabla h(x) =0$.


texts: How do scientists come up with the correct Hidden Markov Model parameters and topology to use?

texts:
I understand how a Hidden Markov Model is used in genomic sequences, such as finding a gene. But I don't understand how to come up with a particular Markov model. I mean, how many states should the model have? How many possible transitions? Should the model have a loop?

How would they know that their model is optimal?

Do they imagine, say 10 different models, benchmark those 10 models and publish the best one?


texts: Categorization of approaches to deal with imbalanced classes

texts:
What is the best way to categorize the approaches which have been developed to deal with imbalance class problem?

This article categorizes them into:

Preprocessing: includes oversampling, undersampling and hybrid
methods,
Cost-sensitive learning: includes direct methods and
meta-learning which the latter further divides into thresholding and
sampling,
Ensemble techniques: includes cost-sensitive ensembles and data
preprocessing in conjunction with ensemble learning.

The second classification:

Data Pre-processing: includes distribution change and weighting the
data space. One-class learning is considered as distribution change.
Special-purpose Learning Methods
Prediction Post-processing: includes threshold method and cost-sensitive post-processing
Hybrid Methods:

The third article:

Data-level methods
Algorithm-level methods
Hybrid methods

The last classification also considers output adjustment as an independent approach.

Thanks in advance.


texts: In DQN, what is the real reason we don't backpropagate through the target network?

texts:
Actually, if we are to backpropagate through the target network, there is no use for the target network anymore.

Let's say we don't use any target network. We hence enforce only the "temporal difference" allowing gradients to flow through both ways i.e. q(s,) and q(s',).

This completes the gradient and prevents known instabilities of DQN. While this training is much more stable, it is not widely used.

My question is why not? Where is the catch?


texts: Feature engineering for time series (audio signals)

texts:
I have a task to perform classification of audio signals using any suitable algorimth. After some research I found out, that CNN from this paper shows promising results. However, it still needs to be improved. Current data is scaled along each sample (that is each audio signals has zero mean and the same variance(i.e. 1)), and that leads to the loss of information about amplitude and variance of signal. Is there a way to append this information(i.e. if I compute it separately before scaling) to current signal to make CNN use it?

Thanks in advance for any help!


texts: Which features are causing a class to be classified correctly or incorrectly?

texts:
I am doing a project that involves training and testing different algorithms to predict a developer's profile type (Frontend, Fullstack, QA, ML, etc.) using that developer's skills (AWS, Selenium, Hadoop, Java, etc.).
I am looking more in depth at correct and incorrect classifications made by the Logistic Regression (OVR) algorithm to understand whether the presence or absence of particular skills cause this to happen. With my example, I use Fullstack-Fullstack as a correct classification and Fullstack-ML as incorrect.
***# returns skills present in correct classifications which are not present in incorrect classifications***
correct_classification_skills.difference(incorrect_classification_skills)

Returned:
*['ASP.NET', 'AWS', 'Adobe Photoshop', 'Agile', 'Ajax',
       'Amazon Web Services (AWS)', 'Angular', 'AngularJS', 'Apache', 'Azure',
       'Backbone.js', 'Blockchain', 'Bootstrap', 'C++', 'CSS', 'Cassandra',
       'Docker', 'Drupal', 'Eclipse', 'Elixir', 'Express.js', 'Flask',
       'Full-stack', 'Git', 'GitHub', 'Google Analytics', 'HTML', 'HTML5',
       'JIRA', 'JSON', 'Java', 'JavaScript', 'MSSQL', 'MacOS', 'Meteor',
       'Microsoft', 'Mobile', 'MySQL', 'Nginx', 'Node.js',
       'Object-oriented Programming (OOP)', 'PHP', 'PostgreSQL', 'Python',
       'REST APIs', 'React', 'React Native', 'React.js', 'Redux', 'Ruby',
       'SCSS', 'Shopify', 'Swift', 'Ubuntu', 'Visual Studio', 'WordPress']*

***# returns skills present in incorrect classifications which are not present in correct classifications***
incorrect_classification_skills.difference(correct_classification_skills)

Returned:
*['SQL']*

My question is: is one more important than the other? In other words, is it more important to look at the skills absent or present?
Thank you so much for the help!


texts: How can I plot line plots based on an input python dataframe?

texts:
I need help to create a plot using 3 different columns from a dataframe. my dataframe looks like this:

index      CMPGN_NM  COST_SUM    SUMRY_DT

2   GSA_SMB_SMB_Generic_BMM      8985  2018-05-17

3   GSA_SMB_SMB_Generic_BMM      7456  2018-05-18

4   GSA_SMB_SMB_Generic_BMM      5761  2018-05-19

10  GSA_SMB_SMB_Generic_BMM      4251  2018-05-20

5   GSA_SMB_SMB_Generic_BMM     10521  2018-05-21

6   GSA_SMB_SMB_Generic_BMM     10216  2018-05-22

7   GSA_SMB_SMB_Generic_BMM     11023  2018-05-23

9   GSA_SMB_SMB_Generic_BMM     11242  2018-05-24

8   GSA_SMB_SMB_Generic_BMM      8817  2018-05-25

1   GSA_SMB_SMB_Generic_BMM      6937  2018-05-26

0   GSA_SMB_SMB_Generic_BMM      4581  2018-05-27

I would like the output to look like  the graph as below


texts: What are the most suitable machine learning algorithms according to type of data?

texts:
I am beginner to data science. I found that some machine learning algorithms perform better, when given particular kind of data(ie - numerical, categorical, text, graphical).
I searched about this topic on the web, but no luck.

I would like to know what kind of data performs best according to given machine learning algorithm?

It is better to explain briefly why certain kinds of data suitable for certain machine learning algorithms?

Hope answer to this question will help beginners in data science.

Update: It is better if you can explain what are the types of data most suitable for following algorithms.
Naive Bayes, SVM, Regression, K-Means, Deep Neural networks.


texts:
I'm wondering about the difference or the application of the different types of rescaling data.

So far, I'm aware that standardization assumes the data has a gaussian distribution. So if this is the case we should standardize and get values in normal distribution N~(0,1).

If our model does not has assumptions about the data distribution (like FCN or RandomForestClassifiers,...) and we don't know about the data distribution or it is just not gaussian distributed we should normalize the data. But here is my point, there are several methods to normalize e.g. using L2/L1-Norm of the vector (this is how tensorflow has implemented their standard normalization method) or using MinMaxScaling.

So when is normalization using either L1 or L2 norm recommended and when is MinMaxScaling the right choice?


texts: Inconsistency on A. Graves' original Connectionist Temporal Classification (CTC) paper?

texts:
Here's the link to the paper.

For the forward-backward algorithm, they introduce $\alpha_t(s)$ as a definition in eq. (5).
Then they give a recursive formula for it (eq. (6)), and its initialization. However, the two formulae are not equivalent.

For example, let's say we have a labelling $\mathbf{l}=[\text{c},\text{a},\text{t}]$.

$\alpha_1(1)$ using eq. (5):

$\mathcal{B}(\pi_1)=l_1=\text{c}$, so $\pi: \pi_1=\text{c}$ is the only possible path. Therefore $\alpha_1(1)=y_{l_1}^1=y_{\text{c}}^1$ and not $y_\varepsilon^1$ as they state in the initialization (where $\varepsilon$ is the blank label).

Another example. $\alpha_2(1)$ using eq. (5):

$\mathcal{B}(\pi_{1:2})=l_1=\text{c}$, so the possible paths are $\pi: \pi_{1:2}\in\{[\text{c},\text{c}],[\text{c},\varepsilon],[\varepsilon,\text{c}]\}$. Therefore $\alpha_2(1)=y_{\text{c}}^1y_{\text{c}}^2+y_{\text{c}}^1y_{\varepsilon}^2+y_{\varepsilon}^1y_{\text{c}}^2$. However, using equation (6) we get a different result:

We have $l'_1=\varepsilon$ so we are in the first case. Then $\alpha_2(1)=\bar{\alpha}_2(1)y_{l'_1}^2=(\alpha_1(1)+\alpha_1(0))y_\varepsilon^2=(y_\varepsilon^1+0)y_\varepsilon^2=y_\varepsilon^1y_\varepsilon^2$.

I tried many other cases as well, and it seems like, the way they are written, the two formulae are completely different. Am I missing something?


texts:
I'm involved in a ship transit time prediction project. Is about prediction the time that a ship's cargo takes to go from port A to B in order to contract the fastest carrier company and tell the client how long it going to take, because a several times a the carrier company takes more or less (most of the times more).
We have historical database with predictive variables and stuff, the point is if the real transit time of cargo is 10 days is worse to the model estimate 9 days than eleven days, even that the absolute error is 1.
So a need a custom metric to punish positive errors more than negative ones and punish absolute large errors as well, calling error = actual - predicted.
I thought in using a weighted mean squared error, but I don't know how to balance the weights. Do you guys may help me? Do you know some other metrics that meet that requirements?
Thanks very much!


texts: What knoweldge framework supports this type of query?

texts:
I have knowledge that is organized as in the example below, where items or nodes can belong to multiple hierarchies and can have arbitrary numbers of children and parents:

sports
    golf
    baseball
    tennis
equipment
    golf
        clubs
            club1
    baseball
        bats
            bat1
    tennis
        racket
            racket1

I need to be able to run queries like "Show me all sports that use clubs or rackets, and their associated equipment."  The queries do not need to be posed in natural language, but the results must be displayed in a hierarchy:

sports
    golf
        club1
    tennis
        racket1

I'm going crazy trying to figure out what I need.  It doesn't seem to quite be a relational database, an ontology, or a taxonomy.  Or maybe I don't quite understand how to use one or more of those to do what I need.  What standard approach is most similar to what I'm trying to do?

As always with stackexchange questions, I'm utterly terrified of asking a question that doesn't fit the exact requirements of a particular forum.  I tried to make it as specific and answerable as possible.  I BEG YOU PLEASE, do not delete or lock this question without at least explaining why it is unacceptable or offering a suggestion for making it acceptable.  Thank you!


texts: Quantifying the imbalanceness of a dataset

texts:
after looking a lot in the literature there is really a lot of how to work with imbalanced dataset but so far I can not find a

definition of a imbalanced
metric that quantifies how much imbalanced is dataset A compared to dataset B.

I have even tried to define one such metric myself but I fail to capture with a single metric the following two cases:

One class has all the measurements
Measurements are more balanced but one or two classes have very low measurement or even zero.

Can you give me some more information from the literature that perhaps I was not able to find out or at least try to discuss the issue of definition of imbalanceness .

Thanks a lot
Alex


texts: Grouping high dimension Y-space to lower dimensions

texts:
I have a ML problem with 300 variables to predict, Meaning I have a multi-label binary classification problem with a Y space of 300 and with about 2000 rows only.
Thus we assume that there are 2^300 permutations. However, it is not so.
Let me explain with an example.
X1,....X400 | Y1,Y2,Y3...........Y299,Y300
'healthy',..'sad'| 0,1..1,0
'unhealthy'...'sad'|1,0,....0
However when i aggregate across the rows,there are maybe about 100 combinations far less than 2^300 combinations such as this.
so Y1,Y2,Y100 => C1 OR Y33,Y44,Y291,Y299,Y300 =>C2 ... till C100
Thus, I can change it to a multi-class problem to predict these 100 classes instead.
However, there are a lot of classes with count of 1 or 2. More than 50%. Thus, this approach doesnt work so well.
Is there an algorithmic way to still do the multi-label but by grouping the variables in the most optimal way like from Y space to Z space with dimension 5 for example.
{Y1,Y10,Y222,Y232}=> Z1 can be grouped together
{Y2...} => Z2 can be grouped together
Thus,we can predict on the Z space instead. At this point, I am not sure whether the Y variables need to be exclusive to one Z group or not.
But I am open to all suggestions at this moment ..
Cheers


texts:
Unique ID        Student Name      BMI       LOW BMI        HIGH BMI
HYD_61601_1200  Akshay Kumar Kale   17.0    NON-UNDER BMI   CANCER
YDD_62329_1008  MAMATHA  ERRA      12.9     NON-UNDER BMI   CANCER
HYD_61601_1000  SURESH BATTU       21.49    NON-UNDER BMI   CANCER
HYD_61601_1021  Vishal  Kavali     13.69    NON-UNDER BMI   CANCER
HYD_61601_1212  DIVYATEJA  M       15.5     NON-UNDER BMI   CANCER
MBNR_61413_2003 Aishaarya P        13.39    NON-UNDER BMI   CANCER
MBNR_61413_1230 SUGUNA             21.5     NON-UNDER BMI   CANCER

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_csv('HB.csv')
dataset.head()
dataset['HB']<12.5
dataset['LOW HB'] = dataset['HB'] < 12.5
dataset['HB']>12.5
dataset['HIGH HB'] = dataset['HB'] > 12.5
temp = {True:'ANEMIA', False:'NON-ANEMIA'}
dataset['LOW HB'] = dataset['LOW HB'].map(temp)
temp = {True:'CANCER', False:'NON-CANCAER'}
dataset['HIGH HB'] = dataset['HIGH HB'].map(temp)

above data and code i just want to know whether i can assign multiple values r not(for example, i want assign to cancer new values like stage 1, stage 2, final stage) if it possible can any suggest me code r condition to be applied


texts: Table function output and order of arguments

texts: I have a silly question. Below is the output of a logistic regression analysis I did. I notice that when I switch the order of the arguments I put in the table function in R that it also switch the false positives and the false negatives values but it did not switch the the location of the Female and Male rows and columns. This to me seems like it could really affect the interpretation if the false positives/negatives can change just from the order the arguments are put into the table. I need to know which order output is correct and the proper order for placing arguments in the table function


texts: What magnitude to use for sample_weight in keras.fit() for skewed datasets?

texts:
I've got a skewed dataset, but I want to keep it that way: don't want to log-scale/normalize, instead I want the network to generalize to the tail, but predict the outliers accurately. I'm using tf.keras.Model.fit(sample_weight=?). My hunch was to use total_count/count(outliers) for the outlier sample_weights, per code I found online:
from sklearn.utils import class_weight
weights = np.ones(len(y))
weights[outliers_mask] = 2
cw = class_weight.compute_sample_weight('balanced', (weights != 1))
sw = cw.max() / cw.min()
weights[outliers_mask] = sw
print('sample weight', sw)
# -> 2900

model.fix(x, y, sample_weight=weights)

But that ends up giving very poor results; it generalizes to the outliers. Instead if I set it between 2->50 (still experimenting) I get good results. I suppose 2.9k seemed outrageous to me, and 2-50 seems better; but I guess I don't know how to think about sample_weight?


texts: How to predict tf-Idf vector with Neural Network?

texts:
I designed a neural network in Torch for prediction of articles. I want to predict, which article will be read by user based on your reading history.

Input to the network is tf-idf vector of article and another information about user.
Output from the network should be tf-idf vector of next article, which user read.

Tf-idf vector is normalized by SoftMax function.

I know the best solution would be, if a output was a representation of classes but there are a lot of articles and by that there will be a lot of classes. So I represent articles with tf-idf vector on the output.

I tried train with DistKLDivCriterion ( Kullback–Leibler divergence ) but it didn't converge. How to design a network for prediction tf-idf vectors? Which criterion should I use?

tmodel = nn.Sequential()
tmodel:add(nn.Sequencer(nn.FastLSTM(537, 300, rho )))
tmodel:add(nn.Sequencer(nn.Dropout(0,5)))
tmodel:add(nn.Sequencer(nn.Linear(300,527)))
tmodel:add(nn.Sequencer(nn.LogSoftMax(527)))

criterion = nn.SequencerCriterion(nn.DistKLDivCriterion())


texts: Guidelines for selecting an optimizer for training neural networks

texts:
I have been using neural networks for a while now. However, one thing that I constantly struggle with is the selection of an optimizer for training the network (using backprop). What I usually do is just start with one (e.g. standard SGD) and then try other others pretty much randomly. I was wondering if there's a better (and less random) approach to finding a good optimizer, e.g. from this list:

SGD (with or without momentum)
AdaDelta
AdaGrad
RMSProp
Adam

In particular, I am interested if there's some theoretical justification for picking one over another given the training data has some property, e.g. it being sparse. I would also imagine that some optimizers work better than others in specific domains, e.g. when training convolutional networks vs. feed-forward networks or classification vs. regression.

If any of you have developed some strategy and/or intuition on how you pick optimizers, I'd be greatly interested in hearing it. Furthermore, if there's some work that provides theoretical justification for picking one over another, that would be even better.


texts: Is there any package in python similar to darch in R?

texts: I want to work in Deep Belief Network using Python. Is there any package available in python similar to darch in R.


texts: when my RF log_loss socre is different then submit score?

texts:
I train the RF and I sumbit to the data competition
but my train score is so much different then their submit socre
is there way to decrease this gap?
Here is my code with RF

%%time
# Our data is already scaled we should split our training and test sets
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss
def under_over_test(X,y):
    RFC_METRIC = 'gini'  #metric used for RandomForrestClassifier
    #RFC_METRIC= 'multi_logloss'
    NUM_ESTIMATORS = 100 #number of estimators used for RandomForrestClassifier
    NO_JOBS = 2018 #number of parallel jobs used for RandomForrestClassifier
    RANDOM_STATE = 42
    MAX_ROUNDS = 2000 #lgb iterations
    EARLY_STOP = 50 #lgb early stop
    OPT_ROUNDS = 1000  #To be adjusted based on best validation rounds
    VERBOSE_EVAL = 50 #Print out metric result
    clf = RandomForestClassifier(n_jobs=NO_JOBS,
                             random_state=RANDOM_STATE,
                             criterion=RFC_METRIC,
                             n_estimators=NUM_ESTIMATORS,
                             verbose=True
                            )

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    clf.fit(X_train, y_train)
    clf_probs = clf.predict_proba(X_test)
    score_origin = log_loss(y_test, clf_probs)


texts: With NN Model data preprocessing, is One Hot Encoding WITH PCA a good or bad idea?

texts: With Tabular Data (containing both continuous and categorical data) preprocessing, does the One Hot Encoding of Categorical Features help or hinder the effectiveness of PCA prior to Neural Network model training?  With a traditional ML model (e.g. Random Forest, KNN, Perceptron, etc.), my understanding is that One Hot Encoding would be necessary, otherwise the model trainer (typically sklearn) would assume some relationship in categorical data, that doesn’t necessarily exist.  In the case of a Neural Network, however, I don’t know if this would be the case.  I also don’t know how PCA will handle categorical data, hence the question.  Can anyone shed some light on this?  Thank You.


texts: Tattoo Image Recognition - Should I Crop Training Data Background

texts:
I am trying to train a neural network to detect objects within a tattoo. I couldn't find any existing labeled dataset so I need to manually create and label my own. I only understand the basics of machine learning and was going to start by using TensorFlow's Object Detection (https://github.com/tensorflow/models/tree/master/research/object_detection)

For my labeling I am stuck between two options. Using https://scontent-ort2-1.cdninstagram.com/vp/85794872ffd8b8166b1dd559fcf718b0/5C90FA77/t51.2885-15/e15/11326797_987161747982140_1156040219_n.jpg as an example:

Approach 1:
Create a simple bounding box around each object in the tattoo: Multiple boxes: 1 around the rose, the clock and the words. The box for the clock however may bleed into the roses, if the tattoo was more complicated and compact it would bleed a lot (Bleed as in parts of the rose are in the clock's bounding box).

Approach 2:
Cut out each object exactly so there is no bleeding and every object is exactly what it is and fill the negative space with black.

I understand that background noise is important. When the NN is training against many images it detects that the background isn't significant and can pick up the features of just object being detected, however, since all my images are tattoos which are on people's bodies the background for everything is just skin tone. I am afraid if I don't have an exact depiction of the tattoo object it will not train accurately especially since every tattoo is different.

I am not sure which approach to take or if I should go a different directions. Thanks


texts:
how to assign back categorical variables to train and test data after training and testing using inverse_transform? Like training and testing, data will have encoded numerical values. So, how to assign back categorical values to those variables to train and test dataset after training and testing?
Please help me with this.


texts: How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

texts:
Let's say our data is discrete-valued and belongs to one of $K$ classes.
The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \prod_{k = 1}^{K}\mu_{k}^{x_{k}}$ where x is a one-hot vector given as $\textbf{x} = [x_{1} x_{2} ... x_{K}]^{T}$ and $\boldsymbol{\mu} = [\mu_{1} ... \mu_{K} ]^{T}$ are the parameters.
Suppose $D = \{\mathbf{x}_{1}, \text{ } \mathbf{x}_{2}, \text{ } ... ,\text{ }\mathbf{x}_{N}\}$ is our data.

The log likelihood is:
$\log p(D|\boldsymbol{\mu}) = \sum_{k = 1}^{K} m_{k} \log{\mu_{k}}$
where $m_{k} = \sum_{n = 1}^{N} x_{nk}$
To get the MLE solution, we have to solve the following optimization problem:
$\max_{\boldsymbol{\mu}} \sum_{k = 1}^{K} m_{k} \log{\mu_{k}} \hskip 1em \text{such that} \hskip 1em \mu_{k} \geq 0, \hskip  0.5em \sum_{k = 1}^{K} \mu_{k} = 1$
To solve this we write the following Lagrangian.
$L(\boldsymbol{\mu}, \mathbf{u}, v) = \sum_{k = 1}^{K} m_{k} \log{\mu_{k}} - \sum_{k = 1}^{K} u_{k}\mu_{k} + v\left( \sum_{k = 1}^{K}\mu_{k} - 1\right)$ \
The primal problem formulation is then
$\boldsymbol{\hat{\mu}} = \inf_{\boldsymbol{\mu}} \sup_{u_{k} \geq 0, v} L(\boldsymbol{\mu}, \mathbf{u}, v)$
I have no idea how to proceed further. Have no clue how to solve the primal problem.


texts: separate decision tree for categorical feature values

texts: Given either, different decision trees each based on a particular feature value (like separate models for each male and female) or a single decision tree, should both give the same result?


texts: Which Algorithm or combination of Algorithms to use to develop supervised Video Event detection?

texts: I have to develop a video event detection tool in ticketing counter.The tool must take photographs of persons who jumb over gate without taking tickets.I have a set of videos in which people jumbing over gate.So with that data in hand How can i implement a Video event detection tool?I am new to video analytics.I don't know where to start.Where can i find some good tutorial about supervised video event detection.?


texts: Temporal outlier Analysis on sensor data

texts:
I am working to find anomaly/outliers in sensor data using unsupervised machine learning (without training dataset). I have around 20000 samples taken per minute of various sensors. I just need to detect the outliers(point, collective and contextual). I have been through many articles but not getting the right link.

clustering is meaningless and suppose one reading (500) during the day is normal but during night it is an outlier.

I am doing in python, already done with preprocessing of data using Knn Imputation.

can someone tell me what should I do next, all the algorithms I find have some or the the other flaws!


texts: Problem with x axis on swarmplot and added line with seaborn

texts:
I'm trying to add custom lines to seaborn swarmplot (based on this answer) and it seems that x axis is wrongly scaled.

Data preparation:

import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris()

X_full = pd.DataFrame(data = iris["data"], columns = iris["feature_names"])
y_full = pd.Series(data = iris["target"])

# selecting two classes and two features
y = y_full[(y_full == 0) | (y_full == 1)]
X = X_full[["sepal width (cm)", "petal width (cm)"]].loc[y.index]

and actual plot:

import seaborn as sns

grid = sns.JointGrid(x = X["sepal width (cm)"], y = X["petal width (cm)"])
grid = grid.plot_joint(sns.swarmplot, hue = y)

xmin = X["sepal width (cm)"].min()
xmax = X["sepal width (cm)"].max()

ymin = X["petal width (cm)"].min()
ymax = X["petal width (cm)"].max()

grid.ax_joint.plot([1, 2, 3], [1, 2, 3], 'b-', linewidth = 2)

grid.ax_joint.plot([xmin, xmax], [ymin, ymax] )

Result:

Thin line should go from (xmin, ymin) to (xmax,ymax). On the y axis it looks correct, but x axis is wrong.

How to fix plot so coordinates will be plotted correctly?