texts: Identifying top predictors from a mix of categorical and ordinal data

texts:
I have a dataset with 261 predictors scraped from a larger set of survey questions. 224 have values which are in a range of scale (some 1-10, some 1-4, some simply binary, all using 0 where no value is given), and the rest are unordered categories.

I'm trying to perform classification using these predictors and identify the top n predictors. Am thinking of the following approach:

convert the 224 ordered predictors into numeric, centered, and scaled.
Run separate modeling (I use caret from R): one for using the numeric predictors, another using the  remaining 37 categorical predictors (both cross-validated within each modeling exercise).
Choose the respective best-fitting models modelN and modelC for the numeric and categorical predictors.
Choose top n (say 10) predictors from model N and model C.
Combine them in an ensemble model that can handle both numeric and categorical data (say, random forest).
Choose top n predictors in the ensemble model.

I am going through this a roundabout way rather than directly fitting all predictors into an ensemble model to try and reduce the complexity of the problem first (and because in R, I'm having a problem with too many levels from the predictors).

Would this be a valid approach to identifying the n most salient predictors?
Any possible issues to mitigate?


texts: Dissecting performance issues with Random Forest

texts:
My task is to identify potential situation for trading and determine whether a candidate is going to succeed or not. I have a system in place to identify candidates, but there is a high rate of false positives.

To try to reduce it, I have been training a Random Forest in the hopes of pre-emptively eliminating high risk candidates. The problem is that I am using two separate sources of data: one which is a 10-year historical record, and the other which is obtained in real-time and stored in a separate database. If I split the historical data into training and testing (let's say 0.7 - 0.3 split), the results are improbably perfect: no mistakes in the classification for categories with 100's of observations. However, when I try to apply the trained model to the observations from the real-time data, the labelling accuracy is about 50%. This has presisted, despite attempts to tweak hyperparameters and introduce new features.

How could I investigate if there are substantial differences between the data from the two sources, and how could I try to go about ameliorating this situation?


texts: KMeans Clustering on Heterogeneous Dataset


texts:
I am extracting email ids and storing them into a new column variable, but I am getting the issue:

    import re
    def email_extract(comments):
        comments1 =re.findall(r'[\w\.-]+@[\w\.-]+',comments)
       return comments1

   data["email_id"] = data.COMMENTS.apply(lambda x: email_extract(x))

--- TypeError                                 Traceback (most recent call
last) <ipython-input-33-d9b73bdc4f8e> in <module>()
----> 1 data["email_id"] = data.COMMENTS.apply(lambda x: email_extract(x))

C:\ProgramData\Anaconda4\lib\site-packages\pandas\core\series.py in
apply(self, func, convert_dtype, args, **kwds)    3192
else:    3193                 values = self.astype(object).values
-> 3194                 mapped = lib.map_infer(values, f, convert=convert_dtype)
3195     3196         if len(mapped) and
isinstance(mapped[0], Series):

pandas/_libs/src\inference.pyx in pandas._libs.lib.map_infer()

<ipython-input-33-d9b73bdc4f8e> in <lambda>(x)
----> 1 data["email_id"] = data.COMMENTS.apply(lambda x: email_extract(x))

<ipython-input-32-97f3705d1972> in email_extract(comments)
      2 def email_extract(comments):
      3     #re_pattern = re.compile(r'[\w\.-]+@[\w\.-]+')
----> 4     comments1 =re.findall(r'[\w\.-]+@[\w\.-]+',comments)
      5     return comments1

C:\ProgramData\Anaconda4\lib\re.py in findall(pattern, string, flags)
    221
    222     Empty matches are included in the result."""
--> 223     return _compile(pattern, flags).findall(string)
    224
    225 def finditer(pattern, string, flags=0):

TypeError: expected string or bytes-like object

How can I fix this issue?


texts: What are best practices for collaborative feature engineering?

texts:
I work in a large company on several data science projects. For each of the projects me and my colleagues construct features that have some predictive value for the specific target in that project.

Some project are similar in that they a predict something for the same kind of entity, for example customers or goods.

It would make sense to me to share features between projects that are about the same entity. Or at the least, make it easy to reuse features from another project. For example, in some project someone could construct the feature "customer since" which would indicate the number of years someone is a customer. In some other project some constructed a feature "estimated age" that is the outcome of some machine learning pipeline. In a third project I might want to use both features.

What are best practices in sharing these features? Should I share code or materialized outcomes? Are there packages that aid this process? How does your company solve this?


texts: How and Why to rescale image range between [0,1] and [-1,1]

texts:
I am trying to implement model described in Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network in which author says in section 3.2 that

  We scaled the range of the LR input images
  to [0, 1] and for the HR images to [−1, 1]. The MSE loss
  was thus calculated on images of intensity range [−1, 1]

Now I know that when Gray-scaled images are represented in float it's value is in range [0,1] but how do you scale your colour images in range of [0,1] and [-1,1]? And why authors are using two different scales for input and High resolution images?

Any insights would be helpful.


texts: Is it possible to train probabilistic model to return several distributions?

texts:
I have nonlinear data of function y(x), which is let's say parabolic. At some points of x there are several y's (look at the picture).

Is it possible to train a probabilistic model to return several distributions (when needed) i.e. several means and variances. For example: when I feed a (x=a) to the model -> it returns 2 red distributions (2 means and 2 variances), and when I feed b (x=b) to the model -> it returns 1 blue distribution.

Thanks.


texts: How to find the anomaly detection using K Means clustering by considering all labels

texts:
I have 3000 rows of data in excel, I am trying to find the anomaly detection using K Means clustering

id,product,store,revenue,store_capacity,state
1,Ball,AB,222,1000,CA
1,Pen,AB,234,1452,WD
2,Books,CD,543,888,MA
2,Ink,EF,123,9865,NY

if it is two variable we can identify easily if it multi variable, how to consider it?


texts: Unbalanced discounted reward in reinforcement learning : is it a problem?

texts:
Discounted rewards seems unbalanced to me.

If we take as example an episode with 4 actions, where each action receive a reward of +1 :

  +1 -> +1 -> +1 -> +1

The discounted reward for the last action is : 1
The discounted reward for the first action (considering gamma = 1 for simplicity) is : 4

Intuitively both action are as good as the other, because both received same reward.
But their total reward is different, unbalanced.

So when we will backpropagate, first action will be favored over last action ?


texts: How to use flow_from_directory in keras for csv

texts:
flow_from_directory in Keras requires images to be in different subdirectories. However, I have the images in a single directory with a csv file specifying the image name and target classes.

How do I use flow from directory directly from csv files named train.csv and test.csv?


texts: Time Series Generation - Multi Dimensional Time Series Data

texts:
Disclaimer: Mathematicians please don't be mad at me for the use of some of the terminologies in this post. I am an Engineer. :-)

Background:
So I am currently working on a problem where I have to generate a time series sequence of a process in which n actors are moving in a 2d space. But i don't know if this is even possible .The process being learned by some machine learning model M.

BTW! I have never worked with time series data, but have a good experience with training models on images and signals, without a sequence, so i have been reading up on it on the go.

So to start of with trying something very simple, I took a football player position dataset from : Here . And I am trying to model it as a supervised learning problem where I try to predict the positions of n players at timestamp T, given that at timestamp T-1. But I very quickly realised that it wouldnt work because the positions of the players also depend on the position of the ball and that of the opponent team players.

Anyway my questions are as follows :-
1. How do i model the dataset? Will it just be a (Nx2xNo.Timestamps) like 3-d tensor dataset(N corresponding to the players. 2 for the x-position and y-position. and No. of timestamps as the last dimension)?

Is my way of modelling the time series generation problem as a supervised learning problem correct?
what Preprocessing steps should i be looking at? Also how do i handle missing values.
The reason why i dropped the idea about using the soccer dataset : Here again because it only includes positions of one team. The other team didnt wear sensors :-( . I read something about exogenous variables also affecting the process, when reading something about the ARIMA model.
If all this is possible and I hope it is (cos impossible is nothing!) what models should i be looking at? Because i ultimately have to work on this problem on a very different dataset... I have past experience with training Neural Network models like CNNs and ANNs, and feel very comfortable working with Neural Networks, and ideally would love to do so here. Uptil now my research has pointed me towards LSTMs RNNs and the ARIMA model.

Please guide me on the same as i'm very new to time series analysis.


texts:
I once heard that filtering spam by using blacklists is not a good approach, since some user searching for entries in your dataset may be looking for particular information from the sources blocked. Also it'd become a burden to continuously validate the current state of each spammer blocked, checking if the site/domain still disseminate spam data.

Considering that any approach must be efficient and scalable, so as to support filtering on very large datasets, what are the strategies available to get rid of spam in a non-biased manner?

Edit: if possible, any example of strategy, even if just the intuition behind it, would be very welcome along with the answer.


texts: How to get weightage of each factor for a prediction in a neural network classifier?

texts:
I am working on a model to predict which employee is going to resign from a firm. The dataset has columns like Date of Birth, Date of Joining, Department, Gender, Marital Status, Years at company etc.

Using Tensorflow, I created a neural network classifier which is able to give the predictions (Going to leave/Not going to leave) and the probability that an employee is going to leave.

Let's say that the model gives 87% probability that a particular person is going to leave. I want to know which factor is contributing how much to this person's resignation (i.e. to find out the reason for the particular person's leaving)

How can I do that?

I'm using Jupyter notebook for the code and Keras for the neural network.


texts:
I would like to take a set of documents where each document already has an assigned popularity change value ("trending", i.e. -10%, +25%, etc.), and create a machine learning model that would be able to predict a "trending" value for a new document.

After creating TF-IDFs for all documents, dropping some of the rarest words that only appear in 1 document, possibly removing stop words, etc., I'm a little lost. Is it the right approach to use some kind of sentiment analysis model that would train the TF-IDF vectors on the popularity change values? I am also thinking about how to train a regular regression model for this task, but it doesn't seem like an easy choice to integrate with TD-IDF data.


texts: [R]Fuzzy C-Means, different between ppclust vs e1071?

texts:
no. of cluster = 10, data points = 6000

library(ppclust)
cm <- fcm(x,centers = cen)

takes ~ 10 minutes

library(e1071)
cm <- cmeans(x,cen,1000)

takes ~ 1 minute

the only reason I prefer ppclust is because it allows nstart, meaning I am not stuck with local optima.

Given both are doing same thing, why is one taking 10x time then other?


texts:
I have a dataset like this. The data has been collected through a questionnaire and I am going to do some exploratory data analysis.

windows <- c("yes", "no","yes","yes","no")
sql     <- c("no","yes","no","no","no")
excel  <- c("yes","yes","yes","no","yes")
salary <- c(100,200,300,400,500 )

test<- as.data.frame (cbind(windows,sql,excel,salary),stringsAsFactors=TRUE)
test[,"salary"] <- as.numeric(as.character(test[,"salary"] ))

I have an outcome variable (salary) in my dataset and a couple of input variables (tools). How can I visualize a horizontal box plot like this:


texts:
The xgboost package has two files that must be used for ranking:

train.txt with the data
train.txt.group with the group of each observation

I don't understand two things in each file:

What should I use as positive/negative classes? In ranking, there is no such thing as a positive/negative class...
What should I use as group? Say that I have observations x1 > x2 > x3. How do I express this lexicographical order using groups?


texts: How much of a background in programming is necessary to become a data scientist?

texts: I once knew some Java, but that was close to 10 years ago.  Assuming I can learn a language to get into data analytics.... what language do you recommend?


texts: Marketing Mix Model (MMM) using XGBoost?

texts:
Does anyone know of any literature on Marketing Mix Modelling (MMM) using XGBoost? Is this a viable techniques for MMM modelling? What would the advantages/disadvantages be?

I think it would be a great option due to that interactions effects are dealt with in a good way, and the model also avoids inflated values. However, I am not sure of whether XGBoost is the best model for dealing with the multi colliearity often present in MMM data.

I would be happy for any thoughts on this.


texts: What technologies are fastest at performing joins on large datasets?

texts:
By "large", I mean in the range of 100m to 10b rows.

I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the data is distributed well for the given query.

Are there other technologies that I should be looking at here? If so, what are the trade offs?


texts: Understanding of Convolutional Neural Network (CNN)

texts: I am a beginner in ML and I would like to learn CNN with Math behind. If you suggest any good blog or documentation which help me get depth knowledge about CNN.


texts: Applying network diffusion to a graph created with the igraph library in R

texts:
I have created a graph using the igraph library.
library(igraph)
transactions <- readRDS("transactions.rds")
nodes <- readRDS("nodes.rds")

g<-graph_from_data_frame(d=transactions,vertices = nodes,directed = T)

I am using the subset of the original database which I have stored here https://github.com/leonelparrales22/NetworkDiffusion
What I want to know is, when stimulating a node in the network, what is the impact it is going to generate by infecting neighboring nodes.
For example, what is the percentage of the network that will be affected if I stimulate node X?
I would like to make a simulation or know what metrics can help to realize something like the following image:

I was trying with this library,
netdiffuseR
But it asks for a TOA argument, which I don't understand very well what it refers to. And I don't know if I can calculate it or it is something that had to have been previously collected.


texts: Clustering similar sequences using hidden markov model

texts:
I have several sequences of different lengths. For example,
Seq1 -- "A", "B", "A", "A", "K", "L", "C", "B","C", "D"
Seq2 -- "A", "C", "A", "D", "C", "A", "K", "D"
Seq3 -- "B", "C", "A", "A", "B", "K", "A", "D", "L", "A", "B","C","D"
.................................
......................................
Seq40 - "C","D", "A", "A", "D", "B", "K", "K", "C","D"

I want to cluster these sequences using HMM. My idea is to cluster them based on their hidden states. Is there any standard method for clustering sequences using HMM? Also, I know in python, hmmlearn and in R, seqHMM package are good. How can I fit these sequences to cluster them?


texts: Scikit Learn: KMeans Clustering 3D data over a time period (dimentionality reduction?)

texts:
I have a dataset of xyz coordinates with a date component in a pandas dataframe

ex:

date1: $[x_1,y_1,z_1]$,
date2: $[x_2,y_2,z_2]$,
date3: $[x_3,y_3,z_3]$,
..

I would like to classify a sample of object positions over the period of a week
(using indexes to re-map the classification label back to the date), like this:

Week 1: $[x_1,y_1,z_1], [x_2, y_2, z_2], [x_3,y_3,z_3], [x_4,y_4,z_4], [x_5,y_5,z_5], [x_6,y_6,z_6], [x_7,y_7,z_7]$,
Week 2: $[x_8,y_8,z_8],[x_9,y_9,z_9],[x_{10},y_{10},z_{10}],[x_{11},y_{11},z_{11}],[x_{12},y_{12},z_{12}],[x_{13},y_{13},z_{13}],[x_{14},y_{14},z_{14}]$,

When I try to run KMeans it returns

k_means = KMeans(n_clusters=cclasses)
k_means.fit(process_set.hpc)
date_classes = k_means.labels_

ValueError: Found array with dim 3. Expected <= 2

Questions:

Do I have to run it through Principal Component Analysis (PCA) first? if so, how do I maintain date mapping to the classification created?
Are there any other methods I could use?
Am I doing everything completely backwards and should consider a different approach, any thoughts?

Thanks!


texts: How to determine if character sequence is English word or noise

texts:
What kind of features you will try to extract from list of words for future predicting, is it existing word or just mess of characters ?

There is description of task that I found there.

You have to write a program that can answer whether a given word is English. This would be easy — you'd just need to look the word up in the dictionary — but there is an important restriction: your program must not be larger than 64 KiB.

So, I thought it would be possible to use logistic regression for solving the problem. I don't have a lot of experience with data-mining but the task is interesting for me.

Thanks.


texts:
Is it possible for results to fluctuate a little bit even if I set set.seed? Kindly comment on the degree of freedom allowed or the possible standard deviation from the result set if seeds are fixed?

I set seeds in R and I get slightly varied results ranging from 0.1 to 0.5 S.D from previous runs. I want to know if it is normal


texts: Has anybody used alternative hyperparameter optimization techniques (other than default one) in SK-Learn?

texts:
I've been using Sklearn for Gaussian process regression that has L-BFGS-B (“fmin_l_bfgs_b”) as a default optimization algorithm. I want to implement some other internally available algorithms like  this and some other third party algorithm like Firefly algorithm.

I tried to use the  trust-region-reflective algorithm with the following way:

def nll_fn(X_train, Y_train, noise):
    ''' Returns a function that computes the negative log-likelihood for training data X_train and Y_train and given noise level. Args: X_train: training locations (m x d). Y_train: training targets (m x 1). noise: known noise level of Y_train. Returns: Minimization objective. '''
    def step(theta):
        K = kernel(X_train, X_train, l=theta[0], sigma_f=theta[1]) + noise**2 * np.eye(len(X_train))
        # Compute determinant via Cholesky decomposition
        return np.sum(np.log(np.diagonal(cholesky(K)))) + 0.5 * Y_train.T.dot(inv(K).dot(Y_train)) + 0.5 * len(X_train) * np.log(2*np.pi)
    return step

def trust_region_optimizer(obj_func, initial_theta, bounds):
    trust_region_method = least_squares(obj_func,initial_theta,bounds,method='trf')
    return (trust_region_method.x,trust_region_method.fun)

However, I get the error as:

trust_region_method = least_squares(obj_func,initial_theta,bounds,method='trf')
  File "C:\Python\lib\site-packages\scipy\optimize\_lsq\least_squares.py", line 744, in least_squares
    raise ValueError("`jac` must be '2-point', '3-point', 'cs' or "
ValueError: `jac` must be '2-point', '3-point', 'cs' or callable.

Has anybody implemented any other optimization algorithm in Sklearn? Any sample code or tutorial links would be very much helpful. Moreover, any sample codes for optimization algorithms in any platform (E.g. Matlab) would be very supportive.

Thank you.


texts:
I have a large dataset and I would like to convert these categorical data into numeric in binary form to perform k means clustering in R. However, I get an error in value. This is the formula that I have entered into the excel sheet: =LEFT(B2,FIND("  ",B2,1)-1).

This is my dataset.


texts: Difference between train, test split before preprocessing and after preprocessing

texts:
I am new to machine learning. I am bit confused in preprocessing. Generally,

Scenario-1: I am splitting the dataset into train,test and validation and applying the transformations like fit_transform on train and transform on test.

Scenario-2: The other method is applying transformations on the entire dataset first and then split the dataset into train,test and validation. I am bit confused in choosing , dividing the data before preprocessing and feature engineering or after preprocessing and feature engineering. Looking for a nice answer with effects and casues.


texts: The reason behind using a pre-trained model?

texts: These last month I have been studying all about word embeddings and the most known pre-trained word embeddings, Word2Vec, GloVe, FastText, etc. I have read many times how important It is to take advantage of pre-trained models when doing a given task however I don't understand how a pre-trained model can adapt to my given corpus. Furthermore, If I have new words not present in the pre-trained model will I be able to use this pre-trained model to learn the embeddings for the new words?


texts: Cross-Validation in Anomaly Detection with Labelled Data

texts:
I am working on a project where I train anomaly detection algorithms Isolation Forest and Auto-Encoder. My data is labelled so I have the ground truth but the nature of the problem requires unsupervised/semi-supervised anomaly detection approach rather than simple classification. Thus I will use the labels for validation only.
Since I will not train the model with the labels, unlike supervised learning where I would have X_train, X_test, y_train and y_test, what is the right approach for model validation here?
If this were supervised learning, I would split data into 3 parts: train, CV and test, doing K-Fold CV. But now I feel like I can simply divide my data into 2: train and test and simply fit all of the train data, predict and tune the models according to. Then finally, predict the test data.
So my question is, should I include some kind of CV in this model? What is the right way here?


texts:
I saw in DQN nature paper 2015 https://www.nature.com/articles/nature14236(Extended Data Table 4) some comparisons between DQN and  linear Q-learning.

The ratio of reward for different games are very different. For instance I saw the ratio of reward of Linear Q-learning is between 3.8 to 100 times more than DQN? What is the reason of this big difference? How we can clarify this? Is it related to complexity or sensitivity of game? I am not very familiar with complexity and properties of these games.

In addition, in Extended Data Table 2 of this paper it has written it is compared with best linear learner?

Would you please explained this sentence:"Best Linear Learner is the best result obtained by a linear function approximator on different types of hand designed feature"

What is the meaning that by best? The best results during test of linear? Because I only know one linear regression

  y=ax+b

, I only replace the DNN section of DQN with a linear regression (i.e. just input and output and linear without any hidden layer) I am confused with best word in this table.

Also, do you have seen the comparison with Logistic Regression Q-learning? (What I mean by logistic regression Q-learning is to repalce the deep neural network section of DQN with a Sigmoid function between input and output without any hidden layer) What is the ratio of result of this approach in comparison with DQN?


texts:
I'm trying to understand how to feed data into LSTM layer of Keras, but I'm in trouble and I don't understand how to do it.
I've have a dataset composed by words and each words is embedded with a vector of 839 elements, so the shape of my dataset is (x, 839).

I want to feed my dataset into the LSTM layer, but I don't properly understand the 3D object wanted by Keras, composed by (batch_size, timesteps, feature). I want to feed one word per time into the LSTM, how can I do?

Thanks in advance.

UPDATE
I still receive an error about input shape:

  ValueError: Error when checking input: expected lstm_11_input to have
  shape (2, 839) but got array with shape (839, 1)

I'm using batch_size_shape(batch_size, timesteps, feature) at the moment.
This is the code:

class KendallTauHistory(Callback):
  def __init__(self, dataset, y_true, groups):
    self.y_true = y_true
    self.dataset = dataset
    self.groups = groups

  def on_epoch_end(self, epoch, logs=None):
    predictions = self.model.predict(self.dataset)
    predictions = predictions.flatten()
    predictions = list(map(lambda element: element + np.random.uniform(0.0, 1.0) * 0.02 - 0.01, predictions))
    # For batch training
    ranked_predictions = np.array([])
    kendalls = np.array([])
    start_range = 0
    for group in self.groups:
        end_range = (start_range + group[1]) # Batch is a group of words with same group id
        batch_predictions = predictions[start_range:end_range]
        batch_labels = self.y_true[start_range:end_range]
        batch_predictions = list(map(lambda element: element + np.random.uniform(0.0, 1.0) * 0.02 - 0.01, batch_predictions))
        ranked_predictions = np.append(ranked_predictions, np.floor(rankdata(batch_predictions)))
        kendalls = np.append(kendalls, kendalltau(batch_labels, batch_predictions))
        start_range = end_range
    #self.y_true = self.y_true[0:len(ranked_predictions)]
    print('\nORIGINAL LABELS: {0}\n'.format(self.y_true))
    print('PREDICTED LABELS: {0}'.format(ranked_predictions))
    print("\nEpoch Kendall's tau: {0}".format(np.mean(kendalls)))

model = tf.keras.Sequential()
model.add(LSTM(units=10, batch_input_shape=(None, 2, 839)))
model.add(Dense(15, activation='sigmoid'))

model.summary()

model.compile(loss=listnet_loss, optimizer=keras.optimizers.Nadam(learning_rate=0.000005, beta_1=0.9, beta_2=0.999))
real_labels = np.array([])
losses = np.array([])

with tf.device('/GPU:0'):
  model.fit(training_dataset, training_dataset_labels, epochs=10, workers=10,
            verbose=1, callbacks=[KendallTauHistory(training_dataset, training_dataset_labels, groups_id_count)])


texts:
I am working on a prediction model where I must find out the destination of a boat based on its actual coordinates and heading (compass direction) :
In[8]: X.head()
Out[8]:
   latitude  longitude     heading
0  0.094700   0.094700  332.398791
1  0.090828   0.090828  197.320172
2  0.085800   0.085800  140.537550
3  0.081676   0.081676  128.891893
4  0.077804   0.077804  129.881418

Latitude and longitude are in degrees (normalised here) and heading is in degrees.
The output variable is an integer between 1 and n (there are n possible destinations).
My problem here is to make the neural network understand how heading difference works : 5 degrees and 15 degrees are as close as 5 degrees and 355 degrees because we have to use a modulo 360 after making the difference :
In[11]: diff_heading(355, 5)
Out[11]: 10
In[12]: diff_heading(15, 5)
Out[12]: 10

I've got some acceptable results when I put the coordinates and heading in a basic multilayer perceptron neural network without any preprocessing on the heading. But I'm pretty sure it could be way more better if the network could take into account this heading problem. For now, the network must attribute weights to the heading which indicate that 0 and 360 degrees are opposite, which is false.
Do you have any idea on how to process the data or how to change the network structure to achieve this ?


texts: Automatic topic labelling for topic modelling

texts: I am just curious to know if there is a way to automatically get the lables for the topics in Topic modelling. It would be really helpful if there's any python implementation of it.


texts: How to measure the correlation between categorical variables and a continuous variable

texts:
I have the following list of the names of the categorical variables in my dataset:

categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
                       'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
                       'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
                       'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']

in this function definition, I am one-hot encoding each column with the help of a for loop:

def feature_encoding(df, categorical_list):

    # One Hot Encoding the columns gathered in categorical_columns
    for col in categorical_list:

        # take one-hot encoding
        OHE_sdf = pd.get_dummies(df[categorical_list])

        # drop the old categorical column from original df
        df.drop(col, axis = 1, inplace = True)

        # attach one-hot encoded columns to original dataframe
        df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

I don't want to use all of these columns in my training so now I am at the dimensionality reduction phase of my work.  I want to measure the correlation between each of these columns and my SalePrice variable (numerical) and weed out the columns with low correlation.

I've read that Chi-square test is generally used for measuring the correlation of categorical variables but I have not seen an implementation where it was a list of categorical variables vs a continuous variable.


texts: Strategies for continuously assessing and improving model performance

texts:
I am building a supervised machine learning model to generate forecast.

So I would have historic data like this:

SKU, Month, .... other features, Actual Volume

That I can use a model to generate forecast, using the actual volume as the label.

Of course, there would be a variance between the forecast volume and the actual volume

What are the proper ways to leverage such data, without generating any data leakage, to incorporate such info to train the model to minimize the variance?

Should the data be fed back to the data with moving average, etc. and retrain, or is there other better strategy to properly assess the performance of the model and learn from it?

The data will be time series data with various features such as exchange rate, salesperson, etc.


texts: Overfitting and COLT/Statistical Learning Theory

texts:
The aspect of over-fitting is typically viewed from the perspective of both- accuracy and model complexity.
To mitigate over-fitting, we usually have the practical approach of having k-fold validations, training-validation-test set.

Question: theoretically, can we leverage the Statistical Learning theory (COLT) to draw bounds on the confidence of how well learning has happened,
and how well classification can happen over unseen examples?

Example, many a time we consider the minimum number of samples (upper bound/sufficient samples) needed to learn (VC Dimension).
Agreed, it is an overestimate, plus |H| or VC Dimension may be unknown in practice.
The other perspective wherein, given the number of samples 'm', and 'delta' (probability of failure) -
we find error bound (on unseen examples)- Can we theoretically interpret this error bound to be an estimate of over-fitting? I get this intuition from Nello Cristianini's book on SVMs.

Appreciate any pointers.


texts: Multiple models vs. Single model for prediction

texts:
I am using the Darknet Convolutional Neural Networks to detect people (as in, humans) and furniture in a single image.

If I train the model twice, one for people, one for furniture. I seem to get better results by lowering the threshold opposed to training the model once on both people and furniture.

Now my question is this: would this be considered an anti-pattern? Or is it common practice to solve my problem like this?


texts: Machine Learning - Precision and Recall - differences in interpretation and preferring one over other

texts:
I have summarising this from lot of blogs about Precision and Recall.
Precision is:

Proportion of actual positives that classifier has predicted as positive.

meaning out of the sample identified as positives by classifier as positive, how many are actually positive?
and Recall is:

Proportion of actual positives were predicted as positive correctly.

meaning out of the ground truth positives, how many were identified correctly by the classifier as positive?

That sounded very confusing to me. I couldn't interpret difference between both of them and relate each to real examples. some very small questions about interpretation I have are:

if avoiding false-positives matter the most to me, i should be measuring precision; And if avoiding false-negatives matters the most to me, i should be measuring recall. Is my understanding correct?
Suppose, I am predicting if a patient should be given a vaccine, that when given to healthy person is catastrophic and hence should only be given to an affected person; and I can't afford giving vaccine to healthy people. assuming positive stands for should-give-vaccine and negative is should-not-give-vaccine, should I be measuring Precision? or Recall of my classifier?
Suppose, I am predicting if an email is spam(+ve) or non-spam(-ve). and I can't afford a spam email being classified as non-spam, meaning can't afford false-negatives, should I be measuring Precision? or Recall of my classifier?
What does it mean to have high precision(> 0.95) and low recall(< 0.05)? And what does it mean to have low precision(> 0.95) and high recall(< 0.05)?

Put simply, in what kind of cases is to preferable or good choice to use Precision over Recall as metric and vice versa. I get the definition and I can't relate it to real examples to answer when one is preferable over other, so I would really like some clarification.


texts: Distinguish real building pictures from rendered ones

texts:
I need to train an algorithm to identify rendered building images from real pictures. Here's an example:

A rendered picture

A real picture

The real pictures will vary from Good quality to Poor quality, (Like the one above) and will have many different aspect ratios. The same goes for the rendered pictures.

So what would you suggest me to use to do that, does anyone knows something that already do something similar?


texts: Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning

texts:
It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the activation functions instead of ELU?

In other words what is the advantage of ELU over $log(1+e^x)$?


texts:
I'm working on my first NN following a tensorflow tut and trying to use my own data.
After about 80 attempts of formatting my data and trying to load it into a dataset to train I'm throwing the towel.

Here is how my data currently looks

syslog_data = [
[302014,0,0,63878,30,3,1], [302014,0,0,3891,0,0,0], [302014,0,0,15928,0,0,2], [305013,5,0,123,99999,0,3],
[302014,0,0,5185,0,0,0], [305013,5,0,123,99999,0,3], [302014,0,0,56085,0,0,0], [110002,4,2,50074,99999,0,4],

In this the last item in each list is the label.
If you can tell me if I need to reformat my data and how or just how to get it loaded into a dataset properly.

Thanks for any help or advice you can give

Here is the full code:

import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from . import syslog

print(tf.VERSION)
print(tf.keras.__version__)

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation='relu'))
# Add another:
model.add(layers.Dense(64, activation='relu'))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation='softmax'))

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

dataset = tf.data.dataset.from_tensor_slices(syslog)

model.fit(dataset, epochs=10, steps_per_epoch=30)


texts:
I am  trying to classify stock images into 4 categories:

Category 1: Image patterns which gives more than 10% profit next 3
day.
Category 2: Image patterns which gives less than 10% profit but
greater than 0% next 3 day.
Category 3: Image patterns which gives more than -0.01% but greater
than -10% loss next 3 day.
Category 4: Image patterns which gives more than -10% loss next 3 day

I have divided these Images in to 4 category, below are the Images from category 1 and category 2.

Image contains 10 day stock movement, and classification category are generate from next 3 day stock data.

Red Bar is volume Traded of that stock.

Pink line is RSI(14) (Relative strength Index)

Yellow line is stock price.

Blue and Green line are moving average 10, 20 respectively.

I know by seeing the image it is hard to differentiate between them but this is what I am trying to find out can I classify it using CNN.

My CNN:

  (layer1): Sequential(
    (0): Conv2d(3, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (layer2): Sequential(
    (0): Conv2d(64, 128, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (layer3): Sequential(
    (0): Conv2d(128, 256, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (drop_out): Dropout(p=0.5, inplace=False)
  (fc1): Linear(in_features=65536, out_features=1024, bias=True)
  (fc2): Linear(in_features=1024, out_features=512, bias=True)
  (fc3): Linear(in_features=512, out_features=128, bias=True)
  (fc4): Linear(in_features=128, out_features=4, bias=True)

Optimizer: Adam, Loss: CrossEntropyLoss.

But the problem is CNN ends up classifying same class for each image to obtain 25% accuracy.

Am I working in wrong direction, should I work on imporving data or try other technique like LSTM?


texts: Can I use Gensim doc2vec model for classification new documents?

texts: Can I use Gensim doc2vec model for classification new documents via infer_vector? All my tests gave too bad results, even for big datasets (10GB utf-8 texts)...


texts:
I have  customer buying data with each row specifying an item bought by customer. The problem is that even if at the same time customer buys five items then there are five different rows for it and as a result the total number of rows in data have gone too much to train. what can i do to reduce the size of data so that i can train it effectively.Just to give the context of the problem, i want to recommend products to the customers based on their buying data.

Dataset size: (7981262, 16)

Data description:

Variable                                           Description
customerID                                    unique customer ID
DOB                                         date of birth of customer
Gender                                               gender
State                                          customer's state
PinCode                                pincode of area where customer lives
transactionDate                                  date of transaction
store_code                                 unique code of store
store_description                             description of store
till_no                                       counter no. in the store
transaction_number_by_till                 unique transaction number by counter,
transactionDate, store_code
promo_code                      if promotional code (offer) used in the transaction
promo_description                            description of the offer
product_code                               unique code of the product purchased
product_description                       description of the product purchased
sale_price_after_promo                 sale price of the product after applying
promotion
discountUsed                  after promo, customer used this discount(s) on transaction


texts: Feature selection/reduction technique for combination of features in image processing

texts:
I have a combination of features extracted from 3 descriptors. Namely, GLCM based features (correlation, homogeneity, energy, and contrast ), Local binary patterns (256), and discrete wavelet transform coefficients,
and a dataset of 40 cases. I am using SVM with RBF kernel.
Would there be a need for the feature selection? If so, what do you recommend?


texts: How to visualize image segmentation results

texts:
I am using u-net to do semantic segmentation for N>1 classes. The input size is (128,128,3), the output size will be (128,128,N). what is the correct way see the prediction as an image ot size n1 x n2 x 1 ?

My try: I took the maximum if every (i,j) vector (of size N) in the output tensor to get an scalar and plot a 128 x 128 image.


texts: How to extract true positives data (complete row with data) after training and testing from test dataset?

texts:
How do you extract true positive data from testing data after training and testing?
For example, in the test data, I have two rows and one row is true positives and the other is false negatives. However, I would like rows which only have true positive values. How do you extract the complete row from the testing data after training and testing?


texts: Is there any technique, which can decide no. of bags for Bagged Logistic regression?

texts:
In case of Bagged Logistic regression, people suggest more the bags better will be results.
There should be some threshold for more. Is there any technique available which can suggest no. of bags for the type of data available.

For ex.
Details about data
No. of features - 6
No. of records - 1 M\
Bag size - 20%
No. of Bags - ?

For above mentioned details, Is there any way to devise no. of bags?


texts: How do I find the count of a particular column, based on another column(date) using pandas?

texts:
I have a dataframe with 3 columns, such as SoldDate,Model and TotalSoldCount. How do I create a new column, 'CountSoldbyMonth' which will give the count of each of the many models sold monthly?

Date        Model  TotalSoldCount
Jan 19        A          4
Jan 19        A          4
Jan 19        A          4
Jan 19        B          6
Jan 19        C          2
Jan 19        C          2
Feb 19        A          4
Feb 19        B          6
Feb 19        B          6
Feb 19        B          6
Mar 19        B          6
Mar 19        B          6

The new df should look like this.

Date      Model     TotalSoldCount     CountSoldbyMonth
Jan 19     A               4                    3
Jan 19     A               4                    3
Jan 19     A               4                    3
Jan 19     B               6                    1
Jan 19     C               2                    2
Jan 19     C               2                    2
Feb 19     A               4                    1
Feb 19     B               6                    3
Feb 19     B               6                    3
Feb 19     B               6                    3
Mar 19     B               6                    2
Mar 19     B               6                    2

I tried doing
df['CountSoldbyMonth'] = df.groupby(['date','model']).totalsoldcount.transform('sum')
but it is generating a different value.


texts:
I'm trying to predict the possible diagnosis given a consultation reason. I have ID's for all the data. So my data kind of looks like below

Reason            | Diagnosis
------------------------------
448, 124          | 9
551, 448, 122     | 9, 12
111, 110          | 32
143               | 43

There can be up to 10 reasons and upto 5 diagnoses in my training data.

What I'm looking for in the algorithm or model is that it accepts 1 - 10 reason_ids as input and returns top 5 possibilities for diagnosis with % of probability.

I'm good at python so if there is any open source model or code I can look at it will be great.


texts: XGBoost: Typical gamma and min_child_weight range

texts:
What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm?
Is the range of min_child_weight correlated with the number of feature or samples in the training dataset? Some of the available resources recommend setting this value to 1/sqrt(event_rate), what is the corresponding rule when training a regression model?


texts: how to resolve this error in Tesseract : (2, 'Usage: pytesseract [-l lang] input_file')

texts:
pytesseract.pytesseract.tesseract_cmd = r'/usr/local/bin/pytesseract'
img = Image.open("/content/drive/My Drive/abc.jpg")
print(pytesseract.image_to_string(img))

stacktrace:

  --> print(pytesseract.image_to_string(img))

  sr/local/lib/python3.6/dist-packages/pytesseract/pytesseract.py in
  image_to_string(image, lang, config, nice, output_type, timeout) 346
  Output.DICT: lambda: {'text': run_and_get_output(*args)}, 347
  Output.STRING: lambda: run_and_get_output(*args),
  --> 348 }output_type 349 350

  /usr/local/lib/python3.6/dist-packages/pytesseract/pytesseract.py in
  () 345 Output.BYTES: lambda: run_and_get_output(*(args + [True])), 346
  Output.DICT: lambda: {'text': run_and_get_output(*args)},
  --> 347 Output.STRING: lambda: run_and_get_output(*args), 348 }output_type 349

  /usr/local/lib/python3.6/dist-packages/pytesseract/pytesseract.py in
  run_and_get_output(image, extension, lang, config, nice, timeout,
  return_bytes) 256 } 257
  --> 258 run_tesseract(**kwargs) 259 filename = kwargs['output_filename_base'] + extsep + extension 260 with
  open(filename, 'rb') as output_file:

  /usr/local/lib/python3.6/dist-packages/pytesseract/pytesseract.py in
  run_tesseract(input_filename, output_filename_base, extension, lang,
  config, nice, timeout) 232 with timeout_manager(proc, timeout) as
  error_string: 233 if proc.returncode:
  --> 234 raise TesseractError(proc.returncode, get_errors(error_string)) 235 236

  TesseractError: (2, 'Usage: pytesseract [-l lang] input_file')


texts: I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be grateful if I know get some snippet in python to find the probability of labels.


texts:
I am trying to build a 2 level stacking model in order to tackle a multiclass classification problems with 8 classes.
My base (level 1) models andd their micro f1 scores in the test set are:

Random Forest Classifier (0.51)
XGBoost Classifier (0.54)
LightGBM Classifier (0.54)
Logistic Regression (0.44)
keras neural network (0.57)
keras neural netqork (0.56)

As a level 2 model I use an XGBClassifier which is not tuned.
I use 7 fold cross validation to produce the meta features for level 2 model.
The code I use to produce the meta features for the simple classifiers is:

ntrain = X_train.shape[0]
ntest = X_test.shape[0]
seed = 0
nfolds = 7
kf = StratifiedKFold(nfolds, random_state=seed)
def get_meta(clf, Χ_train, y_train, Χ_test):
meta_train = np.zeros((ntrain,))
meta_test = np.zeros((ntest,))

for i, (train_index, test_index) in enumerate(kf.split(X_train, y_train)):
    Χ_tr = X_train.iloc[train_index]
    y_tr = y_train.iloc[train_index]
    Χ_te = Χ_train.iloc[test_index]

    clf.train(Χ_tr, y_tr)

    meta_train[test_index] = clf.predict(Χ_te)

clf.fit(X_train,y_train)
meta_test = clf.predict(X_test)
return meta_train.reshape(-1, 1), meta_test.reshape(-1, 1)

and for the keras neural networks is:

def get_meta_keras(clf, Χ_train, y_train, Χ_test, epochs = 200, batch_size = 70, class_weight=class_weights):
meta_train = np.zeros((ntrain,))
meta_test = np.zeros((ntest,))

encoder = LabelEncoder()
encoder.fit(y_train)
encoded_Y = encoder.transform(y_train)
# convert integers to dummy variables (i.e. one hot encoded)
dummy_y = np_utils.to_categorical(encoded_Y)

for i, (train_index, test_index) in enumerate(kf.split(X_train, y_train)):
    Χ_tr = X_train.iloc[train_index]
    y_tr = dummy_y[train_index]
    Χ_te = Χ_train.iloc[test_index]

    clf.fit(Χ_tr, y_tr, epochs = epochs, batch_size = batch_size, class_weight=class_weights)

    meta_train[test_index] = clf.predict_classes(Χ_te)

clf.fit(X_train, dummy_y, epochs = epochs, batch_size = batch_size, class_weight=class_weights)
meta_test = clf.predict_classes(X_test)
return meta_train.reshape(-1, 1), meta_test.reshape(-1, 1)

My final micro f1 score is 0.54 which is less than the best of my base models score. My models are uncorrelated (corr<0.55). I tried to add more simple models like knn, naive bayes etc but the score fell even more.
Why doesn't my stacking approach improve the score ?


texts: Can anyone recommend any command line tool for converting large CSV file into HDF5 format?


texts: Running examples from scikit-learn tutorials

texts:
I am new to scikit-learn. I went through the examples given in the docs and I downloaded the script for recognizing images of hand-written digits. When I made the script to run on my laptop, I got the following errors:
Traceback (most recent call last):
   File "C:\Python34\plot_digits_classification.py", line 22, in <module>
    from sklearn import datasets, svm, metrics
    File "C:\Python34\lib\site-packages\sklearn\__init__.py", line 57, in<module>
    from .base import clone
   File "C:\Python34\lib\site-packages\sklearn\base.py", line 11, in <module>
  from .utils.fixes import signature
  File "C:\Python34\lib\site-packages\sklearn\utils\__init__.py", line 11, in <module>
   from .validation import (as_float_array,
   File "C:\Python34\lib\site-packages\sklearn\utils\validation.py", line 16, in <module>
   from ..utils.fixes import signature
  File "C:\Python34\lib\site-packages\sklearn\utils\fixes.py", line 324, in <module>
   from scipy.sparse.linalg import lsqr as sparse_lsqr
   File "C:\Python34\lib\site-packages\scipy\sparse\linalg\__init__.py", line 109, in <module>
   from .isolve import *
  File "C:\Python34\lib\site-packages\scipy\sparse\linalg\isolve\__init__.py", line 6, in <module>
  from .iterative import *
  File "C:\Python34\lib\site-packages\scipy\sparse\linalg\isolve\iterative.py", line 7, in <module>
  from . import _iterative
  ImportError: DLL load failed: The specified module could not be found.

Please help me. Also, I want to know if I want to load data for prediction, how should I do that? For example, if I want to test a hand written digit that is stored somewhere on my disk, how to prepare that data for loading and passing into this model for prediction?


texts:
I am comparing the performances of several SVM models in matlab using the fitcsvm function,
and I want to double check that I am using the correct syntax
for hard soft amragins and kernel:
the syntax of hard margin should be as follows, in which the hyperparameter of hard margin cost (boxConstraint) should be infinite

%Hard Margin
SVMModel = fitcsvm(x_train,y_train,'BoxConstraint',Inf);

while the soft margin, the boxConstraint (which is the only hyperparameter needed for soft margin)should be tuned and given a suitable value, for example

 %soft Margin
    SVMModel = fitcsvm(x_train,y_train,'BoxConstraint', 7);

or leave it as default (which has the boxconstraint as 1)

SVMModel = fitcsvm(x_train,y_train);

and for the kernel model, say RBF, both boxconstrains and KernelScale (gamma) should be tuned and used

    SVMModel = fitcsvm(x_train,y_train, KernelFunction, 'RBF',...
'BoxConstraint', 7,'KernelScale', '0.3');

assuming the all hyperparameters are tuned, is the previous syntax considered correct to have hard, soft and kernel models in Matlab?

Also, in case of cross-validation of the model, how to get the best hyperparameters while the SVM model is cross-validated? or selecting the hyperparamters step should be before cross-validating the model?


texts: How to build multiple variable regression having a mix of numerical & categorical features?

texts:
There is a need to estimate Annual Average Daily Traffic Volume (AADT).
We have bunch of data about vehicles' speeds during several years. It is noticed that AADT depends on the average number of such samples during some time, so a regression model $Y = f(x_1)$ could help estimating the AADT.
The problem is there are other features affecting the dependency which are both numerical $(x_2, .., x_k)$ and categorical $(c_1 = data\ provider, c_2 = road\ class, .., c_m)$.
We believe that $x_1$ affects the AADT much more than all the other features and the $x_1$ itself could also depend on other features too.
That's why we would like to get a set of regressions $Y = f(x_1)$ depending on $(x_2, ..x_k,  \ c_1, ..c_m)$.
Both $k$ and $m$ are just few.
—
Is it reasonable to cluster dataset by features $(x2, .., x_k,\  y_1, .., y_m)$ first, and then try to find regression $Y=f(x_1)$ in each cluster?
Or is it better to consider all the features $(x1, x2, .., x_k,\ y_1, .., y_m)$ together with $x1$ having more weight than others?
Also note that for multiple variable regression there is a mix of numerical & categorical features.


texts: Validation score (f1) remains the same when swapping labels

texts:
I have an imbalanced dataset (True labels are ~10x than False labels) and thus use the f_beta score as a metric for model performance, as such:

def fbeta(beta):
    def f1(y_true, y_pred):
        def recall(y_true, y_pred):
            true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
            possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
            recall = true_positives / (possible_positives + K.epsilon())
            return recall

        def precision(y_true, y_pred):
            true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
            predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
            precision = true_positives / (predicted_positives + K.epsilon())
            return precision
        beta_squared = beta ** 2
        precision = precision(y_true, y_pred)
        recall = recall(y_true, y_pred)
        fbeta_score = (beta_squared + 1) * (precision * recall) / (beta_squared * precision + recall + K.epsilon())
        return fbeta_score
    return f1

Running my model I get the following output.

  Epoch 00001: val_f1 improved from -inf to 0.90196, saving model to
  output\LSTM\weights\floods_weights.h5         Epoch 2/3
  - 0s - loss: 0.4114 - f1: 0.8522 - val_loss: 0.4193 - val_f1: 0.9020                                                                                Epoch 00002: val_f1 did not improve from 0.90196
  Epoch 3/3
  - 0s - loss: 0.3386 - f1: 0.8867 - val_loss: 0.3589 - val_f1: 0.9020                                                                                Epoch 00003: val_f1 did not improve from 0.90196
  Evaluating :
  51/51 [==============================] - 0s 372us/step
  Final Accuracy :  0.9019607305526733

However, when running the model again with swapped y-labels (True->False, False->True) I get the exact same values and do not understand why as the f-measure should be highly unlikely to give the exact same results.

When assessing the final model using the following code for precision and recall, I do get different results: 0.6 vs 0.95788:

prediction = model.predict(data['X_test'], batch_size=64)
tp = 0
fp = 0
fn = 0
for y, p in zip(data['y_test'], np.argmax(prediction, axis=1)):
    y = y[0]
    if y == 1 and p == 1:
        tp += 1
    elif p == 1 and y == 0:
        fp += 1
    elif p == 0 and y == 1:
        fn += 1

precision = tp / (tp + fp)
recall = tp / (tp + fn)
print(2 * (precision * recall) / (precision + recall))

What's happening?

Edit:

I am compiling the model as such:

beta = 1
model.compile(
    optimizer=Adam(lr=1e-3),
    loss='categorical_crossentropy',
    metrics=[fbeta(beta)]
)


texts: Introducing weights into spectral clustering

texts:
Suppose I have a data set with points $x_i$ and a dissimilarity measure $d_{ij}$ between each pair, as well as a weight $w_{ij}$ that qualifies the quality of this dissimilarity. I have two problems:

The first one is how to introduce weights when performing spectral clustering. Like in the Shi-Malik, Ng-Jordan-Weiss versions, where only the distances are used.
The second one is that my dissimilarity measure does not exists sometimes. In these cases, it has zero weight. How to introduce these cases in the algorithm? I suppose the zero weight solves the problem if I know how to introduce them.

An idea I had is to perform the similarity transformation as:
$$
s_{ij} = w_{ij}  e^{-\frac{d^2_{ij}}{\sigma^2}}
$$
but I feel it does not make sense since the weigth should reflect the importance of the measure, not the measure itself.

The data is not Euclidean, so I can use only this dissimilarity measure. Any reference is appreciated.


texts:
I'm creating a CNN model in keras to categorise data. There's 150 features (represented as columns), each of which has a label of 1 to 9. The issue is I'm confused about how to use this in Keras since there's uneaven numbers of features to labels, with the features being able to be shown with dimensions of 100, 150 and the labels with dimensions of 100,1. Help understanding how to properly train this data on a Keras model would be appreciated.

#Reshape data to add new dimension
X_train = X_train.reshape((100, 150, 1))
Y_train = X_train.reshape((100, 1, 1))
model = Sequential()
model.add(Conv1d(1, kernel_size=3, activation='relu', input_shape=(None, 1)))
model.add(Dense(1)) model.add(Activation('sigmoid'))
model.summary()
model.compile(loss='categorical_hinge', optimizer='adam', metrics=['accuracy'])
model.fit(x=X_train,y=Y_train, epochs=200, batch_size=20)

The current code works, but the Conv1d layer doesn't seem to change the accuracy of the model. How can I make the model get above the 20% accuracy it currently reaches and utilise the Conv1d layer?


texts:
I read in a blog that the decision tree has this disadvantage:

Not fit for continuous variables

If this is true, then why?


texts: How to save and test CNN model on test set after training

texts:
My CNN model is trained on the training set and validated on the validation set, now I want to test it on test set, here is my code:

x_img = tf.placeholder(tf.float32, name='x_img')
y_label = tf.placeholder(tf.float32, name='y_label')
reshape = tf.reshape(x_img, shape=[-1, img_x, img_y, img_z, 1], name='reshape')

def CNN_Model(input):
    conv1 = conv_layer(reshape, num_channels, n_f_conv1, name="conv1")
    max_pool1 = maxpool_layer(conv1, name="max_pool1")
    conv2 = conv_layer(max_pool1, n_f_conv1, n_f_conv2, name="conv2")
    max_pool2 = maxpool_layer(conv2, name="max_pool2")
    shape = 4*4*4*64
    flattened = tf.reshape(max_pool2,shape=[-1, shape], name='flattened')
    fc = fc_layer(flattened, shape, n_node_fc, name="fc")
    dropout1 = dropout(fc, keep_rate, name="dropout1")
    output_layer = output(dropout1, n_node_fc, num_classes, name="output_layer")
    return output_layer

def train_CNN(input):
    train_predict = CNN_Model(x_img)
    with tf.variable_scope("cross_entropy", reuse=tf.AUTO_REUSE):
        lose = tf.nn.softmax_cross_entropy_with_logits_v2(logits=train_predict, labels=y_label, name='cross_entropy')
        cost = tf.reduce_mean(lose, name='reduce_mean_cost')
        tf.summary.scalar("cost", cost)

    with tf.variable_scope("optimization", reuse=tf.AUTO_REUSE):
        optimizer = tf.train.AdamOptimizer(learning_rate, name='AdamOptimizer').minimize(cost)

    init = tf.global_variables_initializer()
    print("Starting session...")
    with tf.Session() as sess:
        sess.run(init)
        all_time = 0
        batch_size = 120
        batch = 0
        print("Starting training...")
        for epoch in range(num_epochs):
            train_batch = train_data[batch:batch_size]
            batch += batch_size
            batch_size += batch_size
            start_time = time.time()
            ep_loss = 0
            for data in train_batch:
                X = data[0]
                Y = data[1]
                _, c = sess.run([optimizer, cost], feed_dict={x_img: X, y_label: Y})
                ep_loss += c
            end_time = time.time()
            all_time += int(end_time-start_time)
            print('Epoch', epoch+1, 'completed out of',num_epochs,'loss:',ep_loss,
                  'time usage: '+str(int(end_time-start_time))+' seconds')

        correct_predict = tf.equal(tf.argmax(train_predict, 1), tf.argmax(y_label, 1))
        accuracy = tf.reduce_mean(tf.cast(correct_predict, tf.float32), name='reduce_mean_acc')
        print("Validation accuracy:", accuracy.eval({x_img:[i[0] for i in validate_data],
                                               y_label:[i[1] for i in validate_data]}))
        print("Test accuracy:", accuracy.eval({x_img:[i[0] for i in test_data],
                                               y_label:[i[1] for i in test_data]}))

I have a test dataset stored as test_data like train_data in the code above, tried to do it in more than one way, but I did not succeed, can anyone share a testing code with me, ofcourse based on my code?


texts: What are the differences and advantages of TensorFlow and Octave for machine learning?

texts:
I have been exploring the different libraries and languages you can use in order to implement machine learning. During this, I have stumbled upon a library TensorFlow and Octave(a high-level programming language) as both are intended for numerical computations.

What are the differences and advantages of using either?


texts: Create a prediction formula from data input

texts:
I have an algorithm which have as an input about 20-25 numbers. Then in every step it uses some of these numbers with a random function to calculate the local result which will lead to the final output of A, B or C.

Since every step has a random function, the formula is not deterministic. This means that with the same input, I could have either A, B or C.

My first thought was to take step by step the algorithm and calculating mathematically the probability of each output. However, it is really difficult due to the size of the core.

My next thought was to use machine learning with supervised algorithm. I can have as many labeled entries as I want.

So, I have the following questions:

How many labeled inputs should I need for a decent approach of the probabilities? Yes, I can have as many as I want, but it needs time to run the algorithm and I want to estimate the cost of the simulations to gather the labeled data.
Which technique do you suggest that works with so many inputs that can give the probability of the three possible outputs?
As an extra question, the algorithm run in 10 steps and there is a possibility that some of the inputs will change in one of the steps. My simple approach is to not include this option on the prediction formula, since I have to set different inputs for some of the steps. If I try the advanced methods, is there any other technique I could use?


texts:
I have two pandas dataframes that look about the same but with different information stored in them. My question will be about how to compare the two dataframes to ensure column and row match before performing some analysis and to obtain a third dataframe of the correlation between the two.

df1 (50x14492):

TYPE GENRE1 GENRE2
Name1 .0945 .0845
Name2 .9074 Nan
Name3 1     0

and df2 (50x14492):

TYPE GENRE1 GENRE2
Name1 .9045 .895
Name2 .074  1
Name3 .5    .045

Hoped for result df3 that is as yet unobtained (50x14492):

TYPE  GENRE1                 GENRE2
Name1 spearsonr(.9045,.9045) spearsonr(.0845,.895)
Name2 spearsonr(.9074,.074)  spearsonr(Nan,1)
Name3 spearsonr(1,.5)        spearsonr(0,.045)

I'd like to compare each row of each column (df1.GENRE1.Name1 to df2.GENRE1.Name1, df1.GENRE1.Name2 to df2.GENRE1.Name2, etc.) but am getting lost in the implementation. In order to try to do this I have the following code:

for key1, value1 in df1.iteritems():
    for key2, value2 in df2.iteritems():
        if key1 == key2:
           # this gets me to df1.GENRE1 == df2.GENRE1
           for newkey1, newval1 in key1.iterrows():
               for newkey2, newval2 in key2.iterrows():
                   if newkey1 == newkey2:
                      # this does not seem to get me to df1.GENRE1.Name1 == df2.GENRE1.Name1
                      scipy.stats.spearmanr(newval1, newval2)

This is allowing me to get as far as to compare the coulmns (df1.GENRE1 and df2.GENRE1 match) but I am not sure how to get to the next logical step of also ensuring that the rows match (df1.GENRE1.Name1 == df2.GENRE1.Name1). Another way to put it, I am unsure of how to ensure the rows match now that I have the columns.

NOTE:
I have tried to use spearmanr on the full two dataframes as such:

corr, p_val = scipy.stats.spearmanr(df1, df2, axis=0, nan_policy='omit')

but rather than getting a new dataframe of the same size (50x14492, telling me that each value in the dataframe was evaluated with spearmanr) I am getting a table back that's 100x100.

Similarly if I use:

corr, p_val = scipy.stats.spearmanr(df1['GENRE1'], df2['GENRE1'], axis=0, nan_policy='omit')

I get the correlation of the two columns as a whole (1x1, rather than each row of that column, which would be of size 1X14492)


texts: TensorFlow: Cannot save trained model (AssertionError)

texts:
My model uses pre-processed data to predict if a customer is a private or a non-private customer. The pre-processing-step is using steps like feature_column.bucketized_column(…), feature_column.embedding_column(…) and so on.
After the training I am trying to save the model but I get the following error:

AssertionError: Tried to export a function which references untracked object Tensor("14867:0", shape=(), dtype=resource).TensorFlow objects (e.sg. tf.Variable) captured by functions must be tracked by assigning them to an attribute of a tracked object or assigned to an attribute of the main object directly.

Here is the relevant code:
(feature_columns, train_ds, val_ds, test_ds) = preprocessing.getPreProcessedDatasets(args.data, args.zip, args.batchSize)

feature_layer = tf.keras.layers.DenseFeatures(feature_columns, trainable=False)

model = tf.keras.models.Sequential([
        feature_layer,
        tf.keras.layers.Dense(1, activation=tf.nn.sigmoid)
    ])

model.compile(optimizer='sgd',
        loss='binary_crossentropy',
        metrics=['accuracy'])

...

 model.fit(train_ds,
              validation_data=val_ds,
              epochs=args.epoch,
              callbacks=[tensorboard_callback])

 model.summary()

 if args.saveModel:
        filepath = "./saved_models/logReg" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + "-e{}-b{}-a{}".format(args.epoch, args.batchSize, accuracy)
        model.save(filepath)

I also tried this: Can't save save/export and load a keras model that uses eager execution without success.


texts:
I have built a classification model that I have to refit.

New instances have been added to the former data set on which the model was build.

I was thinking to proceed that way :

split my new data set in 3 sub data sets : ( train , test_01 ) data sets to refit the model , ( test_02 ) test data set to compare the original model and the refitted one ( on the metric formerly used ).

Of course the ( train , test_01 ) data sets have to contain the original ones ( those used to fit the former model before being refitted ).

Is this the correct way to proceed?


texts: Does it make sense to train a convolutional neural network on lo-res, use on hi-res pictures?

texts:
this is my first machine learning project and actually also my first question here. I am a novice to machine learning with a background in theoretical physics.

I want to use a CNN to detect scratches from manufacturing defects in hi-res images. The images have to be hi-res; downsampling would destroy the very features we want to detect. Also, since we are specifically looking for scratches in certain parts of the product, we cannot simply divide the picture into many tiles and treat each pile - the network will have to "understand" both the small-scale structure (single scratch) as well as the large-scale structure (entire product) of the picture.

I'm concerned about limited model capacity (no of parameters) as well as training time. Also, since the number of training pictures are few, it does not seem to make sense to train a whole network from scratch just using my product pictures.

So, my question is, do you think the following approach would work:
* use a pre-trained (on general lo-res pictures) off-the-shelf CNN (possibly the CNN layers of VGG19?), followed by a pretty radical max-pooling (for downsizing the picture without introducing any further parameters), and then adding 1 or 2 fully connected layers with further successively lower resolution, and training ONLY those last layers using my pictures?

PS. btw I am planning to use Keras as a platform.


texts: How can we use machine learning to distnguish between similarly looking images

texts:
How can I build a model which can distinguish between Milk and Phenyl? I want to predict whether a given item is edible to eat or not. If I train a model with thousands of photos of Milk and Phenyl which are labelled, Won't the model get confused between them because they look very very similar? No matter how much I train, I will be getting poor results, won't I? Another such scenario is in the case of salt and sugar.

I want to know if there is a way to tackle this situation. Please let me know if I can achieve better results with any other approach.

Also Please let me know If I'm unclear.


texts: Is this enough information to fully define a neural network?

texts:
Background:

A recent paper describes using machine learning to predict the outcome of chemical reactions. In the supplementary information (see pages S77-S78), the neural network used is described.

I would like to try implementing this network as a first project for learning neural networks and TensorFlow. However, I don't know enough to tell whether the description in the supplementary information is complete.

Question:

Does the supplementary information (see pages S77-S78 only) provide enough information to implement the network? Or did the author leave a lot out (perhaps on purpose)?

Description of the data for non-chemists:

The paper considers one specific type of chemical reaction. For this reaction, you have to combine five types of species: molecule 1, molecule 2, a catalyst, a base, and a solvent to dissolve everything. For each species, there are multiple options. You could use a water-based or an alcohol-based solvent, for example.

There are thousands of possible combinations for this chemical reaction. We have data that pairs each combination with the reaction yield, a number between 0 and 1. We have a bunch of vectors, basically, and we want to predict the yields of new combinations.

Reference:

The recent paper.


texts: Obtaining frequently occurring phrases using Word2Vec

texts:
I'm rather new to Word2Vec, having started working on it about a week ago.
My question is this: Is there a way to obtain frequently occurring phrases in a large document using Word2Vec along with a score to denote the 'frequency'?


texts:
We have MSE and RMSE as evaluation metrics for regression problems. I have for some problems people use Weighted Mean Squared Error (WMSE) as the evaluation metrix.

Below is the WMSE formula:

Can anyone explain me the real need of WMSE and when not to use MSE.

Thanks in advance.


texts: Predict sinus with keras feed forward neural network

texts:
I have a very simple feed forward neural network with keras that should learn a sinus. Why is the predictive power so bad and what is generally the best way to pinpoint issues with a network?

In the code below, I have one input neuron, 10 in the hidden layer, and one output. I would expect the network to perform much more accurately.

import numpy as np
from keras.layers import Dense, Activation
from keras.models import Sequential

x = np.arange(100)
y = np.sin(x)

model = Sequential([
    Dense(10, input_shape=(1,)),
    Activation('sigmoid'),
    Dense(1),
    Activation('sigmoid')
])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['accuracy'])
model.fit(x, y, epochs=10, batch_size=1)

scores = model.evaluate(x, y, verbose=0)
print("Baseline Error: %.2f%%" % (100-scores[1]*100))

print(model.predict(np.array([.5])))

Output:

  1/100 [..............................] - ETA: 0s - loss: 1.2016 - acc: 0.0000e+00
 79/100 [======================>.......] - ETA: 0s - loss: 0.4665 - acc: 0.0127
100/100 [==============================] - 0s - loss: 0.5044 - acc: 0.0100
Baseline Error: 99.00%
[[ 0.35267803]]


texts: MLE for Poisson conditioned on multivariate Gaussian?

texts:
I am writing some Python code to fit 2D Gaussians to fluorescent emitters on a dark background to determine the subpixel-resolution (x, y) position of the fluorescent emitter. The crude, pixel-resolution (x, y) locations of the pixels are stored in a list xy. The height of the Gaussian represents the predicted pixel intensity at that location. Each 2D Gaussian has 5 parameters, and my end goal is to find the optimal value of those 5 parameters for each peak using MLE.
The number of photons that arrive at each pixel can be assumed to be Poisson-distributed (discrete random variable). The parameter of this Poisson random variable is equal to the height of the 2D Gaussian at that pixel, i.e. the Poisson random variable is conditioned on the multivariate Gaussian. I've been thinking for a few days now, but I can't figure it out - what would an MLE algorithm for a Poisson conditioned on a multivariate Gaussian look like? My main point of confusion is that the Poisson random variable has some optimal parameter, but that just gives me the height of the 2D Gaussian at that pixel, and I want the Gaussian parameters that would give me that optimal height.
def twoDGauss(xy, x_guess, y_guess, size_guess, count_guess, background_guess):
    x, y = xy[:, 0], xy[:, 1]
    psf = count_guess/(2 * np.pi * size_guess ** 2)
    psf *= np.exp(-((x - x_guess) ** 2 + (y - y_guess) ** 2)/(2 * size_guess ** 2))
    psf += background_guess
    return psf

That's my code defining the 2D Gaussian so far. I'm following the ideas in Ovesny et al, 2014, specifically in the supplementary section, if anyone is interested.


texts: Split large dataset for predictive modeling using rsparkling -sparklyr

texts:
I am trying to build machine learning models (GBM, RF, Staking)
on top of a dataset that is about 3G in size on my local computer.
However, I only have 4G memory (only 2G are available).

My question is : is it logical to split the whole data on 20% for
training set, 10% on validation set, and the 70% for the testing part?
I split also the test set on 7 equal subsets with same distributions.
I am doing this due to the fact I cannot test a model on full dataset.

Still I am not really convinced about this solution, and I am not sure
that it is good enough to get a robust final model.
 What can I do? I am new to machine learning and big data.


texts:
I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest.

Below you can find the correlation among all (independent and dependent) variables. We can quickly observe that only "GRE Score", "TOEFL Score" and "CGPA" considerably affect the "Chance of Admit" variable. So, it makes sense to eliminate all other variables from the predictive model.

Now among the "GRE Score", "TOEFL Score" and "CGPA" variables, we can see that they are all highly corellated (this also makes sense in real life as you always expect a good student to get good grades in these tests). I cannot decide which variables to keep for my final model. Could I keep all of them ? or how do I decide which one to exclude ?

Any help is appreciated.


texts: Is there a good systematic approach to explore and analyze data (prior to modelling)?

texts:
I have found a few examples of kernels on Kaggle people have made where they seem to follow a certain methodology in order to systematically analyze and explore the data, to make sure to find all outliers, missing values etc. They are just practical examples and leave quite a few questions.
I would just assume there must be one, or a few, recipes/methodologies/flowcharts that I can learn/use in order to have a good checklist and systematically work my way through my data and catch all data quality issues as well as note them in a well defined way. I.e. 1. Do this, 2. Do that, if result A do this, if result B note that for later this way 3. Do this etc.
What should I look into in order to find something like this? I can't imagine this not existing. It seems very much like Data Science Course 101 (which I am currently taking but it seems like a lousy course) but I can't find anything.
Any references to YouTube-videos or Coursera-courses that explain this would be welcome as well.


texts: Given Youtube Videos, is there a face recognition technology to detect faces in all new videos?

texts:
Given Youtube Videos, is there a face recognition technology to detect faces in all new videos and categorize them in a database for future use?

My research showed that especially rate limits were a problem with the api. Apart from that, given the huge amount of videos youtube uploads, processing power is another.


texts: XGBoost evaluation metric unbalanced data - custom eval metric

texts:
I have built a model using the xgboost package (in R), my data is unbalanced (5000 positives vs 95000 negatives), with a binary classification output (0,1).

I have performed cross validation with the evaluation metric AUC Area under the ROC curve which I now believe to be wrong since this is better used for balanced data sets.

I analyse the final results of the model using the Area Under the Precision Recal curve (AUPRC) and the Matthews Correlation Coefficient (MCC), however I now believe that I should have been evaluating the cross validation models with the AUPRC and MCC also and completely forget about the AUROC.

I cannot find much in the literature which uses CV with the evaluation metric of AUPRC and MCC.

I just want to make sure that I am thinking correctly and that my previous evaluation method is wrong and the AUPRC / MCC would be a better way to go.


texts: What is the difference between Permutation feature importance vs. RandomForest feature importance? What are the disadvantages vs. advantages of the two techniques?


texts: What is this called by gathering the meaning from a sentence?

texts:
What would this process of gathering the meaning of a sentence be called? What would the segments derived from the sentence be called?

"John and Derrek both love cake"
    -> John loves cake
    -> Derrek loves cake

"John was mad that the weather was rainy today"
     -> John was mad
     -> weather was rainy today


texts:
I want to run several Association rule mining techniques such as Apriori, Eclat and FP Growth. I want to know the format of the data to run these algorithms as they are disabled (marked in grey)to me.

Also I don't see the algorithm Eclat in the 'Association' tab of Weka. Please recommend me a suitable tool/approach to perform this.


texts: How to determine the complexity of an English sentence?

texts:
I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small research in a classroom of 60 students.

I have mined over hundred thousand sentences from Wikipedia for various English words (Including Barrons'800 words and 1000 most common English words)

Entire data is available at https://buildmyvocab.in

In order to maintain the quality of content, I filtered out sentences which were longer than 160 characters since they might be difficult to understand.

As a next step, I want to be able to automate the process of sorting this content in the order of ease of understanding. I myself am a non-native English speaker. I want to know what features I can use to separate easy sentences from difficult ones.

Also, do you think this is possible?


texts: How to apply Content-based image retrieval for scanned images?

texts:
Is there a way to differentiate between scanned images(with only text) that are well-lit and with good contrast, from the ones that are not(that have poor quality), making use of Content-based image retrieval? More exactly, is there a feature extraction method that can help to query a database to retrieve the well-lit, good contrast, scanned images?

Thank you!


texts: Non-convex feasible region in a linear programming problem

texts:
Problem:  I am working on a linear programming problem, i.e. a linear objective function to minimize:

$\mathbf{c}\cdot\mathbf{x}$,
where $\mathbf{c},\mathbf{x}\in\mathbb{R}^{N}$

Subject to constraints:

$\mathbf{x}\geq 0$ and $\mathbf{A}\mathbf{x} \leq \mathbf{b}$
where $\mathbf{A}\in\mathbb{R}^{MxN}$ and $\mathbf{b}\in\mathbb{R}^{M}$

I am using CyLP and the COIN-OR simplex solver to perform the optimization.

Question: A feature I would like to be able to build into
the problem is the ability to handle some atypical constraints:

Suppose $x_{i}$ is a component of $\mathbf{x}$. I want to apply the following constraints:

$x_{i} = 0$ or $\mathrm{lb} \leq x_{i} \leq \mathrm{ub}$.

In geometric terms, I want $x_{i}\in [0]\cup [\mathrm{lb}, \mathrm{ub}]$.
Unlike the original problem, where all $x_{i}$ are bounded within convex sets (i.e. non-piecewise intervals).

Is there a good way to modify an LP for the above atypical constraint?


texts: Validity of PU learning while using character-level encoding using CNNs for classifying text data

texts:
I'm trying to classify a large set of documents (~100M) as valid or invalid, based upon a small given set of labeled valid documents (~3k). I'd like to know if the PU learning approach described in this paper combined with the character-level string encoding described in this paper is likely to give good results or not, and why.

Some preliminary info about the dataset (ignore if you want to skip ahead to the model):

Each document has <= 50 words.
Each word has <= 50 characters.
Each character has <= 50 words.
Any randomly selected document pair is likely to have very few common words.
There are a significant number of proper nouns in the dataset that have to be captured.
Original set for training the network contains ~10k unlabeled documents and ~3k labeled ones. The assumption is that the 10k unlabeled documents selected will be mostly different from the 3k labeled ones.
Train-test-validation split is 80%-10%-10%.

The first goal is to write a classifier that distinguishes labeled and unlabeled documents. For completeness' sake, the code for this classifier (written using Keras) is as follows:

def cnn_model(input_batch_shape):
  input_layer = layers.Input(batch_shape=input_batch_shape)
  fixed_length_letter = layers.Dense(16, activation="relu")(input_layer)
  reshaped_fixed_length_letter = layers.Reshape((keras.backend.int_shape(fixed_length_letter)[1], keras.backend.int_shape(fixed_length_letter)[2], keras.backend.int_shape(fixed_length_letter)[3], 1))(fixed_length_letter)
  conv_word_1 = layers.Conv3D(16, (1, int(keras.backend.int_shape(reshaped_fixed_length_letter)[2]/5), keras.backend.int_shape(reshaped_fixed_length_letter)[3]), activation="relu")(reshaped_fixed_length_letter)
  conv_word_1 = layers.MaxPooling3D((1,5,1))(conv_word_1)
  conv_word_2 = layers.Conv3D(16, (1, int(keras.backend.int_shape(conv_word_1)[2]/4), keras.backend.int_shape(conv_word_1)[3]), activation="relu")(conv_word_1)
  conv_word_2 = layers.MaxPooling3D((1,4,1))(conv_word_2)
  conv_sentence_1 = layers.Conv3D(8, (int(keras.backend.int_shape(conv_word_2)[1]/5), 1, 1))(conv_word_2)
  conv_sentence_1 = layers.MaxPooling3D((5, 1, 1))(conv_sentence_1)
  conv_sentence_2 = layers.Conv3D(8, (int(keras.backend.int_shape(conv_sentence_1)[1]/4), 1, 1))(conv_sentence_1)
  conv_sentence_2 = layers.MaxPooling3D((4, 1, 1))(conv_sentence_2)
  flattened_sentence = layers.Flatten()(conv_sentence_2)
  output_layer = layers.Dense(1, activation="softmax")(flattened_sentence)
  model = models.Model(inputs=input_layer, outputs=output_layer)
  return model

model_check_labeled = cnn_model((16, 50, 50, 39))
model_check_labeled.compile(optimizers.Adam(), losses.binary_crossentropy, metrics=['accuracy'])
model_check_labeled.summary()

As the code shows, I've dropped the part of Dos Santos and Gatti de Bayser's proposed architecture where they encode the words into vectors as well, since I don't think my vocabulary of words is large enough to generate a good encoding with the number of valid samples I have.

This led to the following accuracy and loss curves:

As the graph shows, my classifier gave me ~25% accuracy - meaning that there have been no meaningful features captured at all. I would like to know if this is expected, given the architecture of my classifier and the dataset I have described. I would also like to know if there are other approaches I could use that would help at least classify new documents as labeled or unlabeled.

Please tell me if the description of the dataset sounds too vague. I'll try and present some examples that have similar features.

Papers cited:

Elkan, Charles & Noto, Keith. (2008). Learning classifiers from only positive and unlabeled data. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 213-220. 10.1145/1401890.1401920.
Dos Santos, Cicero & Gatti de Bayser, Maira. (2014). Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts.


texts:
I have music play data organized by the day on which each track was played, from March 1st, 2015 to August 30th, 2015. The data set contains count data for every day a song was played. I'd like to predict play counts for each track for an out-of-time window, from 2015-09-01 to 2015-09-30.
Details of the data
The data is from Alibaba's TianChi Competition
Action Table (mars_tianchi_user_actions)

Any user’s action data for a song is indicated by a unique row
Songs Table（mars_tianchi_songs）

Result Set
The participants need to predict the artists’ plays data in the following two months (20150901-20151030).
Participant’s Result Table (mars_tianchi_artist_plays_predict)

For song id daa234f183aee2373d20987b247cd768(all the song ids are hash values). The play plot is:

As you can see it, the variance of each day's play for this track is quite large.
I also have data on who played each track, with each user ids represented as a hash value.
I'd like some guidance on which analysis method I could use here. I am considering time series analysis.


texts:
I've built a Decision Tree Classifier to practice with the sklearn library. My first task was to shuffle the iris dataset and split it keeping only the last 10 elements for the test. Then, after the training part I predicted the class of these elements and printed other useful metrics to understand what I'm doing. My doubt comes after building the roc curve, since I find it is really different from the other I'v seen in some example. This is the code:
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt
from sklearn import tree
import numpy as np
import graphviz

iris = load_iris()

clf_ex1 = tree.DecisionTreeClassifier(criterion="entropy",random_state=300,min_samples_leaf=5,
                                      class_weight={0:1,1:10,2:10})

np.random.seed(0)

indices = np.random.permutation(len(iris.data))
indices_training=indices[:-10]
indices_test=indices[-10:]

iris_X_train = iris.data[indices_training]
iris_y_train = iris.target[indices_training]
iris_X_test  = iris.data[indices_test]
iris_y_test  = iris.target[indices_test]

clf_ex1 = clf_ex1.fit(iris_X_train, iris_y_train)

predicted_y_test = clf_ex1.predict(iris_X_test)

print("Predictions:")
print(predicted_y_test)
print("True classes:")
print(iris_y_test)

# print some metrics results
acc_score = accuracy_score(iris_y_test, predicted_y_test)
print("--------")
print("Accuracy score: "+ str(acc_score))
print("--------")
f1=f1_score(iris_y_test, predicted_y_test, average='macro')
print("F1 score: "+str(f1))
print("--------")

scores = cross_val_score(clf_ex1, iris.data, iris.target, cv=5)
print("Cross validation scores: "+str(scores))

# Confusion Matrix
print("--------")
print("Confusion matrix:")
print(confusion_matrix(iris_y_test, predicted_y_test))

# Building the ROC Curve
y_test_prob = clf_ex1.predict_proba(iris_X_test)

# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 0)
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 1)
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 2)
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

# Building the Plot for each class
plt.figure()
lw = 2
plt.plot(fpr_cl0, tpr_cl0, color='darkorange',
         lw=lw, label='ROC curve class 0 (area = %0.2f)' % roc_auc_cl0)
plt.plot(fpr_cl1, tpr_cl1, color='cornflowerblue',
         lw=lw, label='ROC curve class 1 (area = %0.2f)' % roc_auc_cl1)
plt.plot(fpr_cl2, tpr_cl2, color='aqua',
         lw=lw, label='ROC curve class 2 (area = %0.2f)' % roc_auc_cl2)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic example')
plt.legend(loc="lower right")
plt.show()

And these are the results:

Are they consistent with the predictions?
If I change the weights of the classes in my DecisionTreeClassifier, but I get the same predictions, is it normal that the final plot do not changes?


texts: What does big O mean in KNN optimal weights?

texts:
Wiki gives this definition of KNN

In pattern recognition, the k-nearest neighbors algorithm (k-NN) is a
non-parametric method used for classification and regression. In both
cases, the input consists of the k closest training examples in the
feature space. The output depends on whether k-NN is used for
classification or regression:

In k-NN classification, the output is a class membership. An object    is classified by a plurality vote of its neighbors, with the object
being assigned to the class most common among its k nearest neighbors
(k is a positive integer, typically small). If k = 1, then the object
is simply assigned to the class of that single nearest neighbor.
In k-NN regression, the output is the property value for the object.    This value is the average of the values of k nearest
neighbors.

k-NN is a type of instance-based learning, or lazy learning, where the
function is only approximated locally and all computation is deferred
until classification.
Both for classification and regression, a useful technique can be to
assign weights to the contributions of the neighbors, so that the
nearer neighbors contribute more to the average than the more distant
ones. For example, a common weighting scheme consists in giving each
neighbor a weight of 1/d, where d is the distance to the neighbor.

and this explanation about "The weighted nearest neighbour classifier"

The k-nearest neighbour classifier can be viewed as assigning the k
nearest neighbours a weight 1/k and all others 0 weight. This can be
generalised to weighted nearest neighbour classifiers. That is, where
the ith nearest neighbour is assigned a weight ${\displaystyle
> w_{ni}}$, with ${\displaystyle \sum _{i=1}^{n}w_{ni}=1}$. An analogous
result on the strong consistency of weighted nearest neighbour
classifiers also holds.
Let $C_{n}^{wnn}$ denote the weighted nearest classifier with weights $\{w_{{ni}}\}_{{i=1}}^{n}$.
Subject to regularity conditions on the class distributions the excess risk has the following asymptotic expansion
${\mathcal  {R}}_{{\mathcal  {R}}}(C_{{n}}^{{wnn}})-{\mathcal  {R}}_{{{\mathcal  {R}}}}(C^{{Bayes}})=\left(B_{1}s_{n}^{2}+B_{2}t_{n}^{2}\right)\{1+o(1)\},$

and this formula

With optimal weights the dominant term in the asymptotic expansion of the excess risk is ${\mathcal  {O}}(n^{{-{\frac  4{d+4}}}})$

Does $\mathcal  {O}$ here mean the Big O notation or something else?


texts: Data.table package is claimed to be faster than Data.frame. what are the implementation changes that made this possible? How can one leverage the power of this package for data analysis?


texts: How to set LSTM for train on multiple time series on keras?

texts:
Greeting, everyone! I have $X = (x_1,x_2,...,x_n)$, and $Y = (y_1,y_2,...,y_n)$ where $x_i$ - is a time series, and $y_i$ - is the output time series for $x_i$. e.g. for $x_{ik}$ ($i$ - index of time series, $i = 1,n$, $k$ - number of pattern(idk how it's named), $k=1,t$, $t$ - size of $x_i$), I have $y_{ik}$, I'll pray that you understand me))

What I want to do: I want to train LSTM-based NN, for predicting $X$ to $Y$. As you seen, for each $x_ik$, I have $y_ik$, so I think that it will be great, if NN will learn on whole time series, not just on the last point, question is how I should set my NN in Keras?

For the end, once again, I try to explain:
$x_i = (x_{i1}, x_{i2},...,x_{ik})$ and output is $y_i = (y_{i1}, y_{i2},...,y_{ik})$, but for now I train my model with $x_i = (x_{i1}, x_{i2},...,x_{ik})$ and $y_i = y_{ik}$ - so it's just one last point.


texts: How to tell if the “clusters” I see in my pair plots are statistically significant or occurring by random chance?

texts:
I have a data set with one row per subject. Some variables include laboratory parameters for blood chemistry, hematology, etc. I also have some flag variables: any = 1 if the subject experienced an adverse event, 0 if not; and ser_flag = 1 if the subject experienced a serious adverse event, 0 if not.

There doesn't seem to be any difference in the distribution of laboratory parameters between subjects who experienced an adverse event (any=1) and subjects who did not (any = 0). When I do a pair plot of all the lab parameters against each other and color by the any flag, there doesn't seem to be any clustering or separation of subjects.

However, when I do the same pair plot and color by ser_flag - I notice that the 20 subjects who experienced a serious adverse events seem to be clustered together in many of the plots.

What test (if any) can I use to determine if these clusters I think I am seeing are occurring randomly, by chance...or if they statistically significant?


texts:
As is clear from the figure, the blue points, which don't follow the trend, are anomalous points.

I'm wondering about the best non-parametric method to detect those points. I have tested some outlier detection methods such as standard deviation, etc. but they don't provide good results while it is clear from the figure.


texts: Encode time-series of different lengths with keras

texts:
I have time-series as my data (one time-series per training example). I would like to encode the data within these series in a fixed-length vector of features using a keras model.
The problem is that my different examples' time-series don't have the same lengths. I haven't found a way of doing that. The problem of the encoder-decoder thing is that if the input lengths vary, the output lengths do this also. But I would like to have an output of length 10 let's say that would encode the data of the time-series, regardless of its length.
My question could have a strong link with: https://deepai.org/publication/towards-a-universal-neural-network-encoder-for-time-series but I haven't found any practical implementation.
Any help is very appreciated.