texts: Why not use Scaler.fit_transform on total dataframe?

texts:
In sklearn I'm normalizing the data with MinMaxScaler. The example I'm following uses

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

X_train, X_test, y_train, y_test = train_test_split(X_crime, y_crime,random_state = 0)

X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Now I wonder why this is done separately on the train and test set, and not on the X_crime dataframe like:

X_crime_scaled = scaler.fit_transform(X_crime)

X_train_scaled, X_test_scaled, y_train, y_test = train_test_split(X_crime_scaled, y_crime, random_state = 0)

the R-squared score is higher and with this option I know all my values are normalized between 0 and 1.


texts: Initialising states in a multilayer sequence to sequence model

texts:
With a sequence to sequence model where the enocoder and decoder are both comprised of one layer each, the initial state of the decoder is initialised to use the final states of the encoder layer.

In the case of a multi-layer sequence to sequence model where there are many layers in the encoder and the decoder, should every layer in the decoder be initialised with the final state of the encoder or just the first layer of the decoder and why?


texts: How do you calculate sample difference in terms of sensor signals?

texts:
A paper I read called Preprocessing Techniques for Context Recognition from  Accelerometer Data refers to sample difference as the delta value between signals in a pairwise arrangement of samples that allows a basic comparison between the intensity of user activity.

How would you do the pairwise arrangement? Would it require you to have different files of data representing different classes?

For example, I have a CSV:

 1495573445.162, 0, 0.021973, 0.012283, -0.995468, 1
 1495573445.172, 0, 0.021072, 0.013779, -0.994308, 1
 1495573445.182, 0, 0.020157, 0.015717, -0.995575, 1
 1495573445.192, 0, 0.017883, 0.012756, -0.993927, 1

where the second, third, and fourth columns are the axes of accelerometer data.

I have several files named for one gesture and several others for another gesture and would like to use this sample difference statistic to help classify the data.

Also as a secondary question, this was listed as a preprocessing technique but it sounds like it's more of a feature. Could I get clarification on that as well?


texts: to generate consistent encoding for words in Keras using tf.keras.preprocessing.text.one_hot

texts:
I am using keras(tensorflow) to convert text into encodings using tensorflow.keras.preprocessing.text.one_hot
I have used it for training dataset as below
from tensorflow.keras.preprocessing.text import one_hot

corpus = ['nice app']
onehot_repr = [one_hot(words, 10000) for words in corpus]

print(onehot_repr)
# [5779, 2969]

It's ok upto this point.
But when I use the one_hot for my testing set it generates different encoding.
I have created a Flask API to test, So how can use same encoding for both train and test set
Result from API is :
[[5129, 4965]] for same text ['nice app']


texts:
I am working on a binary classification problem. The target variable is not linearly separable, so I've decided to use LightGBM with default parameters (I only play with n_estimators on range from 10 - 100).

When I output Gain (feature importance for LightGBM) it has extremely high values on the x-axis. When I increase the number of estimators x-axis gain grows even higher. It seems to me that the model is overfitted to a single feature [1].

On the other hand, split feature importances seem to have a nice distribution without extreme values on the x-axis[2].

What is the reason for this behavior with extreme Gain values? Regularization?


texts: How is the fit function in SimpleImputer working to find the mean in the Salary column as well when just the Age column is given as its argument?

texts:
The only argument inside the fit function of SimpleImputer is: 'Age'. Yet the returned output worked on the 'Salary' column as well. That is what I am unable to understand.
Here is my code (considering all the necessary libraries imported):
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values = np.nan)
imputer = imputer.fit(df[['Age']])
X[:, 1:3] = imputer.fit_transform(X[:, 1:3])
print(X)

Dataset:
   Country   Age   Salary Purchased
0   France  44.0  72000.0        No
1    Spain  27.0  48000.0       Yes
2  Germany  30.0  54000.0        No
3    Spain  38.0  61000.0        No
4  Germany  40.0      NaN       Yes
5   France  35.0  58000.0       Yes
6    Spain   NaN  52000.0        No
7   France  48.0  79000.0       Yes
8  Germany  50.0  83000.0        No
9   France  37.0  67000.0       Yes

Output:
[['France' 44.0 72000.0]
 ['Spain' 27.0 48000.0]
 ['Germany' 30.0 54000.0]
 ['Spain' 38.0 61000.0]
 ['Germany' 40.0 63777.77777777778]
 ['France' 35.0 58000.0]
 ['Spain' 38.77777777777778 52000.0]
 ['France' 48.0 79000.0]
 ['Germany' 50.0 83000.0]
 ['France' 37.0 67000.0]]


texts:
I'm trying to understand the connection between loss function and backpropagation.
From what I understood until now, backpropagation is used to get and update matrices and bias used in forward propagation in the LSTM algorithm to get current cell and hidden states.
And loss function takes the predicted output and real output from the training set.
But which part is the training part of the LSTM?
Are they somehow connected ?
What is the training objective for the LSTM model?


texts: Difference between "reducing batch_size" and "increasing epochs" to decrease loss amount?

texts:
In my experience, both reducing batch_size and increasing epochs can decrease loss amount. But I like to know is there any difference to using which one? Has decreased loss amount same meaning and it's not important you reached it with what way(I mean has no effects on results)?

For ex, I got same loss amount 2.5e-4 with both the following case:

1. batch_size = 1 , epochs = 100
2. batch_size = 60 , epochs = 1000

Are they same result?


texts: Remove attributes with missing values exceeding a given threshold in WEKA

texts:
I imported csv file into WEKA, i have features that have missing value that has missing value percentage of 70% or above, i want to remove these features by weka or also sort that features by missing value percentage desc and then remove some features manually.

How can i do that? i'm new to machine learning and WEKA.


texts: How to compute the Jaccard Similarity in this example? (Jaccard vs. Cosine)

texts:
I am trying to understand the difference between Jaccard and Cosine. However, there seem to be a disagreement in the answers provided in Applications and differences for Jaccard similarity and Cosine Similarity.

I am seeking if anyone could step me through the calculations of the Jaccard Similarity in this Cosine Similarity example from https://bioinformatics.oxfordjournals.org/content/suppl/2009/10/24/btp613.DC1/bioinf-2008-1835-File004.pdf

Given:

Question: How do we compute the Jaccard Similarity index between t1 and t2?

Thank you.


texts:
I want my model batch size to be a dynamic shape, and I've assigned none as batch size, but that's causing an error.
Here, in the first line, I specified batch size as None:
inp=L.Input(shape=(28,28,1),batch_size=None)
conv1=L.Conv2D(filters=256,kernel_size=(2,2),activation='relu',padding='valid')(inp)
maxpool1=L.MaxPooling2D(pool_size=(1,1))(conv1)
conv2=L.Conv2D(filters=128,kernel_size=(9,9),activation='relu',padding='valid')(maxpool1)
conv2=L.Conv2D(filters=8*16,kernel_size=(9,9),strides=2,padding='valid',activation=None)(conv2)
# Adding the squash activation
reshape2=L.Reshape([-1,8])(conv2)
squashed_output=L.Lambda(squash)(reshape2)

CapsuleLayer code:
class CapsuleLayer(L.Layer):
    def __init__(self,num_capsule,dim_capsule,routing=3,kernel_initializer='glorot_uniform',**kwargs):
        super(CapsuleLayer,self).__init__(**kwargs)
        self.num_capsule=num_capsule
        self.dim_capsule=dim_capsule
        self.routing=routing
        self.kernel_initializer=kernel_initializer
    def build(self,input_shape):
        assert len(input_shape) >= 3
        self.input_num_capsule=input_shape[1]
        self.input_dim_capsule=input_shape[2]

        #transforming the matrix
        self.W= self.add_weight(shape=[self.num_capsule,self.input_num_capsule,self.dim_capsule,self.input_dim_capsule],initializer=self.kernel_initializer,name='w')
        self.built=True
    def call(self,inputs,training=None):
        input_expand=tf.expand_dims(tf.expand_dims(inputs,1),-1)
        inputs_tiled=K.tile(input_expand,[1,self.num_capsule,1,1,1])
        input_hat=tf.squeeze(tf.map_fn(lambda x: tf.matmul(self.W,x),elems=inputs_tiled))
        b=tf.zeros(shape=[inputs.shape[0] ,self.num_capsule,1,self.input_num_capsule])
        assert self.routing > 0
        for i in range(self.routing):
            c=tf.nn.softmax(b,axis=1)
            output=squash(tf.matmul(c,input_hat))
            if i<self.routing-1:
                b+=tf.matmul(output,input_hat,transpose_b=True)
        return tf.squeeze(output)
    def compute_output_shape(self,input_shape):
        return tuple([None,self.num_capsule,self.dim_capsule])
    def get_config(self):
        config = {
            'num_capsule': self.num_capsule,
            'dim_capsule': self.dim_capsule,
            'routings': self.routings
        }
        base_config = super(CapsuleLayer, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

The following line when CapsuleLayer function called, the error raised:
digitcaps = CapsuleLayer(num_capsule=3, dim_capsule=16, routing=3, name='digitcaps')(squashed_output)

The error:

TypeError: Expected int32, got None of type 'NoneType' instead.

When I assign a number to the batch size, no error occurred.


texts: Image Classification approach for Text images

texts:
Let me introduce myself as a beginner to machine learning problems.

We are trying to build a system to classify images of text-data like bills, orders, bank-statements etc. We started with the image classification approach with some-what larger data-set to train. This gave a good percentage of predictions in test.

But in real-world samples, the problem we see is that, there is no standard layouts  or strict rules in producing these text-documents. So the image classification is not giving good results any more. Classifications are going wrong because of common white backgrounds / logos is what we suspect.
Different attempts to gray-scale the images, remove the non-text contents etc we tried, but it does not improve much.

Since arguments to shift to OCR based recognition & classification are increasing we are having a thought towards that too.

My doubt is image classification for text-images is a right approach or not?


texts: How to add a new column with labels in a dataframe?

texts:
I have thousands of sentences that I would need to label based on their sentiment.
An example is

Testo
"Può dirmi quale fotocamera della Sony fa delle belle foto?"
"Che macchina fantastica! Ha smesso di funzionare dopo due giorni"
"Quel pallavolista ha un fisico pauroso"
"Questo iPad è fantastico"

I have tried using Fasttext (Python Package) but I do not know how to add the sentiment label in a new column. Also, I got the wrong sentiment (below it is shown the expected output).

Testo                                                            Sentiment
"Può dirmi quale fotocamera della Sony fa delle belle foto?"         0
"Che macchina fantastica! Ha smesso di funzionare dopo due giorni"  -1
"Quel pallavolista ha un fisico pauroso"                             1
"Questo iPad è fantastico"                                           1

How can I do it?


texts: Time series forecasting for stable pattern with some sudden changes

texts:
In my case, the time series is around a constant value with very small fluctuations. But, sometimes the signal starts increasing or decreasing for some duration of time. For my application, such increase/decrease can be risky if it causes the signal to cross some threshold. What I want is to forecast if such an increase/decrease will cause the signal to go beyond a predefined threshold. In other words, if the increase/decrease persisted for the next 15mins what will be the signal value, and will it exceed the threshold. I don't have data related to the cause of such increase/decrease. It is a univariate signal.
What I did is fitting exponential smoothing every 15mins based on the last 15mins data and used it to forecast what will happen in the next 15mins
I am a bit new to such topics and I don't know if what I am doing is making sense and what is the recommended way of solving such an issue


texts: Is there a difference between "classification" and "labeling"?

texts:
Until recently, I thought that "labeling" and "classification" are synonyms. But when I started another question about terminology in computer vision I thought about it: Is there a difference between "labeling" and "classification"?

I thought that the "class" is the concept you want to detect and "label" is what you assign to data. So the "class" is a concept which leads to the data and "label" is only the name. Hence "labeling" would be the same as "classification" as both want to make a statement about the underlying class which lead to the data.

Articles

A quick search via Google Scholar revealed that some articles use both terms in the title:

Markus Eich, Malgorzata Dabrowska, and Frank Kirchner: "Semantic Labeling: Classification of 3D Entities Based on Spatial Feature Descriptors"
Chunlin Li, Dmitry B. Goldgof, and Lawrence 0. Hall: "Knowledge-based classification and tissue labeling of MR images of human brain"
Ray Blanchard: "The classification and labeling of nonhomosexual gender dysphorias" - another research area but probably it is the same difference between the two words?

So I guess there is a difference between "labeling" and "classification". What is the difference?

Google N-Gram

classification seems to be a much boarder term.


texts: How to calculate the different metrics for multi class classification

texts:
My confusion matrix has the following structure:
                (Predicted)

C=   ( actual)    [TN FP
                 FN TP]

How can I calculate the Mathews Correlation Coefficient (MCC) value for multi-class expressed as
MCC = (TP .* TN - FP .* FN) ./ ... sqrt( (TP + FP) .* (TP + FN) .* (TN + FP) .* (TN + FN) );
Also, I have some doubts regarding the calculation of the following measures for multi-class. Please correct me where wrong.
for i=1:nClasses
   TN(i)=C(i,i);
    FP(i)=sum(C(i,:))-C(i,i);
    FN(i)=sum(C(:,i))-C(i,i);
    TP(i)=sum(C(:))-TP(i)-FP(i)-FN(i);
end


texts: Standardization, Oversampling and Evaluation

texts:
Please help me to understand which of the following is correct and why.

X_train, X_test : Split the training set into training and test sets
X_train_std : Standardize X_train
X_train_res : Oversample X_train_std
model.fit : Train the model using X_train_res
X_test_std : Standardize X_test using the parameters calculated in 2)
model.evaluate : Evaluate the model using X_test_std

Or:

X_train, X_test : Split the training set into training and test sets
X_train_std : Standardize X_train
X_train_res : Oversample X_train_std
X_train_res_std : Standardize X_train_res
model.fit : Train the model using X_train_res_std
X_test_std : Standardize X_test using the parameters calculated in 4)
model.evaluate : Evaluate the model using X_test_std


texts: Reinforcement learning: easily learnable state representation

texts:
I have created a simple OpenAI Gym environment, which consists of:

A continuous 2D world with x and y in range [0.0, 1.0]
A rabbit which slowly moves randomly in the world with a constant speed
A 'wolf' controlled by the agent. The wolf moves at a constant speed
The actions are [turn left by a constant angle, turn right by a constant angle, do nothing (continue straight)]
The state is [agent_x, agent_y, agent_bearing, rabbit_x, rabbit_y, rabbit_bearning]. Bearings are in radians [0.0, 2*pi]. All values are floating point numbers.
The reward is 30 for catching the rabbit (catching means the agent getting sufficiently close to the rabbit). -0.1 for each timestep without catching the rabbit.
Maximum timesteps 260

I am having trouble solving this environment. Agents trained in it get scores only slightly better than a random agent even after long training sessions. I have tried Deep Q-learning (with experience replay, target network) REINFORCE (with and without baseline) and PPO.

Conceptually, the problem is quite simple. The agent just needs to learn to turn towards the rabbit. However, it occurs to me that the state representation might make the problem more difficult, since only one of the six variables is directly under the agent's control, and three of them (the rabbit state) are completely random.

Does state representation generally affect how difficult a problem is? Is this a poor state representation? Are there rules of thumb for how to design states? Would it help to reformulating the state, eg. to [distance_from_agent_to_rabbit, angle_between_agent_and_rabbit]?

Environment source code here.


texts: Optimal proportion between the amount of Class = 1 and the amount of Class = 0?

texts:
I am quite new machine learning methods, so I may not write proper technical formulas.

My question is about the optimal proportion between sample size in Class = 1 and Class = 0 in a binary classification problem. (I might use different models, but I will start with decision tree).

I am trying to develop a risk model for a population of 300.000 by using some risk factors, and each risk factor has a risk-weight. So basically it is a accumulative model

  Risk_amount = Risk_factor_1 * weight_1 + Risk_factor_2 * weight_2 + Risk_factor_3 * weight_3 ...

My big question is if a Risk_amount really belongs to  Class = 1 or not.

I want my ML-model to find the most optimal weights to those risk factors.

I already know that 1000 out of 300.000 are risky in real. So the amount of target data with Class = 1 is 1000.

So the total amount of target data with non-risky population Class = 0 is 299.000

The question is what is the optimal proportion between the amount of Class = 1 and the amount of Class = 0?

1000 Class = 1 vs. 1000 Class = 0

or

1000 Class = 1 vs. the whole Class = 0 as 299.000?


texts: Is there any public implementation / publication with Hintons capsules idea?

texts:
In Hintons talk "What's wrong about convolutional nets" (Late 2016 or early 2015, I guess) he talks about capsules to make a modular CNN.

Is there any publicly available implementation or papers about this idea?


texts: Text classification with neural network number of input neurons

texts:
I am classifying documents I have around 4000 of them that I am trying to categorise into 5 categories. I am using a bag of words model which equates to about 18,000 unique words (features) and therefore I have an input layer of a neural network with 18,0000 inputs which doesn't seem right.

It is taking a huge amount of memory to try and train this network and so much time it will never converge!

Is there a way of reducing the number of input neurons seeing as a large portion of this data will be nulls?


texts:
I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (mainly because of the domain-specific requirements). I assume we cannot use Med7 or other English-language specific NER's for this purpose. I'd like to know the options and suggestions on how to proceed. I'd also like to know if BioBERT could come handy for this purpose, particularly by combining it with camemBERT or any other French language models.
If there're no readymade options available, I'm planning to translate French to English and then run the NER. I hope there's some potential for this approach.


texts: How hidden units are conditionally independent in restricted Boltzmann machines

texts: Why explaining away concept is not applicable in restricted Boltzmann machines? Their hidden units form a V structure from which probabilistic influence can flow given the observed visible variable. Why is this a problem in deep belief nets?


texts: Looking for a causality to effect dataset

texts:
I am looking for a causality dataset that would look like this:

animal + car -> accident
speeding -> traffic ticket
tomatoes + lettuce -> salad
virus -> illness
tooth decay -> dentist

Does this already exists?


texts: Text classification based on n-grams and similarity

texts:
I have tried to cluster hundred texts using k-means clustering. I would like to consider other algorithms to group text based on their content and try to spot news not related to other news (topic different).
I would like to know if there is some other method/algorithm that could be used to classify text into two or more groups based on their level of similarity.
Let’s say I have thousands of text that use the same keywords but in different context.
Some text may have the same sentence structure, for example:

The pen is on the table
The black pen is on the table.
Where is my pen? Your pen is on the table.
I cannot find my pen. I was using it a few minutes ago.

I would like to be able to cluster the sentences above based on their structure, i.e. using n-grams with n manually set.

In this way I should be able to include the first three sentences in the same cluster, as their level of similarity in using same/similar words is higher compared to the last sentence.
It is about compute for each sentence the similarity with others.

Do you have any idea on how I could do that?


texts: Multivariate Time Series Anomalous Entry Detection

texts:
I have a multivariate data set of the following structure.

It is a time series sequence of logs with additional string attribute columns id1 and id2. If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them. I tried using LSTM, but I'm confused in how to use it for textual data, specially when the values that can come in as inputs are not always the same (one time window might have more entries than other).

The similar entries may not always be together but may be very frequent in a specific time window.

I am going to be classifying anomalies at the end of every day in a batch, so I'd like to feed the trained model a bunch of log entries and then find anomalies in it.

I'd appreciate any help w.r.t. to what direction to look in and if LSTM is even required for this.


texts:
I am trying to model a packet data with 1 dimensional CNN but I have a very imbalanced classes in my target. I have 3 classes as class 0 has 53000 cases, class 1 has 300 cases and class 2 has 150 cases. Thanks in advance!
I have tried what you have suggested but I do not think I am getting a good results(loss and accuracy), from the model.


texts: Computer science corpus for training a language model

texts:
I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it.

Is there anything out-of-the box that I could use?
*I tried to look for the sciBERT corpus, can not find how to access it.

Thanks!


texts: Out of stock / Spike in demand prediction

texts:
The goal is to predict out-of-stock situations, either quantitatively (the gap) or qualitatively (out-of-stock likely to happen in next few weeks). Background:

We have existing demand planning process, through heuristics and software packages (using time series models). Most of the time they perform well, and recorded out-of-stock is rare (~ 8% of total orders rejected due to out-of-stock). Hidden cases might exist, meaning customer did not even place the order after knowing we ran out of stock.
Out-of-stock happens "in clusters". Out of a two year period, we have only 3-4 out-of-stock periods, each lasting one to two weeks. They did not happen when the demand was the highest, because we planned for that.

Challenges:

Time series can capture the average demand, however they cannot model extreme / unexpected cases, which cause out-of-stock. Using upper confidence interval can cover some extreme cases, however this significantly increases warehouse cost.
Not enough data for machine learning - essentially only 3 out-of-stock cases because they happen "in cluster".

Please suggest how I can handle this - any input welcomed...


texts: Best methods to solve class imbalance problem and why?

texts:
I have a data set where I need to detect fraud. 99% are not fraud and 1% are.

What methods can be used to solve problems where classes are imbalanced?


texts:
As of scikit-learn version 0.19.1, there is no implementation for calculating feature importance in an Isolation Forest. I'm also having trouble finding any online resources proposing ways to get at the problem. Does anyone know of any established methods for doing this or have any suggestions?

Here are some ideas I've been thinking about:

Calculate some kind of 'isolation metric' for each node in each tree (such as % of samples split) and get an average of this metric for each splitting feature.
After the model has been fit, go through each feature one at a time, randomly permute the data for that feature, and calculate the anomaly scores. Then calculate the average change in the anomaly scores.

All insights welcome. Thanks!


texts: Understanding Terminology in Goodfellow's paper on GANS

texts:
I am trying to understand Ian Goodfellow et al's paper Generative Adversarial Nets here.

In section 3 the author's write:

  The adversarial modeling framework is most straightforward to apply
  when the models are both multilayer perceptrons. To learn the
  generator’s distribution $p_g$ over data $x$, we define a prior on input
  noise variables $p_z(z)$, then represent a mapping to data space as $G(z;
$ $\theta_g)$, where $G$ is a differentiable function represented by a multilayer
  perceptron with parameters $\theta_g$.

I have a few questions.

1) From what I understand, the generator is a multi-layer perceptron input with a random vector sampled from a predefined latent space, (where wikipedia gives the example of a multivariate normal distribution). What does it mean to learn the generator's distribution $p_g$ over data $x$?

2) What does "we define a prior on input noise variables $p_z(z)$" mean? I understand that in Bayesian statistical inference, that a prior probability distribution, called a prior for short, gives a probability distribution on outcomes before some evidence is collected.

Any insights appreciated.


texts: Improving misclassification for one class in a multi-class classification task

texts:
Here I am trying to use 3 convolution layer neural network to classify a set of images (train data: (3249) , validation data: (487), test data: (326))
I have one class which is misclassified and I cannot understand what to do next. I have tried to reduce the value of dropout layer, but results got worst.
I know that the next solutions could be useful if I had bad classification for all classes :

Get more data

Try New model architecture, try something better.

Decrease number of features (you may need to do this manually)

Introduce regularization such as the L2 regularization

Make your network shallower (less layers)

Use less number of hidden units

What do you thing could be a good choice if I have only misclassifcation of one class?
Number of  total samples per class :
Black rot: 1180
Esca: 1383
healthy: 423
leaf blight: 1076

I had split the two datasets as follow:
x_train, _x, y_train, _y = train_test_split(x,y,test_size=0.2, stratify = y, random_state = 1)
x_valid,x_test, y_valid, y_test = train_test_split(_x,_y,test_size=0.4, stratify = _y, random_state = 1)


texts: Regression for prediction: is there any benefit of regularization?

texts:
What is the benefit of having flatter regression line in linear regression? Is there a proven benefit for prediction? Are there experiments that show regression with regularization performs better on the test data  than the same regression without regularization? What is theoretical justification for regularization? I've read that it reduces complexity. But did somebody prove in any way that reducing complexity by flattening the line is beneficial for prediction?
Is there a proof of benefit of regularization for non-linear regression?


texts: Calculate average Intersection over Union

texts:
I want to have a global IoU metric for each class in a segmentation model with a neural net. The idea is, once the net is trained, doing the forward pass over all training examples an calculate the IoU, I'm thinking in two approaches (for each class):
1) Calculate IoU for each training instance, and finally, calculate the mean IoU (per class)
2) Accumulate the intersections and unions along all the training instances, (per class) and finally taking the ratio.

To illustrate the problem, let's take two training instances in which for class=0, intersection_1 = 2, intersection_2 = 3, union_1=7, union_2=6. The mean IoU (approach 1) wil be 0.3929 and the second approach will be 5/13 = 0.3846. What method do you think will give better/unbiased result?


texts: what is store in leaf node and split node in Annoy or any approximate nearest neighbor model built using tree?

texts:
Im trying to understand the working of annoy and have read the code _make_tree since im not from C++ background im trying hard to figure out the logic of whats stored in leaf node and split node ,you below mentioned blog on annoy was very insightful but i could understand below to points
the question may sound silly but if you could share these details it would be really helpful ,
we feed only the vectors and its id (assume index number) to annoy
1.what technique is used to compressed vectors in annoy to represent in a smaller dimensions?
2.what is stored in leaf node ?
nodes with n_descendants == 1 the vector is a data point
https://github.com/erikbern/ann-presentation
all nodes are having 2 descendants ,can you guide and tell which nodes has n_descendants == 1 in this mnist tree diagram shown in your repo .
https://erikbern.com/assets/2015/09/tree-2-graphviz1-300x203.png
in above blog you showed a sample small tree ,Assume if its that final tree dose 60 in leaf node represent
the vector of 60 points( dose points represents the index of that particular vector or vector compressed into smaller dimension)
or
60 original dimension vectors of the respective 60 points in orange hyperplane ?
nodes with n_descendants > K the vector is the normal of the split plane.
what dose "the vector is the normal of the split plane" mean im unable find its meaning in other blogs/google
3.what is stored in split node/root node ?
dose it store the list of all the child node/child points below its hierarchy
or
dose it store a vector of for example say mean of all the vector points under it ?
in the sample tree image split node represented in dark blue color store the index of 255(157+98=255)points ?
4.During search if we feed a vector how this vector traverse to the leaf node in tree ?
do this vector do a similarity check with all the split nodes? if its assumed that  split node  represent mean of vectors under it ?


texts: Upper bound on size of sample set for decision trees

texts:
Say I have an instance space with 4 features and I know that a decision tree with 8 nodes can represent the target function I want to learn. I want to give an upper bound on the size of the sample set needed in order to achieve a true error of at most x%.

I found this theorem in a text book.

  Let $\mathcal{H}$ be an hypothesis class and let $\epsilon, \delta > 0$. If a
  training set $S$ of size

  $$n\geq\frac{1}{\epsilon}\operatorname{ln}\left(\frac{|\mathcal{H}|}{\delta}\right)$$

  is drawn from distribution $\mathcal{D}$, then with probability greater than or equal to $1-\delta$ every $h \in \mathcal{H}$ with true error $err_D(h) \geq \epsilon$ has training error $err_S(h) > 0$. Equivalently, with probability greater than or equal to $1-\delta$, every $h \in \mathcal{H}$ with training error zero has true error less than $\epsilon$.

My question is, can I use this theorem to give such an upper bound? If yes, what would the size of my hypothesis space $|\mathcal{H}|$ be in this case? If not, how can I give this upper bound?


texts:
During the split function of the decision tree, I am getting an assertion error stating:

Error > Assert: type(C) == dict

I couldn't find any errors in the code up until now. Could you please correct it if there is any problem with it?

My code:

def split(X,Y,i):

    def __find_best_split(self):
                best_split = {}
                for column in self.X:
                information_gain,split =
    self.__find_best_split_for_column(column)
                    if split: continue
                    if not best_split or best_split["information_gain"]
       <information_gain:
                       best_split =
    {"split":split,"col":column,"information_gain":information_gain}

                    C= (best_split)

                return C


texts: Is balancing class data for imbalanced problems helpful or just folklore when considering thresholds?

texts:
Caveat:  I'm aware that imbalanced data questions are a dead horse, but I haven't found an answer to this  flavor of it directly.
When working with highly imbalanced data (e.g. binary class cases), the common wisdom is to try training on an upsampling of the minority class, downsampling of the majority class, some combination of the two, or something like SMOTE. The potential gains of doing so are obvious if one accepts default model thresholds and tunes for matrix-based metrics like accuracy or f1. However...
It is my experience that if one instead optimizes for a threshold-invariant metric first (like ROC AUC or PR AUC or similar) and then adjusts the threshold to meet the specific business need, resampling loses its benefit swiftly and often hurts more than helps.
Is the common wisdom of balancing class data (or its various flavors) then folklore given the above strategy? If not, in what cases should one generally expect it to help, and in what cases hurt?


texts: Calculate the number of TP, FP, FN for the object detection task with many classes

texts:
I am trying to count the number of True Positives (TP), False Positives (FP), and False Negatives (FN) for object detection task with two classes. I assumed that:

TP - is the detection with intersection over union (IoU) > threshold, same class and only the first detection of a given object

FP - is the number of all Predictions minus number of TP (FP = PRED - TP)

FN - is the number of not ground truths and should be equal to ALL_GTS - TP

Based on above I wrote the following method:
def detail_results(gts, model_pred,th):
    TP = 0
    FP = 0
    FN = 0
    GTS = 0
    PRED = 0

    for image_idx in range(len(gts)):
        gt = gts[image_idx]
        pred = model_pred[image_idx]
        GTS+=len(gt)
        PRED+= len(pred)
        ious = []
        iou_idxs = []
        for d_gt in gt:
            founded = False
            for d_pred in pred:
                iou = bb_intersection_over_union( d_pred[0:4],d_gt[0:4])
                if iou != 0.0:
                    # iou bigger than threshold, same class, not double detection
                    if iou >= th and d_gt[4] == d_pred[4] and not founded:
                        TP+=1
                        founded = True
            # GT than was not founded
            if not founded:
                FN+=1
    FP = PRED - TP
    return GTS, PRED, TP, FP, FN

Where:

gts contains Ground Truth Bounding Boxes in the following format: [x1, y1, w, h, class]
model_pred contains my model predictions in the same format
th is IoU Threshold
Method bb_intersection_over_union calculates IoU of GT and
prediction bounding boxes.

But results returned by this method are different than calculated by the embedded method from the Darknet framework:

Where I made a mistake?
Do you know some libraries that can calculate TP, FP, FN from bounding boxes coords?


texts: What is the best Sequence Mining Algorithm to use for Hand Written digits Recognition system

texts: What would be the best Sequence Mining Algorithm to use for Hand Written digit Recognition system (other then Deep learning ) ??


texts:
My friend is in the business of getting cats to the top of mountains. He currently uses a set of heuristics to decide which cats are most likely to get to the top.

For the ones he likes, he feeds, at great expense, in the hope that they will be strong enough to get to the top. For the ones he doesn't like, he thinks it not worth his time / money to feed, and he's happy and surprised if they get to the top anyway. From looking at the data, we know that on average, the ones he doesn't feed get to the top less often.

I am training a classifier to help him decide which to feed. Does it make sense to include rows in the training data where he fed them and where he didn't? When I excluded the rows where he did not feed them, the model did worse, which I did not expect.

Forgive the example. It's contrived, but I think retains all of the characteristics of the actual situation.


texts: I'm working on a sound classification problem. For that, I'm converting the audio signals into spectrogram images and using transfer learning to classify them. Currently I'm using pretrained models like AlexNet, Mobilenet, Xception etc which were trained on the ImageNet dataset. These models are primarily trained for different object recognition and producing decent result in classifying spectrograms. Although I think that a model trained on a large spectrogram dataset will produce better results. Is there any such model available?


texts: Dealing with normalized regression output

texts: I have a regression model that is trained on a bunch of features and normalized targets so naturally when I use the model to predict on a new input, the output is also normalized (well not normalized per se but not what I expect to see). How should I deal with this? I tried using the inverse_transform function in sklearn that is in most scalers but it's not giving me correct results. This is probably because it wasn't already fitted on this data and therefore doesn't know how to inverse it. What can I do?


texts:
http://cocodataset.org/#detection-eval
"ARmax=1: AR given 1 detection per image"
"AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs."

Above link, they described ARmax=1, but I didn't understand exactly. I understood general recall concept in machine learning or other areas. Could you explain ARmax=1 more detail ??


texts: Dose finding slope/intercept using the formula of m,b gives best fit line always In linear regression?

texts:
In liner regression We have to fit different lines and chose one with minimum error so What is the motive of having a formula for m,b that can give slope and intercept value in the regression line ,when it cannot give best fit line directly ?

1.Consider i applied the value in dataset on the  formula of m,b and found the regression line  yhat = 17.5835x+6 and for example just assume   error calculated for this line was  3
2.Consider i fit another line randomly (i am not using the formula of m,b to find value of m,b  assume m,b value for this random line was 16,3) my 2nd regression line is yhat = 16x+3and  for example just assume   error calculated for this line was 1.5
Linear Regression Goal : to choose best fit line that has minimum error
so my second line is better than the 1st line in this case
What is the point of having a formula which gives value for slope "m", intercept "b" when it cannot give best fit line directly ?
OR is my understanding incoorect Dose finding slope/intercept using the formula of m,b  gives best line always ?
if its YES   then there is no need to try mulitple lines and calculate error  and choose line with min error
if its No  then whats the point of having a formula for slope m,intercept b when it cannot give the best fit line . dose that mean maths/stats community need to change this forumla for slope,intercept


texts: How to find unique occurance of a value based an another value using pandas?

texts:
Below is my dataset.

Student_code   Subject
___________    _______
123            Maths
123            Chemistry
124            Maths
125            Chemistry
126            Physics
126            Maths
127            Physics
127            Chemistry

I would like to find number of students who have just enrolled for one course. In this example, given "Maths", i should get 1 as answer as student id 124 enrolled just for Maths.


texts: Model selection by Interpreting p-value of anova function

texts:
I am trying to interpret the p-values for model selection. Here is a sample code taken from a book (An Intro. to stat. Learning, page 290, by Gareth James et al.)

￼
Null-hypothsis: Model M1 is sufficient to explain the data against the alternative hypothesis that a more complex model M2is required.

With this statement, I think p-value should be high so that null-hypotheis can be rejected and a model M2 is selected. Keeping this in mind, I think we should select Model 5 as it is having highest p-value, but authors mention either of Model 3 or Model 4 is a good fit. I don't understand why Model 5 is not a good fit.


texts: Where can I find a dataset for long sequence text chunking?

texts:
Context:

I have documents with reviews of articles that have the following structure:

Introduction: a description of the review, dates and metadata that will be discarded. (avg~180 words, std~30 words)
Loop with:

Reference: the reference article to review. A single review can talk about multiple articles. (avg~8 words, std~2 words)

Subject: topic of the following comments. There can be more than one subject for the following comments. (avg~7 words, std~3 words)

Comment: atomic comment about the article. There can be multiple comments per subjects. (avg~45 words, std~30 words)

Final: a brief summary of the review and other info that will be discarded. (avg~165 words, std~40 words)

I've to identify orderly the different chunks of references, subjects and comments. I've already designed a pipeline, and have a dataset to test and train my models, I'm using BIO tags for chunking as output, but I can adapt the pipeline to use BMEWO (BILOU).

I'm searching for a similar problem dataset, to compare how my pipeline is performing against the best metrics accomplished in that problem.

I've already searched in:

https://archive.ics.uci.edu/ml/datasets.html

https://www.kaggle.com/datasets

Until now the closest dataset I've found is the CoNLL for named entity recognition or grammar tagging(noun phrases, prepositional phrases, verb phrases).

But that problems seems very different from mine:

Small length per chunk in relation with my problem.
The length of every chunk class are similar, In my case the size of the comments are very different than the subjects and references.
The outside tag could be everywhere while in my problem only can be at the beginning or the end.
These problems seems more focus on short grammatic structure instead of long sequence meaning.

I'm searching with the following keywords: chunk, long-sequence, text. I'm not sure if the word chunk could be misleading.

Where can I keep searching datasets? What keywords can I use to improve my search? There is a similar problem in which I could try to transform mine to test against?


texts: SKLearn DT regressor - good enough score?

texts: What constitutes as a "good enough" score for a Decision Tree Regressor? The .score() function gives us a general score about our model. This can be 1 if the model predicts all data with a 100% accuracy, and can be arbitrarily worse. If I understand correctly, a score of 0 means the prediction is quasi constant. But starting with what value can we say that the prediction is "usable" (i know this is ambiguous, but still). Is a score of 0.4-0.5 good enough to use the predictions in practice as opposed to not using any machine learning techniques?


texts: in tf_upgrade_v2 module, 'cp949' codec error comes out

texts:
I'm migrating my tensorflow v1 code to v2 code as follows

tf_upgrade_v2 --infile Untitled1.ipynb --outfile tensorfoo-upgraded.py

but

'cp949' codec can't decode byte 0x9d in position 295: illegal multibyte sequence

comes out. I noticed this is error of windows, because my ipynb is encoded as utf-8. not ansi.
I recreated my ipynb via jupyter notebook, but seems it still encoded as utf-8.
Is there any solution to utf-8 encoding error in tf_upgrade_v2?

Extracting code lines from original notebook
2020-04-06 17:20:06.360581: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Traceback (most recent call last):
  File "c:\users\owner\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\owner\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Scripts\tf_upgrade_v2.exe\__main__.py", line 7, in <module>
  File "c:\users\owner\appdata\local\programs\python\python36\lib\site-packages\tensorflow_core\tools\compatibility\tf_upgrade_v2_main.py", line 147, in main
    args.input_file, output_file, upgrade)
  File "c:\users\owner\appdata\local\programs\python\python36\lib\site-packages\tensorflow_core\tools\compatibility\tf_upgrade_v2_main.py", line 47, in process_file
    ipynb.process_file(in_filename, out_filename, upgrader)
  File "c:\users\owner\appdata\local\programs\python\python36\lib\site-packages\tensorflow_core\tools\compatibility\ipynb.py", line 43, in process_file
    raw_code, notebook = _get_code(in_filename)
  File "c:\users\owner\appdata\local\programs\python\python36\lib\site-packages\tensorflow_core\tools\compatibility\ipynb.py", line 113, in _get_code
    notebook = json.load(in_file)
  File "c:\users\owner\appdata\local\programs\python\python36\lib\json\__init__.py", line 296, in load
    return loads(fp.read(),
UnicodeDecodeError: 'cp949' codec can't decode byte 0x9d in position 295: illegal multibyte sequence


texts:
I have a time series data with the date and temperature records of a city. Following are my observations from the time series analysis:

By plotting the graph of date vs temperature seasonality is observed.
Performing adfuller test we find that the data is already stationary, so d=0.
Perform Partial Autocorrelation and Autocorrelation with First Seasonal Difference and found p=2 and q=10 respectively.

Code to Train Model
model=sm.tsa.statespace.SARIMAX(df['temperature'],order=(1, 1, 1),seasonal_order=(2,0,10,12))
results=model.fit()

This fit function runs indefinitely and does not reach an output. I am running on a on Google Colab CPU.
How to fix this issue?


texts: Stacking with missing predictions from models

texts:
There are several models A, B, ..., Z. These models may not give a prediction for any of entities from id1, id2, id3, .., idn.

What approaches will be more suitable for stacking in these situations?

For example,

model A can not predict for id1, but can predict for id2, ..., idn;
model B can not predict for id2, but can predict for id1, id3, ...,
idn;
model C can not predict for id2 and id3, but can predict for id1, ..., id4, ..., idn.


texts: In Google AutoML Tables, how do I export model then deploy it to docker container in Cloud Run

texts:
AutoML Table endpoint is really expensive and I was wondering if someone could point me in the right direction in how to export then deploy to a docker container in Cloud Run similar to this tutorial but it's for AutoML Vision:
https://medium.com/@juri.sarbach/how-to-deploy-your-automl-model-in-a-cost-effective-way-5efdd377d4d2


texts: How to find whether a dataset is blanced or imbalanced?

texts: I have few dataset to experiment classification(Multi-class). These datasets are about 400GB. I wanted to know whether the dataset is balanced or imbalanced. How to know that dataset is balance or imbalanced using any scientific way?


texts: Neural Network Initialization - Every layer?

texts: Does every layer of a Neural Network require weight initialization or just the first? Does the first layer feed into the next layer and initialize itself? My intuition is that every layer needs its own initialization but I'm finding it hard to see that said explicitly. Thanks!


texts: How to validate a clustering model without a ground truth?

texts: Im dealing with a dataset (text messages about source code comments) that are not labeled. I don't have a assumption about the implicits classes in this dataset. I want to discovery (by clustering) the common hidden patterns shared by the groups of messages. This is a unsupervised learning problem. I was asked how i will  validate this method (patterns discovery, clusters) without a dataset of correct answers to measure the output of the model with the "reality". Im neither a specialist in the field of the messages dataset to manualy inspect and label the data. So, how to approach this question or provide a scientific explanation about the model output? How to prove that the clusters generated by the model are reasonable or correct?


texts: How does exactly class_weight in Keras work?

texts:
I'm working on a multi-label problem in Keras, using binary-cross-entropy loss function with sigmoid activation. Let's say I have 4 classes, so a response might look like this:

[1, 0, 0, 1]

Though a vast majority of the responses will actually be zero's all around

[0, 0, 0, 0]

The data is also very imbalanced between the different classes, and if i did a sum on y, the total number of positives in each class would look something like this

[2000, 500, 1000, 250]

So to give equal weight to each class I would think that I should construct a dictionary-like so

weights = {0: 2.5, 1: 10., 2: 5., 3: 20}

And feed that to the class_weight parameter in Keras.

Here is my question though:

If my response variable was binary (only first class for example), I would need to feed a dictionary that defines both a factor for the 0 an 1

weights = {0: 1, 1: 2.5}

Which I take to mean that positive samples are weighted higher than negatives. But, how do weighted samples work with the multi-label (or multi-class for that matter)? since in that case, I've only given weights to the classes in general, not specifically positives and negatives

This should be the loss function I'm working with

−(ylog(p)+(1−y)log(1−p))

And with binary-cross-entropy and sigmoid, this is applied to each class and the total loss is then the sum of all class losses for all samples.

But where exactly is the class weight applied here? Is it multiplied on the entire function? If that's the case, then it's not really fixing the class imbalance as I see it, since it also weighs the negatives higher, not just the positives? I would imagine that is desirable is to only multiply positive responses?

If anyone can clear up exactly how this all works so I can understand a bit more what's going on under the hood I would be grateful.


texts: Question about Similarity vs Dissimilarity Matrix

texts:
Right now, I'm working on a coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is, if I want to use one of the many clustering algorithms given in $R$, such as the K-Medoids algorithm, does it require a similarity or dissimilarity matrix as its parameter?

What's the difference between the two?

If I use the Gower Distance from the Daisy function in R, does it output a similarity or dissimilarity matrix?

Also lets assume that I have $n$ features and they are all categorical (this is just an example) I a custom distance function where when comparing two data points $G$ and $H$, I use the formula $$\sum_i^nX_i$$
where $X_i = 1 $ if feature $i$ of $G$, $G_i$ and feature i of $H$, $H_i$ are equal to each other. So, $$X_i=1$$if and only if $G_i=H_i$ for feature $i$ for all of the $n$ categorical features. Will this result in getting a similarity or dissimilarity matrix?

Also, as mentioned above, if I want to use one of the many clustering algorithms given in $R$, such as the K-Medoids algorithm, does it require a similarity or dissimilarity matrix as its parameter?

In general does the similarity or dissimilarity matrix get used for these


texts: Interpretation of level, trend and seasonal indices in holt winters exponential smoothing

texts:
I am trying to learn Holt Winters exponential smoothing. In the algorithm there are three indices involved (level, trend, seasonality) while forecasting.

My questions:

What is the interpretation of these 3 indices?
How do I differentiate between different values of these indices? For example, what is the difference between level 100 and level 200 or what is the difference between trend 2 and trend 5?
How do we initiate these values?

Any help will be greatly appreciated.


texts: A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation

texts:
very good implementation of A2C continuous for Pendulum-v0

Code has snippet to stop execution when mean of last 10 or 20 is higher than -20 but the results look like:

episode: 706   score: [-13.13392661]
episode: 707   score: [-12.91221984]
episode: 708   score: [-50.38036647]
episode: 709   score: [-74.58410041]
episode: 710   score: [-138.1596521]
episode: 711   score: [-87.3867222]
episode: 712   score: [-63.28444052]
episode: 713   score: [-0.37368592]
episode: 714   score: [-13.28473712]
episode: 715   score: [-117.78089523]
episode: 716   score: [-25.65207563]
episode: 717   score: [-0.36829411]
episode: 718   score: [-50.81750735]
episode: 719   score: [-0.33565775]
episode: 720   score: [-0.47168285]
episode: 721   score: [-0.35240929]
episode: 722   score: [-0.40577252]
episode: 723   score: [-0.37114168]
episode: 724   score: [-25.73963544]
episode: 725   score: [-37.70957794]

Even with the reward/10 line, still pretty good. However, I don't understand these lines regarding negation of loss and why the entropy equation looks different from what I saw in Packt Publishing Deep Reinforcement Learning Hands-On per picture below:

The code:

 def actor_optimizer(self):
        #placeholders for actions and advantages parameters coming in
        action = K.placeholder(shape=(None, 1))
        advantages = K.placeholder(shape=(None, 1))

        # mu = K.placeholder(shape=(None, self.action_size))
        # sigma_sq = K.placeholder(shape=(None, self.action_size))

        mu, sigma_sq = self.actor.output

        #defined a custom loss using PDF formula, K.exp is element-wise exponential
        pdf = 1. / K.sqrt(2. * np.pi * sigma_sq) * K.exp(-K.square(action - mu) / (2. * sigma_sq))
        #log pdf why?
        log_pdf = K.log(pdf + K.epsilon())
        #entropy looks different from log(sqrt(2 * pi * e * sigma_sq))
        #Sum of the values in a tensor, alongside the specified axis.
        entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.))

        exp_v = log_pdf * advantages
        #entropy is made small before added to exp_v
        exp_v = K.sum(exp_v + 0.01 * entropy)
        #loss is a negation
        actor_loss = -exp_v

        #use custom loss to perform updates with Adam, ie. get gradients
        optimizer = Adam(lr=self.actor_lr)
        updates = optimizer.get_updates(self.actor.trainable_weights, [], actor_loss)
        #adjust params with custom train function
        train = K.function([self.actor.input, action, advantages], [], updates=updates)
        #return custom train function
        return train

Again, the entropy equation coded was this:  entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.)) which looks different from what's given in the textbook photo above.

Also, why is the loss a negation? actor_loss = -exp_v?

Is it negated because it is gradient ascent rather than gradient descent of the objective function for a policy gradient?


texts: Reinforcement learning for continuous (rather than discrete) actions

texts:
I'm familiar with traditional reinforcement learning where the algorithm must choose a categorical action (e.g., best move in a game or the highest click-through-rate ad among a set of ads). Does reinforcement learning work for problems with continuous actions? Are there any easy to understand references that you recommend? I was hoping to be able to use reinforcement learning to assign credit limits, which are represented in dollars (i.e., continuous amounts).

Edit: I've come across this paper from Google's DeepMind, which so far seems relevant.


texts:
0
I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson triggering a negotiation).
I have the following data aggregated per opportunity/period of time:
One record for each opportunity (positive case) generated in the previous 12M period (rolling), with the behavioral activity in the preceding 6 months happened before the opportunity generation date (if any)
One record for each week/account combination (negative case) in the previous 12M period (rolling), with the behavioral activity in the preceding 6 months happened before Sunday of that week. IF an opportunity was generated on that Sunday, then no record is generated for the account.
So typically a record would consist of a specific opportunity id and the behavioral data for that account in the previous 6 months OR the behavioral activity in the 6 months period for an account in each week of the previous 12 months. something like:
Accout Opp week day N_interactions_Content_X_previous6M
123   1   52    Monday 34
123   0   52    Sunday 36
234   0   52    Sunday 56
My question is with regard to the records where an opportunity exists BUT no Interactions are present in the previous 6 months. So positive cases where all the behavioral variable are set to 0 (all the variables) but a positive case generated (maybe because it was a stand-alone effort of the salesperson - which is not tracked in the systems)
The amount of these positive cases is almost 40% the number of positive cases WITH interactions.
I'm questioning if providing a set of records with all variables set to 0 but representing a positive case would contaminate the learning process of the model (the model would learn that when all variable are set to 0 there is a random chance to have positive cases generated)
Should I exclude these data? There is no Sales data in our systems so can't be added to the model. Does anyone know literature about how to manage these "positive empty" records?
machine-learning


texts: What annotators are used in Cohen Kappa for classification problems?

texts:
I am working on a classification problem using algorithms such as Logistic Regression, Support Vector Machines, Decision Trees, Random Forests and Naive Bayes. My data consists of binary class classification i.e. Fire(1) or No Fire(0). Due to the imbalance in data Cohen Kappa was recommended for evaluation of model performance.

I am using the scikit-learn sklearn.metrics.cohen_kappa_score to compute the cohen kappa score. To compute the value it takes the following inputs

from sklearn.metrics import cohen_kappa_score
cohen_score = cohen_kappa_score(y_test, predictions)
print(cohen_score)

So it takes the y_test and predictions made by the specific model the same inputs used for confusion matrix and classification report.

However, Cohen Kappa is suppose used to measure the inter-rater agreement between observers or annotators. If we were measuring the quality of a document based on scores by 2 reviewers. Those 2 reviewers would be the annotator. But in the case of a classification problem. When I compute the score in scikit learn as mentioned above. What 2 annotators or observers does Cohen Kappa use in classification problems when you compute the score in scikit learn?


texts: Keras Model always predicts the same class

texts:
I am currently trying to build a CNN classifier which takes a ector representing the log of an ECG spectrogram together with its class. What I am currently experiencing is the fact the model always predicts the class '1'.

Here is my code:

def LoadData(filename):

    with open(filename, 'rb') as f:
        x, y = pickle.load(f)

    xTrain, xTest, yTrain, yTest = train_test_split(x, y, test_size=.2)

    return xTrain, xTest, yTrain, yTest

def AugGenerator(xTrain, xTest, yTrain, yTest):

    imagegen = ImageDataGenerator()
    trainGen = imagegen.flow(xTrain, yTrain, batch_size=20)
    testGen = imagegen.flow(xTest, yTest, batch_size=20)

    return trainGen, testGen

def CNN(blockSize, blockCount, inputShape, trainGen, testGen, epochs):

    model = Sequential()

    # Conv Layer
    channels = 32
    for i in range(blockCount):
        for j in range(blockSize):
            if (i, j) == (0, 0):
                conv = Conv2D(channels, kernel_size=(5, 5),
                              input_shape=inputShape, padding='same')
            else:
                conv = Conv2D(channels, kernel_size=(5, 5), padding='same')
            model.add(conv)
            model.add(BatchNormalization())
            model.add(Activation('relu'))
            model.add(Dropout(0.15))
            if j == blockSize - 2:
                channels += 32
        model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
        model.add(Dropout(0.15))

    # Feature aggregation across time
    model.add(Lambda(lambda x: K.mean(x, axis=1)))

    model.add(Flatten())

    # Linear classifier
    model.add(Dense(4, activation='softmax'))

    model.compile(loss=keras.losses.categorical_crossentropy,
                  optimizer=keras.optimizers.Adam(),
                  metrics=['accuracy']) # F1?

    model.fit_generator(trainGen,
                        validation_data=testGen, steps_per_epoch = len(trainGen) / 20,
                        validation_steps = len(testGen) / 20,
                        epochs=epochs, verbose=1)
    return model

if __name__ == '__main__':
    xTrain, xTest, yTrain, yTest = LoadData('./LogSignalsAndLabels.pk1')
    trainGen, testGen = AugGenerator(xTrain, xTest, yTrain, yTest)
    # model = CNN(4, 6, (285, 33, 1), trainGen, testGen, 1)
    # model.save('./cnn_model.h5')
    model = keras.models.load_model('./cnn_model.h5')
    model.summary()

    print('Evaluation...')
    y_predict = model.predict_generator(testGen, steps = len(testGen)).argmax(axis=1)
    print(y_predict)
    yTest = yTest.argmax(axis=1)

    f = open('./evaluation.txt', 'w')
    f.write('model: CNN, epochs: {} \n confusion_matrix: \n {}'.format(1, confusion_matrix(yTest, y_predict)))
    f.close()

The generators trainGen and testGen both contain a matrix of vectors with length 285, representing the spectrograms, together with their labels one hot encoded, e.g: [1., 0., 0., 0.]. After print(y_predict) shows only ones, giving a confusion matrix of this form:

model: CNN, epochs: 1
 confusion_matrix:
 [[   0 1034    0    0]
 [   0  125    0    0]
 [   0  508    0    0]
 [   0   39    0    0]]

For completeness, when I converted the categories into a numeric form, I worked with this way:

y[y=='N'] = 0
y[y=='A'] = 1
y[y=='O'] = 2
y[y=='~'] = 3

y = keras.utils.to_categorical(y)

Where y is the column of the original labels.

What am I missing?

Edit: the using of only one epoch is for question purposes.


texts: Need input on which features to drop in classification model

texts:
This is the correlation of features with my target variables. I have done all the features engineering but I am left with these features.

Any input on what columns to keep for model training and what to drop. Is there any criteria for dropping features that I don't need. It seems credit history is the only feature that has high correlation.

Loan_ID              0.011610
Gender               0.017987
Married              0.091478
Education           -0.085884
Self_Employed       -0.003700
ApplicantIncome     -0.004710
CoapplicantIncome   -0.059187
LoanAmount          -0.037318
Loan_Amount_Term    -0.022549
Credit_History       0.561678
Total_Income        -0.031271
Total_Income_Log     0.007240
LoanAmt_Log         -0.037536
CH__0               -0.540556
CH__1                0.432616
EMI                 -0.011552
EMI_Log             -0.028496
Dependents_1        -0.038740
Dependents_2         0.062384
Dependents_3        -0.026123
Property_Area_1      0.136540
Property_Area_2     -0.043621
Loan_Status          1.000000
Name: Loan_Status, dtype: float64


texts: Ensemble Techniques - Bagging | Subset size

texts:
I do have a question on ensemble techniques Baggging/Boosting.
- What would be the subset size for Bagging?


texts:
I'm looking for a good explanation of how convolutions in deep learning work when applied to multi-channel images. For example,  let's say I  have a 100 x 100 pixel image with three channels, RGB. The input tensor would then have dimensions 100 x 100 x 3.

If I apply a convolution with N filters and a stride of one, will the output dimension be:

100 x 100 x 3 x N ?

or

100 x 100 x N ?

In other words, does the convolution that is applied "flatten" the image, or is the convolution applied on a channel by channel basis?


texts: Replace method in pandas not giving expected result

texts:
I have data as below:

123.12.23.2
110.22.21.23

I want to mask this data as below one

1xx.xx.xx.x

So I tried below code :

 readFile = pd.read_csv("C:/Users/siddhesh.kalgaonkar/Desktop/data01.txt",header=None)
 readFile.columns = ['IP']
 readFile['IP']=readFile['IP'].str.replace("(?<! )","X").astype('str')
 readFile

but I gives me data as below one which is not correct:

                                                  IP
0  XXXXXXXXXXXXXXXXXXX 1XXXXXXXXXXXXXXXXXXXXXXXXX...
1  XXXXXXXXXXXXXXXXXXX 1XXXXXXXXXXXXXXXXXXXXXXXXX...
2  XXXXXXXXXXXXXXXXXXX 1XXXXXXXXXXXXXXXXXXXXXXXXX...

I am new to pandas. So where am I going wrong ?

Also, I want to do it without pandas because the platform on which I would be deploying this code may be won't have pandas. So need to be ready for the other scenario as well. Below is my code:

readFiles=open("C:/Users/siddhesh.kalgaonkar/Desktop/data01.txt","r")
 finalValues = re.sub("(?<! ).","X",readFiles)

It gives below error:

>>> finalValues = re.sub("(?<! ).","X",readFiles)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\siddhesh.kalgaonkar\AppData\Local\Programs\Python\Python36\lib\re.py", line 191, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

I want to split this data on the basis of delimiter (in case I have multiple columns) and then I have to apply regex logic. Please help me out here.


texts:
I'm trying to use LSTM on time-series data in order to generate future sequences that looks like the original sequences in term of values and progression direction.
My approach is:

train RNN to predict a value based on 25 past values then use the model to recursively generate future predictions by appending the predicted values to the original sequence and shift the old values ...

Playing with LSTM cells, I found that the model is not able to learn to generate sequences that looks like the original data. It only predict next value then start converging to an 'equilibrium' or static value which is the same whatever the input sequence is.

I'm wondering if Stateful LSTM would help to learn better from past values and try to predict something close to what it have already? My goal here is to generate sequences that looks like something that the model have seen already.

Please let me know if I'm missing something or if you had a similar situation and you found the best approach to generate timeseries sequences that looks like what the model learned in the past.


texts: What happens if we use Z-Score(Mean Normalisation) in Neural Networks to standardize values

texts:
I'm currently doing the DeepLearning.ai Specialization where they divide pixel intensity with the maximum value it has (255) to standardize the data when working on a classification NN for cats.
What effect does Z_Score have on this?
Will my code fail becuase pixels cannot have negative values which will be assigned by Z-Score?


texts: I have a Keras h5 file that I want to load into the same model but this one is created using PyTorch. Is ONXX a viable intermediary option? What else can I use?


texts:
Hello.!
I have been working on a gujrati handwritten dataset to identify character using kNN.
i'm not able to find the zone of image like above in python.
please help me..!!


texts: Anomaly detection on multidimensional time series

texts:
I have relatively little knowledge of unsupervised machine learning.

I'm working on a project that aims to find anomalies in a set of n data, measured every x ms, and we have no label on all these data.
So I'm here to ask you which algorithms/approach should be the best?

Edit : I'm working in an industrial context, and my data are sensor readings (temperature, pressure, power consumption, etc.) all from the same equipment.


texts: How to avoid MemoryError when using fetch_openml()?

texts:
I am using the following code to get mnist

from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1, cache=True)

It has worked before but now i get this error:

Does anyone know a solution to this? Much appreciated, thanks


texts: How do data scientists reason under ambiguity?

texts:
I am looking for real-wrold examples of how data scientists build models and make decisions under situations of extreme ambiguity as described by Cassie Kozyrkov here:

  A decision under ambiguity involves unknown subjective probabilities
  that the decision-maker feels powerless to contend with, e.g. any
  situation where I have no freaking idea and can’t begin to think about
  what the probabilities might be.

  In this framing, the difference between subjective risk and ambiguity
  has to do with the decision-maker’s skills, perceptions, and feelings
  about whether they’re qualified to make the assumptions required to
  assign subjective probabilities to parameters of the situation.

Various formalisms have been developed to reason under ambiguity such as Dempster-Shafer theory but I cannot find practical, real-world examples of their usage. I could see such techniques being useful for building COVID-19 models, for example.


texts: What are the differences between multivariate data visualizations and multidimensional visualizations?

texts:
I believe this should be the correct place for this topic since it didn't seem to fit naturally in the math stackexchange.

I am writing an essay about data visualization, but I am having trouble nailing down some terminology. I used data from the OECD to investigate trends in the data about OECD member countries. I created a graph in Tableau using data from the OECD that depicts the GDP per capita, average # of years spent in education system, satisfaction score as reported by the member country citizens, and a "Feel Safe" score as reported by the member country citizens (a percentage of the surveyed population who said they would feel safe walking home alone at night). Click the following link to download the full OECD data set I used.

I am confused about whether to label my graph as a multivariate visualization or a multidimensional visualization.

The core of my confusion comes from someone claiming that my graph is only two-dimensional since it can be displayed on a page. However, "dimensions" usually refers to the number of independent variables in a data set and the data displayed in my graph contains four independent variables.

Could someone please tell me what my graphic should be categorized as, and possibly explain some of the differences between multivariate and multidimensional data/visualizations (if any)?


texts: In softmax classifier, why use exp function to do normalization?

texts:
Why use softmax as opposed to standard normalization? In the comment area of the top answer of this question, @Kilian Batzner raised 2 questions which also confuse me a lot. It seems no one gives an explanation except numerical benefits.

I get the reasons for using Cross-Entropy Loss, but how does that relate to the softmax? You said "the softmax function can be seen as trying to minimize the cross-entropy between the predictions and the truth". Suppose, I would use standard / linear normalization, but still use the Cross-Entropy Loss. Then I would also try to minimize the Cross-Entropy. So how is the softmax linked to the Cross-Entropy except for the numerical benefits?
As for the probabilistic view: what is the motivation for looking at log probabilities? The reasoning seems to be a bit like "We use e^x in the softmax, because we interpret x as log-probabilties". With the same reasoning we could say, we use e^e^e^x in the softmax, because we interpret x as log-log-log-probabilities (Exaggerating here, of course). I get the numerical benefits of softmax, but what is the theoretical motivation for using it?


texts:
I'm working with several estimators of all kind. Then, I want to stack these estimators, and the best is if they have low correlation between them.

I suppose that the correlation method depends on the type of dependent variable, if it's categorical or numerical. In my case, it's categorical, and the estimators are classifiers.

How can I do the correlation between two estimators?


texts: How to filter Items in Recommender Systems?

texts:
I have a Recommender System which recommends Articles based on Similarity from 3 Features, "Page-Title, Article Content, Tags". But some of the Articles are NSFW(Related to Adult Topics). I want to keep these Recommendations separate from the normal ones? Any idea about how do I go ahead with it,
I was thinking about keeping the adult articles separate and using another metric to recommend them, but I do not feel its right as it would create a whole different Recommender System.


texts: How to get non-normalized feature importances with random forest in scikit-learn

texts:
I'm using the feature_importances_ attribute in the random forest classifier of scikit-learn to plot the importances of each feature. However, I'd like to plot these importances non-normalized. I have searched around how to do this, but there doesn't seem to be an easy method how to do this. I tried manually:
temp = [t.tree_.compute_feature_importances(normalize=True) for t in clf.estimators_]
arr = np.array(temp)
arr2=[]
for i in range(19):
    arr2.append(sum(arr[:,i]))
arr3 = np.array(arr2)
indices = np.argsort(arr3)[::-1]
indices.reshape(1,-1)

plt.figure()
plt.title("Feature importances")
plt.bar(range(X_train.shape[1]), arr3[indices],
        color="r", align="center")
plt.xticks(range(X_train.shape[1]), indices)
plt.xlim([-1, X_train.shape[1]])
plt.show()

Which gives these results for normalize=True and normalize=False respectively. (the normalize parameter seems to be inverted somehow...?)

For reference, this is the result from using feature_importances_ (the error-bars are not relevant to the question):


texts: Data binning - Why we need to transform Categorical Variables?

texts:
Having a lot of categorical features and other numerics why we need to transform the categorical to binary values? Is it for using the values in mathematics functions of the algorithms?

Thanks!


texts: How to find the dependent variables from a dataset?

texts:
I am stuck at where how can I get the most dependent variables based on the mean

I have this dataset and when I try to:

df.groupby('left').mean()

It gives the output:

And one of my friends said, from that graph the dependent variables for the attribute left will be

1.Satisfaction Level

2.Average Monthly Hours

3.Promotion Last 5 Years

I am wondering How could someone guess that?


texts: Are there specific feature engineering techniques for spectral data?

texts:
I am working on a model that takes 10 wavelengths (UV and NIR data) as input which measure the absorbances of a vegetable, to predict its nutritional density (protein, antioxidants, polyphenols) (quantitative variable.

I've already learnt naive models prediction (linear regression and SVR). Now I want to go further and elaborate a more efficient model.
I would like to know if there are specific feature engineering techniques (like transformation) to apply on spectral data ?
I applied a scatter correction method on my input variables (Multiplicative Scatter Correction) described in this article: https://nirpyresearch.com/two-scatter-correction-techniques-nir-spectroscopy-python/
What should be my next leads ?


texts:
I know that I can use ModelCheckpoint in Keras for checkpointing a model every epoch (or every few epochs, depending on what I want).

I am getting my data for each minibatch from a fit_generator, and it takes a very long time to evaluate each minibatch. I'd like to be able to checkpoint by minibatch instead of by epoch. How can I do this in Keras?


texts:
I am analyzing a population movement pattern and I would like to design a visualization like this one:

Is there any tool or language (preferably R) I can use for that purpose?

Here is a sample of my dataset.


texts: Why do convolutional networks work so well for images?

texts: Convolutional artificial neural networks work well in particular with images. Why?


texts: For short sentences(max length 10 ), which Name entity recognition algorithm is good?

texts:
My Training data look like this .

I have to recognize 4 class for each sentence.
Any algorithm , which have some learning parameters Means not rule based approach . So which method is good for my problem ?


texts: Random forest model in R - predictors and training data types mismatch

texts:
I tried the random forest model in my research topic, but I met a problem during the validation phase.

When, I used the final model of random forest to predict on an independent dataset, I received this message:

  Type of predictors in new data do not match that of the training data

So, to detect the different categories in my factors/variables, I used:

levels(Train$Aquifer.media)           levels(Test$Aquifer.media)

For this factor "Aquifer.media", I have:

Train dataset: "Carbonates rocks"  "Crystalline rocks"  "Siliciclastic sedimentary rocks"  "Unconsolisated sediments rocks"  "Volcanic rocks"

Test Dataset: "Crystalline rocks"  "Siliciclastic sedimentary rocks"  "Unconsolisated sediments rocks"  "Volcanic rocks"

I detected that predictors  were of different categories, I would like to know, how I can solve this problem?

Is it possible  to delete some categories in the factors?


texts: algorithmic difference between image analysis and video analysis

texts:
Is there algorithmic difference between analyzing video and an image, say for example,if I want object recognition? Or do I just have to analyze every frame of the the video just as an image?

Example, detecting an object in a single image is easy compared to video becuase time dimension is added to video. In addition, on video, in every frame, the object is most probably moving, which makes the frames in motion... So how can u handle the time factor and the "in motion" part in the video. Those are the problem I imagine in video, it would be nice if u add ur own thoughts on it. Thanks


texts:
My dataset( Network traffic dataset where we do binary classification)-

Number of features in data is 25
Can we use this kind of dataset in NLP models like transformers? If yes what should be the transformations to be done on this to feed it into models like transformers for binary classification?


texts:
I am trying to build a classifier to predict the ratings of a show during a specific time.

I have extracted around 109 features, some relating to the time field namely,

Day of Year
Month of year
Is on a weekend?
During business hours?
Public Holiday?

I also included some categorical features and used a label binariser for which channel it appeared on, and the broadcaster.

I wanted to check the linearity of the dataset, which would inform me as to whether a linear regressor could be used or something non-linear like a neural network. I decided to do dimensionality reduction using PCA in order to visualise if the dataset was linearly separable in 2D.

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
data_scaled = pca.fit_transform(df[cols])
plt.plot(data_scaled[:,0], data_scaled[:,1], 'ro')
plt.xlabel('first component')
plt.ylabel('second component')
plt.show()

I am very confused by the result and am not able to interpret this.

The plot of the first component:

The plot of the second component:

What could the PCA results tell? What would cause these graphs?


texts: compare pandas dataframes with different lengths

texts:
I have two dataframes of different lengths and I need to add a column to the first one with filtered values, e.g.

df1 = pd.DataFrame({'Object':['cup', 'brick', 'board', 'stone'], 'id':[2, 8, 9, 6]})
df1 = pd.DataFrame({'Thing':['cup', 'board'], 'color':['blue', 'grey']})

and I want to create

df = pd.DataFrame({'Thing':['cup', 'board'], 'color':['blue', 'grey'], 'id':[2, 9]})

All methods I tried to use complained about different lengths.


texts: Issue with implementation of Sequence to Sequence Autoencoder

texts:
Hi,
I want to implement Sequence to Sequence Autoencoder for text message classification purpose whether it is spam or ham.

The complete code is:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from keras.models import Model
from keras.layers import LSTM, Activation, Dense, Dropout, Input, Embedding, RepeatVector, Flatten
from keras.optimizers import RMSprop
from keras.preprocessing.text import Tokenizer
from keras.preprocessing import sequence
from keras.utils import to_categorical

#Load the data
df = pd.read_csv('/spam.csv',delimiter=',',encoding='latin-1')
# Drop the columns that are not required for the neural network.
df.drop(['Unnamed: 2', 'Unnamed: 3', 'Unnamed: 4'],axis=1,inplace=True)
df.info()
# Create input and output vectors. and Process the labels
X = df.v2
Y = df.v1
le = LabelEncoder()
Y = le.fit_transform(Y)
Y = Y.reshape(-1,1)

# Tokenize the data and convert the text to sequences.
# Add padding to ensure that all the sequences have the same shape.
# There are many ways of taking the max_len and here an arbitrary length of 150 is chosen.
max_words = 1000
max_len = 150
tok = Tokenizer(num_words=max_words)
tok.fit_on_texts(X)
sequences = tok.texts_to_sequences(X)
sequences_matrix = sequence.pad_sequences(sequences,maxlen=max_len)

# Split into training and test data.
X_train,X_test,Y_train,Y_test = train_test_split(sequences_matrix,Y,test_size=0.2)

X_train = np.reshape(X_train, (X_train.shape[0], timesteps, X_train.shape[1]))

# Model Definition
latent_dim = 128
timesteps = 1
input_dim = 150 #no of features

inputs = Input(shape=(timesteps, input_dim))
encoded = LSTM(128,return_sequences=True)(inputs)
encoded = LSTM(64)(encoded)

decoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(128,return_sequences=True)(decoded)
decoded = LSTM(input_dim, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)

sequence_autoencoder.compile(optimizer='RMSProp', loss='mse')
sequence_autoencoder.fit(X_train, X_train,
                         epochs=100,
                         batch_size=5,
                         shuffle=True)

When, I'm executing this, getting loss something 7000+. Also, the loss is almost remaining constant even after 100 iterations. Not understanding where I'm going wrong.

Is there an issue with setting timesteps = 1?

If I set timesteps = max_len then getting an error: ValueError: Error when checking input: expected input_1 to have 3 dimensions, but got array with shape (4457, 150).

I tried to solve this by changing X_train=np.expand_dims(X_train, -1) so that shape of X_train becomes (4457, 150, 1). Then I'm getting error "ValueError: Error when checking input: expected input_1 to have shape (150, 150) but got array with shape (150, 1)"

Please help me to come out of this issue.
Thanks in advance!

After successful training of sequence_autoencoder model, I'll take only encoder part from this and attach a classifier layer for classification.

Note: Even LSTM model is sufficient for this task, however, I want to reduce the dimensionality of features extracted from the input and then apply classifier for classification. Another aim is also to understand how to implement LSTM autoencoders.

I have taken help from https://www.kaggle.com/kredy10/simple-lstm-for-text-classification/ for text preprocessing implementation and referred https://blog.keras.io/building-autoencoders-in-keras.html for model building.


texts: Is there a name for a scale which mixes ordinal and nominal?

texts:
The textbooks I have differentiate between nominal, ordinal, interval and ratio scales. The ordinal scale is quite popular in the wild, used for basically all subjective data, and also for dividing interval and ratio scales into better digested bins.

But when dealing with the data itself, the need for different "exceptions" arises. Sometimes it is just different kinds of null flavors, which can be written off as a meta data issue. But there are also more interesting cases. I have encountered satisfaction measurements where the overall satisfaction with a feature is considered to be "excellent", "good" or "bad" if the distribution of answers peaks at the expected points in the scale, but "polarising" if the distribution is U-shaped, and "unclear" if there is a flattish or weirdly multimodal distribution. Obviously, the "excellent", "good" and "bad" part is ordinal, but the last two are nominal. Other examples include a "prefer not to say" box in self-reported measures, if the reluctance of giving information is meaningful under the employed model.

I am pretty sure that we don't yet have qualitative methods to analyze these mixed scales. But I hope that the situation is at least recognized in theoretical texts, even if the only recommendation is "treat them like nominal". Also, while a mixed statistical analysis is difficult, maybe there are at least visualization approaches capable of representing the true complexity of the situation. But I don't know how to look for such literature, since I have never encountered a name for this kind of scale.

Does somebody know of a name (even if it is not widely established) and can point to sources which define it and deal with it?


texts: Preprocessing text so that two words without a separating space (or hyphen separated) are detected

texts: Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the same thing? i.e. replace all such occurrences by "bigram". I should also mention that I have no prior knowledge of what exact bi-grams  are present in the corpus.


texts: Grid seach is unavailable for Keras in case of multiple outputs?

texts:
I do experiments with the following Keras architecture with multiple outputs:

def create_model(conv_kernels = 32, dense_nodes = 512):
    model_input=Input(shape=(img_channels, img_rows, img_cols))
    x = Convolution2D(conv_kernels, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Convolution2D(conv_kernels, (3, 3), kernel_initializer='he_normal')(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Dropout(0.25)(x)
    x = Flatten()(x)

    conv_out = (Dense(dense_nodes, activation='relu', kernel_constraint=maxnorm(3)))(x)

    x1 = Dense(nb_classes, activation='softmax')(conv_out)
    x2 = Dense(nb_classes, activation='softmax')(conv_out)
    x3 = Dense(nb_classes, activation='softmax')(conv_out)
    x4 = Dense(nb_classes, activation='softmax')(conv_out)

    lst = [x1, x2, x3, x4]

    model = Model(inputs=model_input, outputs=lst)
    sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

    return model

When I try to apply grid search this way:

model = KerasClassifier(build_fn=create_model1, epochs=nb_epoch, batch_size=batch_size, verbose=0)
param_grid = dict(conv_kernels = [16, 32, 64], dense_nodes = [128, 256, 512])
grid = GridSearchCV(estimator=model, cv=4, param_grid=param_grid, n_jobs=-1)
grid_result = grid.fit(X_train, Y_train)

I get the following error message:

  ValueError: Found input variables with inconsistent numbers of samples: [9416, 4]

9416 is the number of training examples, 4 is the model output number.

What is the problem here? Grid search is unavailable for multiple outputs? If so, what is the best way to apply parameter search (apart from pure manual method)?


texts: Fast introduction to deep learning in Python, with advanced math and some machine learning backgrounds, but not much Python experience

texts:
I've the following somewhat unusual background and I've managed (probably by luck) to get an industry job of a computer vision researcher using deep learning.

My background: I've a PhD in pure math, and have the following machine learning experience: linear and logistic regressions, support vector machines (SVM), linear and quadratic discriminant analyses (LDA/QDA), decision trees, PCA. I'm familiar with the mathematical theory of all the above, and implemented the LDA/QDA, SVM, PCA in matlab, where I'm proficient in.

However, my experience with Python and deep learning are very limited to zero. I don't know scikit learn, only somewhat familiar with NumPy arrays, but still have problems with the basics like concatenation with empty arrays. I don't know for examples the data structures like classes or lists in Python-just to give you an idea. I know what a neural network is, but don't know anything else, e.g. backpropagation or autoencoder. Hence I don't know the theory of deep learning as well.

Time constraint: So I'm in a situation where I'll have to gain some hands on experience in both Python as well as deep learning, possibly in 2 months (the company has a 4 months trial period and use only Python, but they'll get a sense of my learning in 2 months, hence I've to learn fast).

Question: Given my expertise and time constraints, could you mention a road map (if possible) so that I can get a fast introduction to deep learning and its Python libraries (e.g. Tensor Flow or Keras etc.)? I know there're tons of resources out there, but not all of them can be learnt within a short time span.

Thanks so much in advance!!!


texts:
I am trying my first 'project' concerning machine learning and I am a bit stuck.
However, I am not sure if it's even possible but here goes my question.

What I want to achieve is clustering user groups based on the amount of visits a user does on a certain website.
So I started out with this feature matrix:

USER    abc.be  abc.be/a    abc.be/b    xyz.be  xyz.be/a
123      0        0           0            0      1
456      1        0           1            0      0
789      2        3           1            0      0
321      1        0           1            0      1
654      1        1           1            1      1
987      0        1           0            3      0

So I got in this example 5 features (my 5 different websites).
So then I used PCA to come to 2 dimensions, so I could plot it and see how it went.

My feature matrix (in my example) is 5 columns * 6 rows.

My PCA matrix is 2 columns * 6 rows.

I came to this plot (please note that this plot uses different data then the example but the idea is the same)

The green points are my PCA points
The red circles are my K-Means centroids.

But the part I am struggling with is this: so I got my clusters (red circles) but how can I use this to say:"Looks like most users go to either site A or site B)?

So how can I couple my clusters to a feature label from my feature matrix?

Or how does one approach this?

Any help is appreciated :)


texts: Simple explanation of LSTM data set and training phase

texts:
I cannot understand the training procedure of the LSTM (and other recurrent nets).

My data is time series of length 2000 points. As suggested on the internet (and keras framework), this should be split into subsequences of 200-400 and fed into LSTM (due to performance issues). How is this justified?

According to the maths of back-propagation, we should take 1 sequence (all 2000 points) and feed it to LSTM, which internally will compute the forecast using all 2000 points when spread through time, and update weights in 1 gradient descent step. Thus, we generally feed 1 sequence of 2000 points into LSTM until the optimization converges, which again is the same as having the training set of 1 sample, and each loss function calculation and weight update is 1 epoch.

When we have subsequences (say, 4 subsequences of 500 points each from 2000 point time series), they are not connected to each other. We feed them to LSTM as 1 dataset of 4 samples. Are they passed to LSTM as separate samples, or a batch (with weights adjusted for the whole batch per 1 epoch)? Why is this operation justified? Does it not influence the output, since the samples of 500 points are actually connected to each other?


texts: How to use machine learning to create combine of opposite images side by side

texts:
Inspired by: Two Worlds Pictures

I just want to create a Machine Learning Model that can automatically combine the opposite images into 1 image.

I am thinking about 2 possible solutions:

Pose Estimation: Detect humans and their poses from image data and search an archive by pose, but totally out of context.
Land Lines: explore similar lines.

It's just my ideas, do you have any recommendations?
Thanks


texts: getting the weights of intermediate layer in keras

texts:
I have an image dataset 376 classes each class has 15 pictures corresponds to a person. I would like to get the feature vector that corresponds to each person.

What I have done is, after I compiled the model I then used this link
as a reference to get the weights of the last convolutional layer. However, when I do this, I get the error:

InvalidArgumentError: You must feed a value for placeholder tensor 'conv_layer' with dtype float and shape [?,19,19,360]

How can I resolve this issue?

Here is the code that I have done so far:

train_data = np.array(train_data, dtype=np.float32)
test_data = np.array(test_data, dtype=np.float32)
train_data = train_data / 180 # to make the array values between 0-1
test_data = test_data / 180
train_label = keras.utils.to_categorical(train_label, 376)
test_label = keras.utils.to_categorical(test_label, 376)
# CNN MODEL
model = Sequential()
model.add(Conv2D(180, (3, 3), padding='same', input_shape=(180, 180, 3),
activation="relu")) #180 is the number of filters
model.add(Conv2D(180, (3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.25))
model.add(Conv2D(360, (3, 3), padding='same', activation="relu"))
model.add(Conv2D(360, (3, 3), activation="relu"))
conv_layer = model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.25))
flatten_layer = model.add(Flatten())
model.add(Dense(496, activation="relu"))
model.add(Dropout(0.5))
dense_layer = model.add(Dense(376, activation="softmax"))
#compiling the model
model.compile(
loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy']
)
model.fit(
train_data,
train_label,
batch_size=32,
epochs=40,
verbose = 2 ,
validation_split=0.1,
shuffle=True)
# getting intermediate  layer weights
get_layer_output = K.function([model.layers[0].input],
                              [model.layers[11].output])
layer_output = get_layer_output([conv_layer])[0]


texts: Justification for values used in backpropagation

texts:
I'm learning the method for backpropagation in adjusting weights. A generalization of a formula used to determine the change made to a respective weight is

where  is the rate the total error changes as the i-th weight changes. I get why this value is interesting in the context of what we're trying to accomplish with backpropagation, but I don't totally understand why this is a good value to use when determining how to adjust the new weight, particularly when there's multiple weights influencing the total error. Why is this a "good" number to use to when changing our weight?


texts: Rescaling simulations by machine learning

texts:
I am working in the field of astrophysics. A problem that I consistently find in my projects is that I am always limited by the size of the sky simulations available for a specific observable that I am considering for my project. Simulations are crucial, in particular, to estimate covariance matrices, which are essential elements of any Bayesian inference analysis.

The tendency in the field is to have fairly-high resolution of small fractions of the sky. It is computationally very challenging to generate significantly larger sky areas (something like 1/3 of the sky, which is what would be needed for real applications). Therefore, I was wondering if there is a machine learning-based approach to rescaling simulations that are too small for practical uses, extending them to much larger areas. A few large-area simulations are probably computationally feasible, but I am doubtful that they could be in such a quantity to represent a decent training set for any machine learning algorithm.


texts:
Why Gaussian mixture model uses Expectation maximization instead of Gradient descent?
What other models uses Expectation maximization to find best optimal parameters instead of using gradient descent?