texts: Which algorithm can be used to reduce dimension of multiple time series?

texts:
In my dataset, a data point is essentially a Time series of 6 feature over a year per month so in all, it results in 6*12=72 features.
I need to find class outliers so I perform dimensionality reduction hoping the difference in data is maintained and then apply k-means clustering and compute distance.

For dimensionality reduction I have tried PCA and simple autoencoder to reduce dimension from 72 to 6 but results are unsatisfactory.

Can anyone please suggest any other way to reduce dimension of this type of data.


texts: How to generate data if algo itself is involved in the process with a feedback loop?

texts:
I have an algorithm which would be a rather easy classification task with a set of features and a class output which I would like to solve with a machine learning algo.

But I am having issues and doubts about the data generation. The features my algo uses as inputs are processed beforehand by other algorithms and more importantly, also have a feedback loop to my the algorithm I want to change.

Basically, the better my algo is getting, the less false positives there should be. But with less false positives, I have more and more imbalanced data to work with, which would mean, it is harder to train the algorithm.
I could reduce the performance of my algo on purpose and generate data, but then I am not sure if the data I am getting is any meaningful as there is a feedback loop.

To me this seems like a chicken, egg problem.


texts:
I've been taking the histogram of the optical flow in videos and plotting the kurtosis and skewness of each frame. At the end of the video, I noticed that the skewness and kurtosis follow each other - ie, when the skewness goes up so does the kurtosis and same when it goes down. In fact, the kurtosis almost looks just like a scaled version of the skewness.
I know skewness and kurtosis are supposed to be different concepts entirely, since they are different moments (the graphs of x^3 and x^4 do not look similar at all ) but it just makes me wonder why take both when they both seem to be similar? How are they useful separately? Also would it be due to my distributions? For what distributions would this kind of behaviour occur?
For reference the functions I am using are scipy.stats.skew and scipy.stats.kurtosis so I'm assuming that they are using sample skew and kurtosis


texts: What is the best references / books to get familiar with machine learning algorithms?

texts:
I'm just starting off as a data scientist and i need to understand how regression, random forest and much more.. algorithms function under the hood, the maths behind them and when to use them. I started with some MOOC courses but not a fan, it's just not my style of learning as i need practical examples or books i can dive into with detailed examples i can just apply myself after understanding how each algorithm works and some 20 mins vids with scarce maths explanation is not what i aim for, i've done some research and ended up with these books:

-python for data analysis

-Python Crash Course

-oreilly hands on machine learning with scikit learn and tensorflow

-automate the boring stuff with python

kindly note that i have some python experience(less than a year) and  unfamiliar with tensorflow , keras, scikit-learn etc..

Kindly also note that i didn't mention any maths books to explain how the machine learning algorithms work so i'm open to suggestions  .


texts: Merge columns of coordinates and construct sequence

texts:
I have to following data frame.

    x1     x2          x3
    0   58.33546    15.14554
    0   58.33604    15.14571
    0   58.33645    15.14581
    0   58.33681    15.14591
    0   58.33722    15.14600
    0   58.33810    15.14604
    0   58.33900    15.14599
    0   58.33966    15.14578

Consisting of long and lat, each pair belongs to a string (0,1,3,4 and so on, only the string 0 is displayed above).

I would like to first construct a new column with the pairs (x2,x3). How do I do that?

The coordinates above corresponding to 0 is actually a journey (say X), if there is another journey (Y) whose first coordinate is the same as X last, I would like to merge the sequence X and Y coordinates together and call that journey X (the continuation of X simply).

My data set is called data, for some reason merging them together wont work with

data[, x4 := paste(data$x2, data$x3)]

This will create a new column, only isplaying x2.


texts: Difference between Jordan, Elman and normal RNN

texts:
As far as I know for history, the Jordan network was proposed first in 1986 as a form of RNN with this diagram:

Actually, this is the solution that makes sense when thinking about sequence data that the current output is an input in the next time (with some weight and activation as shown in figure).
However, after this in 1990, Elman network was proposed with feeding back the hidden state not the output like this?

What is the reason or benefit of this modification?
And what is the difference between these two types of network and the ordinary RNN (before LSTM and GRU) we know as shown in this figure?

The figure of RNN seems to be very similar to both of them (Elman especially) because we take the hidden state and make it an input again.
What is the difference between RNN and Elman and Jordan networks? and what is the usage difference in both of them?
Please note that I am taking about RNN before LSTM and GRU. They are out of comparison.


texts:
I am trying to get this repo of Xu's DONUT algorithm running, however I am getting an error I do not quite understand. The readme says I should load raw_data as follows:

timestamp, values, labels = ...
# If there is no label, simply use all zeros.
labels = np.zeros_like(values, dtype=np.int32)

# Complete the timestamp, and obtain the missing point indicators.
timestamp, missing, (values, labels) = \
    complete_timestamp(timestamp, (values, labels))

however, when I do so, I get this error:

ValueError: The shape of ``arrays[0]`` does not agree with the shape of `timestamp` ((109577, 11) vs (109577,))

which does not make sense to me as I can't think of a reason timestamp would be an 11 dim array. When I pass the values as the timestamp arg, I get "timestamp must be a 1D array"

Very confused, hopefully, someone can shed some light.

Here are the checks in the code:

if len(timestamp.shape) != 1:
    raise ValueError('`timestamp` must be a 1-D array')

has_arrays = arrays is not None
arrays = [np.asarray(array) for array in (arrays or ())]
for i, array in enumerate(arrays):
    if array.shape != timestamp.shape:
        raise ValueError('The shape of ``arrays[{}]`` does not agree with '
                         'the shape of `timestamp` ({} vs {})'.
                         format(i, array.shape, timestamp.shape))

As well as the repo itself:
https://github.com/haowen-xu/donut


texts: Which type auto encoder gives best results for text

texts:
I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to use the same auto encoders as the ones I used for the images I get bad results.

I guess the reason for this is that my text is sparse and I have a big vocabulary size of 500K words.

Do you have a link of a working example of an auto encoder for text in Keras?
I saw that in most papers they use cross-entropy as a loss function. How does cross-entropy calculate the loss exactly ? Does it make sense to use cross-entropy even if I do a character by character auto encoder?


texts: Is there a thumb-rule for designing neural-networks?

texts: I know that a neural-network architecture is mostly based on the problem itself and the types of input/output, but still - there's always a "square one" when starting to build one. So my question is - given a input dataset of MxN (M  is the number of records, N is the number of features) and a C possible output classes - is there a thumb-rule to how many layers/units should we start with?


texts:
I am new in datamining. I decided to play with k-means clustering in r with simple аrtificial data.

set.seed(101)
x1 <- runif(100,36.6,37.5)
x2 <- runif(100,37.6,38.4)
x3 <- runif(100,38.5,40)
x<-c(50,150,250)
y<-c(37,38,39.25)

centers<-c(37.05,38,39.25)
all <- c(x1,x2,x3)
c1<-kmeans(all,centers=centers,iter.max = 1000,nstart=1,algorithm="Lloyd")
plot(all,col = c1$cluster)
points(x,y,col="red",pch=19)

I generated three random data sets x1,x2,x3. Then i taked center of each cluster like mean of each set and used k-means algorith of r. In result i get what you see in picture.

What i am making wrong?

Why second cluster contains the part of third set? How to improve result?

Why result is so bad instead sets is lineary separate?

The main question is how to improve clustreing result?


texts: How to measure the stability of hyperparameter selection in a model-building procedure?

texts:
For my project I run several model-building-procedures. I use the mean and standard deviation of the test scores in the outer folds as an estimator for the generalizability of the model-building procedure. However I wonder if there is also a way to estimate the stability of the selected values for my hyperparameters? Something like a distance-measurement?

Example 1:

As an example, let's say I ran a nested cross validation with a Support Vector Machine as Classifier. I implemented C and Kernel as hyperparameters. Note that C is a continous variable whereas Kernel is a categorial variable. I also smoothed my data and implemented the Smoothing FWHM as hyperparameter. I get the following values for my hyperparameters over the outer folds:

You can see that each surrogate-model contains slightly different values for my hyperparameters.

Example 2:

In the second example you can see that each of my surrogate models won with the same selected values or categories for my hyperparameters. Thus this model-buidling-procedure could be called more stable with regards to hyperparameter value selection.


texts: Why won't my SVM learn a sequence of repeated elements

texts:
I recently started playing with SVMs for a one class classification, I was able to get some reasonable classifications from real data and but was trying to optimize the nu and gamma parameters when I came across this example:

In the code below, I train an SVM with an array of ones, then I present the same array of ones for classification and it classifies all ones as outliers.

import pandas as pd
from sklearn import svm
import numpy as np

nu = 0.01
gamma = 1
ones = pd.DataFrame(np.ones(100))
clf = svm.OneClassSVM(nu=nu, kernel="rbf", gamma=gamma)
clf.fit(ones)
ones["predicted"] = clf.predict(ones)
#Returns -1 for all entries

My question is: why does this happen? I thought this data would be trivial for any parameter configuration.


texts: Tensorflow - logistic regression -oneHot Encoder - Transformed array of different size for both train and test

texts:
x_train = tr1.loc[:, ['Sepal Length', 'Sepal Width', 'Petal Length', 'Petal Width']]
#x_train.shape - (120 x 4)

y_train = tr1.loc[:, ['Species']]
#shape - 120 x 3

x_test = test1.loc[:, ['Sepal Length', 'Sepal Width', 'Petal Length', 'Petal Width']]
#shape 30 x 4
y_test = test1.loc[:, ['Species']]
# shape 30 x 3

oneHot = OneHotEncoder()
oneHot.fit(x_train)
# transform
x_train = oneHot.transform(x_train).toarray()
 # fit our y to oneHot encoder
 oneHot.fit(y_train)
 # transform
 y_train = oneHot.transform(y_train).toarray()

 oneHot.fit(x_test)
 # transform
 x_test = oneHot.transform(x_test).toarray()
 # fit our y to oneHot encoder
  oneHot.fit(y_test)
   # transform
  y_test = oneHot.transform(y_test).toarray()

   print("Our features X_test1 in one-hot format")
    print(x_test)

Shape of x_train:  (120, 15)
Shape of y_train:  (120, 3)
Shape of x_test:  (30, 14)
Shape of y_test:  (30, 3)

a) After conversion why is the size x_test = 30 x 14  I assume it has to be  30 x 15 ?

# hyperparameters
learning_rate = 0.0001
num_epochs = 100
display_step = 1

# for visualize purpose in tensorboard we use tf.name_scope
with tf.name_scope("Declaring_placeholder"):
# X is placeholdre for iris features. We will feed data later on
x = tf.placeholder(tf.float32, shape=[None, 15])
# y is placeholder for iris labels. We will feed data later on
y = tf.placeholder(tf.float32, shape=[None, 3])

with tf.name_scope("Declaring_variables"):
    # W is our weights. This will update during training time
    W = tf.Variable(tf.zeros([15, 3]))
    # b is our bias. This will also update during training time
    b = tf.Variable(tf.zeros([3]))

with tf.name_scope("Declaring_functions"):
# our prediction function
y_ = tf.nn.softmax(tf.add(tf.matmul(x, W), b))

b) did I define x, y, W, b correctly because when I run the accuracy I get this error "ValueError: Cannot feed value of shape (30, 14) for Tensor 'Declaring_placeholder_10/Placeholder:0', which has shape '(?, 15)'
"


texts: Anonymize continuous variable for masking purposes

texts:
I am about to kick off a large hackathon event.
We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels deep.
Data provider wants to 'mask' the data such that the original values cannot be reverse-engineered.  I'm not worried about the categorical variables, this is simple.  But the continuous variables are tricky.

a logarithmic transformation is easily reverse engineered
a nonlinear transformation is better, but will mess with the relationship of values between categories
a pure linear transformation would work, but doesn't seem to 'mask' enough.

I need to preserve the relationships between numbers whilst also protecting the actual, true values.
Ideas greatly appreciated.


texts: Compare two tends with big difference in absolute value

texts:
I'm studying the Spotify streams trend for different nations in order to show the impact of the pandemic on music listeners.
So far i retrieved my data, and plotted it.

Obviously, since the various nations have different numbers in population, spotify users ecc... this graphic doesn't tell much.
So i decided to scale every curve dividing its values by its peak.  So for example, the US has a maximum value of 3.50 million streams, i normalized that curve with that value and i did the same for all the other states (with their maximum value) and I obtained this:

Could this be a good approach? In general which approach should I use if i want to compare different curves which have a really big difference in absolute values?
EDIT : In the end i normalized my data using a zscore normalization for each single line. So basically I computed the mean and standard deviation for each state and then I normalized the single state with its mean and std.
This is the resulting plot:

Is this a good approach? Can I now compare the different trends and conclude that there's an overall decrease in the period between 10 and 15 weeks?


texts: Problem on dataset skewness and deviation

texts:
This is the problem statement :

Your data set has missing values and is positively skewed with skewness = 1. Further examination tells you that they are spread along 1.5 standard deviation from the median. How much data would remain in % and Why?


texts:
I am currently building a (simple sequential) fully convolutional network for an object recognition task.

Designing the architecture essentially amounts to choosing kernel size, stride and unit number for each layer.
While doing this, I have to keep track of several properties of my layers:

parameter count
field of view of each kernel, wrt to the input/data
a margin in the last layer that I need to account for when creating the label images

I've written a small script that computes these quantities for different CNN configurations. It's not very complicated, but…

I'm under the impression that everyone using CNNs has to do this, so there should be a (graphical) tool for this. It could also be used to draw the fancy architecture diagrams. Because it works on the same input data.

Is there such a tool? How do you compute these quantities for your models?


texts: Algorithm for document retrieval in QA system

texts: I am working with question answering and machine reading comprehension system. I want to match questions and documents (around 100,000 docs) in database. I've used tf-idf but it accuracy is about 55% and I need to be at least 80%. Could you give me some advice?


texts: How to handle string features in a neural network

texts:
I am new to ANNs. How to handle features whicvh consist of strings? E.g. if I have group featire which are different groups conatining differnt capital Leters and numbers.

Now I have the train data set containing all such important features.

My first guess is just to replace stings with numbers. But you have different uncorrelated groups of strings like the category or the manufacturer.


texts: Compare multiple values from a DataFrame against a single row from another

texts:
I'm trying to compare address values for inaccuracies, for example, given multiple records like:

Reference
Apartment
Address
PostCode

AS097
NaN
00 Name Road
BH1 4HB

AS097
Flat 1 Building Name
00 Name Road
BH1 4HB

AS097
Flat 2 Building Name
00 Name Rd
BH1 4HB

AS097
Flat 3 Building Name
00 Name Road
BH1 4HB

AS097
Flat 4 Building Name
00 Name Road
BH1 4HB

AS097
Flat 5 Building Name
00 Name Road
BH1 4HX

HO056
NaN
23 Street Road
XG9 9GX

I've a dataframe where I store all the "main" addresses by checking if the Apartment column is empty like so:
main_address = df['Apartment'].isnull()

df_st = pd.DataFrame({'Reference':df[main_address].Reference, 'Address':df[main_address].Address, 'PostCode':df[main_address].PostCode})

df_st will look like this:

Reference
Apartment
Address
PostCode

AS097
NaN
00 Name Road
BH1 4HB

HO056
NaN
23 Street Road
XG9 9GX

df has over 1K records, but df_st containing the "main" address ends up with approx. 200 records.
I'm trying to create a new DataFrame where I can identify where the records don't match by domparing df against df_st.
THE PROBLEM
I try the below:
# Clean the Reference values
refs_list = df['Reference'].str.split('/').str[0]
df['Reference'] = refs_list

# Create a new column titled issues and flag if the references match
df['issues'] = np.where(df['Reference'] == df_st['Reference'], 'True', 'False')

I want the above for the Address and PostCode unfortunately this does not work since df and df_st don't have the same shape.
I'm struggling to find a way to achieve a comparison between the two DataFrames df against df_st.
I want to compare all matching Reference row values from df against it's matching from df_st and if one of them don't match create a new column title Issues and store the conflicting column there.
MY DESIRED OUTCOME
Given the data above, after comparing df data against df_st results in a new DataFrame, like below

Reference
Apartment
Address
PostCode
Issues

AS097
NaN
00 Name Road
BH1 4HB
None

AS097
Flat 1 Building Name
00 Name Road
BH1 4HB
None

AS097
Flat 2 Building Name
00 Name Rd
BH1 4HB
Address

AS097
Flat 3 Building Name
00 Name Road
BH1 4HB
None

AS097
Flat 4 Building Name
00 Name Road
BH1 4HB
None

AS097
Flat 5 Building Name
00 Name Road
BH1 4HX
PostCode

HO056
NaN
23 Street Road
XG9 9GX
None

Where Address appears as an issue in the column Issues since the address don't match against df_st, same for PostCode since it differs from df_st
IN A NUT SHELL
All I want to to know how to compare matching rows by Reference from a DataFrame against another and compare the other values Address and PostCode.
Hope that makes sense.


texts: What is the logic of the epoch? For example, 1 time, 2 times, etc. I just do not know what else is working to give better results than I know.


texts: Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

texts:
Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is necessary for independence of individual trees.
But why can't we just resample as follows: for tree 1, randomly sample $m$ observations without replacement out of the $n$, where $m$ is still large enough (of course, provided that $n$ is large enough in the first place). Then replenish all observations and repeat the resampling for tree 2, and so on.
Even in this case, I'd imagine that the individual subsamples would be independent. So is there an additional reason for resampling with replacement in bagging?


texts:
I have the following dataset which is a .json file:

and I would like to get the first word for every string inside lista_asm, so I would like to get: jmp,push,uncomisd,...etc
what I am doing to do this is the following:
dataFrame['opcodes'] = dataFrame['lista_asm'].apply(lambda x:[i.split()[0] for i in x])

but it gives me back the following error message:
   3589             else:
   3590                 values = self.astype(object).values
-> 3591                 mapped = lib.map_infer(values, f, convert=convert_dtype)
   3592
   3593         if len(mapped) and isinstance(mapped[0], Series):

pandas/_libs/lib.pyx in pandas._libs.lib.map_infer()

<ipython-input-18-5506b5721bf1> in <lambda>(x)
----> 1 dataFrame['opcodes'] = dataFrame['lista_asm'].apply(lambda x:[i.split()[0].strip() for i in x])

IndexError: list index out of range

I don't understand what is wrong. Can somebody please help me?
[EDIT]Trying the code:
dataFrame['opcodes'] = dataFrame['lista_asm'].apply(lambda x:x[0].split(" ",2)[0])

and adding :
df = dataFrame[["opcodes", "semantic"]].copy()
df

I get:

what I would like to get is a list of the type [push,mov,..] and this for every row.
It seems like when I do x[0] it does not return the first element of the list, but returns the pharentesis, which is weird. Am I doing something wrong which I don't see?
My objective is to pre-process this dataset in order to feed features to my model, but I haveing hard times in doing so.


texts: How is learning rate calculated in sklearn Lasso regression?

texts:
I was applying different regression models to Kaggle Housing dataset for advanced regression. I am planning to test out lasso, ridge and elastic net. However, none of these models have learning rate as their parameter.

How is the learning rate calculated for these model? Is it dependent on the dataset being trained? I know these models are regularized linear regression and must use learning rate to update their model weights. Or is their a different way to update the model?


texts: Deep Learning with Keras. Classification problem with large data

texts:
I have looked on the internet and found a lot of discussion about image classification problems, but almost none of just classification problems without images. I am trying to build a model with Keras using Python, I have a dataset of 827 features (43 are integers, and the rest is one-hot encoded) and 24,800 samples. My target is 3 classes one-hot encoded as well ( [1 0 0] [0 1 0] [0 0 1] ).

I have found a lot of complex image recognition networks but only simple deep networks for non-image problems. I need help on how I can make the deep neural network architecture for my problem.


texts: Is it possible to measure the object using deep learning

texts: Is there a way we can measure the length, width and the depth of an object in the picture using deep learning?


texts: Showing standard deviation for training curve

texts:
I am training a neural network and I wanted to plot the evolution of different metrics (MSE…) during training. To get an idea of the variations between between different trainings, I am using several models and plotting average value and standard deviation. My problem is the following: I did not manage to find a good way to plot this curve and also did not find any good explanation on internet.
Let's denote by y the metric to be displayed, and x is the number of epochs. On the plots below, one standard deviation is given by the shaded area.
Here is what I tried and why I think it is not working:

A linear scale is not great because the metric spans different order of magnitude. In particular, the early values are much larger than late values. One possibility could be to start the curve at epochs 5, for example, to remove the early large values, but I would prefer to display everything.

A log scale seems much more natural in view of the previous point, however variations can be large, of similar scale compared to the scale of the mean value. Since the space between two ticks is geometrically distributed, one standard deviation below the curve appears much bigger than above. This tends to produce large spikes on the figure, or even completely fill the area below the curve.

I have tried to display the relative standard deviation instead (rescaled by $\log_{10} \mathrm e$). This is the appropriate standard deviation for the variable $z = \ln y$. However, I dont want to plot z but really y in log scale. Nonetheless, I thought it's fine to plot the relative standard deviation if it displays better. But, I am not sure if it's really intuitive what it shows. A worse problem is that the standard deviation appeard very large for small values (since we divide by a number < 0).

Finally, I tried to use the symlog scale but I am not sure it's a good solution: if the linear threshold is large, then I find that that it makes the figure difficult to interpret. If it is small, we get the same problems as for the log scale. Below is a plot for a threshold of 0.01.

Conclusion: what would be the best way to display the standard deviation a log scale?


texts: What should I check if model accuracy is no better than baseline level(random guess)

texts:
I have a data with only 8 columns:

id
created_time
employee_id
rank position
hourly price
num_work_completed
work_category
hired

Hired is the target variable with 1 representing hired and 0 representing not hired, and it is imbalanced with 5.7% hired(1) which makes the baseline accuracy 94.3%
I am trying to build model that predict whether a employee will be hired.
After I finished the EDA, feature engineering(dealing with NAs, encoding categorical variables, normalizing numeric variables), I used 80/20 as the splitting rule and built random forest with rank_position,hourly_price, num_work_completed, work_category_dummy

clf=RandomForestClassifier(n_estimators=100,class_weight=balanced)
clf.fit(X_train,y_train)
y_pred=clf.predict(X_test)

However the model's accuracy(Test accuracy) turned out to be 93% while the baseline is 94.3%.
The training accuracy is 99%. Compared to test accuracy 94.3%, I don't think there's a over-fitting problem
The logistic regression also has the same problem.
Based on correlation blot, most independent variables have pretty weak relationship with target variable smaller than +/- 0.3.
what should I do next to improve my model accuracy?
I tried parameter tuning but it doesn't help a lot.


texts: Keras custom layer using tensorflow function

texts: I was wondering if there is any other way to write my own Keras layer instead of inheritance way as given in their documentation? My colleague is more comfortable in tensorflow and he gave me a tensorflow function that does the job of the layer. Do I necessarily have to rewrite it as Keras' inherited layer?


texts:
So I designed my own CNN with 10 layers of convolutions and no maxpoolings or any other connections. When I ran it on a dataset I got the following loss curve (blue) the other one is accuracy vs number of epochs. What do you think might be the problem here? And how to rectify it?

NOTE: My data-set of size 20 due to lack of computational resources, so the oscillation is probably looking more prominent or happening in the first place.


texts: CNN 1D for time series. Value Error. I did read all the possible answers in StackOverflow regardiing this problem

texts:
I been fighting with this problem for 2 weeks now. And I extensively research for solutions here and in other sites.
I have a dataset of 4 dimensions device_id, time and longitude and latitude.
After preprocessing it look like this i got this shape

data1.shape
(95751, 4)

data1cnn2 = data1.values.reshape(1,95751, 4)

data1cnn2.shape

(1, 95751, 4)

y.shape

(95751,)

Then this is the code for the CNN1d . using Tensor-flow

modelcnn = Sequential()
modelcnn.add(Conv1D(filters=64, kernel_size=2, activation='relu', input_shape=(95751, 4)))
modelcnn.add(MaxPooling1D(pool_size=2))
modelcnn.add(Flatten())
modelcnn.add(Dense(50, activation='relu'))
modelcnn.add(Dense(1))
modelcnn.compile(optimizer='adam', loss='mse')

then I fit the model

modelcnn.fit(data2cnn, y, epochs=1000, verbose=0)

and I get this mistake

ValueError: Input arrays should have the same number of samples as target arrays. Found 1 input samples and 95751 target samples.

modelcnn.summary()

Layer (type)                 Output Shape              Param #
=================================================================
conv1d_2 (Conv1D)            (None, 95750, 64)         576
_________________________________________________________________
max_pooling1d_2 (MaxPooling1 (None, 47875, 64)         0
_________________________________________________________________
flatten_2 (Flatten)          (None, 3064000)           0
_________________________________________________________________
dense_3 (Dense)              (None, 50)                153200050
_________________________________________________________________
dense_4 (Dense)              (None, 1)                 51
=================================================================
Total params: 153,200,677
Trainable params: 153,200,677
Non-trainable params: 0
_

I also try with different solution but got different type of mistakes

I did

data1cnn = data1.values.reshape(95751, 4,1)

data1cnn.shape
(95751, 4, 1)

but i got this other mistake

ValueError: Error when checking input: expected conv1d_2_input to have shape (95751, 4) but got array with shape (4, 1)

modelcnn.summary()
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_2 (Conv1D)            (None, 95750, 64)         576
_________________________________________________________________
max_pooling1d_2 (MaxPooling1 (None, 47875, 64)         0
_________________________________________________________________
flatten_2 (Flatten)          (None, 3064000)           0
_________________________________________________________________
dense_3 (Dense)              (None, 50)                153200050
_________________________________________________________________
dense_4 (Dense)              (None, 1)                 51
=================================================================
Total params: 153,200,677
Trainable params: 153,200,677
Non-trainable params: 0

I am completely stuck and I have read all the question and answers that are here . With no success


texts: Understanding softmax layer in Deep Neural Network

texts:
In a DNN, once the logits vector is produced say

$[y_0,y_1]$ where the number of neuron in the logits layer is 2, the condition holds where $y_0 >= 0$ and $y_1<= 1$.

This vector is then passed into the softmax layer where it squeezes the value to be within the range of 0 to 1. However, how do i prove that $y_0$ will always be $>=0$ and $y_1$ will always be $<=1$?

From the way i understand, both $y_0 + y_1 = 1$ hence $y_0 >= 0$ and $y_1 <=1$ but i find that it doesn't really prove on why $y_0 >= 0$ and $y_1 <=1$. Could anyone assist me on my understanding.


texts:
I'm going to scrape the HTML code from a large number of ULRs and store them on my computer, for machine learning purposes (basically, I'm going to use Python and PyTorch to train a neural network on this data). What is the best way to store the HTML code for all the web pages?
I want to be able to see which URLs I have already scraped, so that I don't have to scrape them again, and for each piece of HTML code (1 piece of HTML code = all extracted HTML code from one URL), I may also want to see which URL it came from (but maybe this turns out to be an unnecessary requirement). I also want to be able to see the timestamp when these pages were created so I can read them in chronological order (I will be able to extract timestamp when I download the web pages), and possibly other metadata. I imagine that the total file size of the HTML code can reach many GB, if not TB, and speed (both for reading and scraping) is a high priority.
In an ideal world, I would be able to just use the URLs as file names, have one file for each piece of HTML code, and store all files in one folder. But I'm not sure this is such a good idea, or even possible, for several reasons, for example:

URLs contain characters that are not allowed or at least not suitable in file names (e.g. '/'). It may be possible to hash each URL and let that be the name of the file, but then I don't know what URL the HTML code came from by looking at the file name.
I'm not sure the operating system likes having millions of files in one folder because of of the file I/O overhead. Maybe grouping a number of pieces of HTML code in each file would be a better, e.g.
I'm not sure whether storing each piece of HTML code in a separate file is optimal when it comes to file reading.
It's not obvious how to store the metadata for each piece of HTML code.


texts:
I already have a functioning $Q(\lambda)$ implementation for a single agent working on a dynamic pricing problem with the goal of maximizing revenue. The problem that I'm working with, however, involves several different products that are replacements for each other, so dynamically pricing them all with independent learners seems incorrect, because the price of one influences the reward of the other. The goal would be to dynamically price them all so as to maximize the sum of each individual revenue.

I've been doing some research to try to find something that applies reinforcement learning in this way, but many multi-agent implementations I have found focus more on competitive games than cooperative, or they assume incomplete knowledge of other agents (I would have complete knowledge of each agent in this scenario). Are there any well-researched/documented applications of cooperative learning in this way?


texts:
I'm learning how to implement and evaluate a Logistic Regression Model, for this I need to change the values of my array from strings to 0 & 1.
I have the following numpy ndarray as a result of a DataFrame.values call
['PAIDOFF', 'COLLECTION', 'COLLECTION', 'PAIDOFF', 'PAIDOFF', 'PAIDOFF', ...]
I would like to know how can I change the values like:
'PAIDOFF' to 0 and 'COLLECTION' to 1
Any help will be much appreciated


texts:
I've been assigned a task to extract features/attributes from product description.

Levi Strauss slim fit jeans
Big shopping bag in pink and gold

I need to be able to extract out attributes such as "Jeans" and "slim fit" or "shopping bag" and "pink" and "gold".
The product description listings are not just for clothes, they can basically be anything.

I am not sure how to approach this problem. I tried implementing a Named Entity Recognizer solution and also a POS implementations, The NER implementation fails to recognize any token and most of the tokens show up as NNP(Proper Nouns) in he POS solution, which doesn't help me out a lot. I need a way to be able to distinguish between the brand name and the features of the Product(like if it is a t-shirt, the color or design(round neck, v-neck) etc).

I did implement a KMean solution which did cluster like products together, but then again it is not the result I am looking for.

Just looking for someone to direct me in the correct direction.


texts: Convolution neural network with 11 million parameters unable to overfit on 100 image samples

texts:
I have been trying to do some sort of image enhancement on grayscale images. I have used both pixel wise loss and perceptual loss (perceptual loss uses classifier between 2 classes trained on the same dataset).

So the input to my network is an bad image and the output is the enhanced image where edges in an image are clearer.

I have written the code in keras

inputs = Input(shape=image_shape)

x = Conv2D(filters=ngf, kernel_size=(7, 7), padding='same')(inputs)
x = BatchNormalization()(x)
x = LeakyReLU(0.2)(x)

n_downsampling = 2
for i in range(n_downsampling):
    mult = 2**i
    x = Conv2D(filters=ngf*mult*2, kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

mult = 2**n_downsampling
for i in range(n_blocks_gen):
    x = res_block(x, ngf*mult, use_dropout=False)

for i in range(n_downsampling):
    mult = 2**(n_downsampling - i)
    x = Conv2DTranspose(filters=int(ngf * mult / 2), kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

#x = ReflectionPadding2D((3, 3))(x)
x = Conv2D(filters=output_nc, kernel_size=(5, 5), padding='same')(x)
x = Conv2D(filters=output_nc, kernel_size=(3, 3))(x)
x = Activation('sigmoid')(x)

outputs = Add()([x, inputs])
#outputs = Lambda(lambda z: K.clip(z, -1, 1))(x)
outputs = Lambda(lambda z: z/2)(outputs)

model = Model(inputs=inputs, outputs=outputs, name='Generator')

Function for res_block

 def res_block(input, filters, kernel_size=(3, 3), strides=(1, 1),
 use_dropout=False):
"""
Instanciate a Keras Resnet Block using sequential API.

:param input: Input tensor
:param filters: Number of filters to use
:param kernel_size: Shape of the kernel for the convolution
:param strides: Shape of the strides for the convolution
:param use_dropout: Boolean value to determine the use of dropout
:return: Keras Model
"""
x = Conv2D(filters=filters,
           kernel_size=kernel_size,
           strides=strides,
           padding='same')(input)
x = BatchNormalization()(x)
x = Activation('relu')(x)

if use_dropout:
    x = Dropout(0.5)(x)

x = Conv2D(filters=filters,
           kernel_size=kernel_size,
           strides=strides,
           padding='same')(x)
x = BatchNormalization()(x)

merged = Add()([input, x])
return merged

Any idea why this wouldn't overfit ? I have been looking for other loss functions but I couldn't find much. I personally tried to make pixel loss penalize the model on pixels where it should have completed a incomplete line in an image by using sobel operator on the enhanced image to get a mask which is multiplied by the difference between the input & target image hoping that the network would focus on completing incomplete edges or so but nothing changed.


texts: Training Neural network classifier using string inputs

texts:
My thesis topic is about building a (deep) neural-network classifier to classify the type of a place. I am given both labels and some inputs in string type. So for example the label "Supermarket" might have a feature like "Food".

How should I feed my string input features to the neural network?

In other words, is there any efficient way to substitute those string inputs with numerical values knowing that there are A LOT of them?


texts: Why aren't languages like C, C++ used for data analytics instead of R, Python?

texts:
I have started learning data science using R, however I have C++ as a subject this semester, and my project is to predict the outcome of a game using C++.
I have not come across many instances (close to none, I did find libraries like Shark though) of implementation in C++.

Is it to do with the fact that C++ isn't as simple to use when it comes to manipulating large amount of data?


texts: How to build a model which can predict the probability of an event based on a set of timeseries data?

texts:
I am trying to build a deep learning neural model using keras and tensorflow which can predict if a certain event will occur based on a set of timeseries data and some fixed data together . For example:
For a given set of entities say , Their price behavior is co-related due to rumor of some event that is going to happen in future.

1) A & B 's fixed data like type, group etc.
2) During a certain period of time - 1 Jan 2015 - 30 Mar 2015 , their price .

Data that I have is

INPUT : Name of entity, Type Of entity , Size , Country, Specific Attributes  and time series stock data from 1 Jan 2015 - 30 Mar 2015
OUTPUT : Y/N . Boolean output if event happend or not.

Now my question is how do I build this since I have some fixed data which doesn't changes over time and some time series data which changes over time.

Options that I thought of are
1) LSTM - But not sure if I should feed in fixed static data.
2) CNN - Not sure if it is the right approach ?

Please let me know what should be my approach to handle such a problem.


texts: Effect of missing annotations in the ground truths on object detectors

texts:
I wanted to know whether not annotating some categories in our images, and feeding them into our neural network, would have any adverse effect on our model in detecting the desired objects.
To better clarify what I mean, here's an example of what I am looking for in my question: Let's say we have a object detector network, which has only 2 categories; "cars" and "persons". We have 10,000 pictures, in which in every image, we have at least one instance of either categories.
After annotating all of the "cars" and all of the "persons" in the first 5,000 images, we feed them to our network, and we see we get an AP score of "1.0" for "cars", and an AP score of "80.0" for "persons". ==> Since our model is already doing a great job at detecting "persons" and to save us some time, we decide not to annotate any more "persons, in the remaining 5,000 images, and to only annotate "cars" instances.
==> So, here's again my same question; will this approach have any adverse effect on detecting the "persons" as well? after feeding the next 5,000 images into the network, will this make "persons" AP score to go up or down?


texts: Machine learning - 'train_test_split' function in scikit-learn: should I repeat it several times?

texts:
I am a beginner in machine learning, and I hope someone can help me.

In Python's 'scikit-learn' library, the function 'train_test_split' splits the dataset into training and test sets. This is done in a random way (possibly using a seed to obtain the same result in repeated executions). But, with a single random split, how much can we trust the result (classification accuracy) obtained through the 'fit' method? I mean... if we are "unlucky", we may obtain a very bad outcome (or the opposite, if we are "lucky"). Shouldn't we repeat the split/fit procedures multiple times (e.g., 100) and then average the classification accuracies obtained? (possibly after parameter tuning by means of cross validation).

I am asking this because previously I used the Python's libraries of Orange Data Mining, which include a method ('proportion_test') that splits the dataset into training and test sets and then evaluates it according to a specific classifier, repeating the operation a specified number of times (e.g., it performs 100 iterations of 70:30 test). My question is: should I manually do this also with the split/fit functions in scikit-learn? (e.g., 100 iterations using 100 different random seeds). Would results be better?

I am very confused about this...

I want to stress that I know about cross validation, leave one out, etc. But, if I understand it right, these techniques are used for model validation (i.e., model parameter tuning). My question is whether the final evaluation of the model should be based on repeated split/fit operations. For example, in the book 'Introduction to Machine Learning with Python' (by Andreas C. Müller and Sarah Guido, O'Reilly), the suggested operation pipeline is: (1) split the original dataset into training set and test set; (2) perform parameter tuning (i.e., best parameter selection) using cross validation on the training set; (3) re-train using the just found best parameters with the training set; (4) perform the final evaluation (calculating the classification accuracy) using the trained model and the (single) test set. My question is: is this enough? Or, once the classifier has been trained using the best parameters (step 3 above), should I repeat the split/fit procedures multiple times on the whole dataset (original training + test sets) to obtain more reliable results? I did this using the Orange library, but maybe it is not necessary (it is not done in the book quoted above).

THANK YOU VERY MUCH in advance for your help!


texts: Mean estimation for nested location data

texts:
I want to estimate the average income for a location. I have nested data in the following way: A block is inside a neighborhood, which is inside a zipcode, which is inside a district, which is inside a region, which is inside a state.

I want to estimate the average income at a block level, and the issue is that I don't have much data at that level. I have much more data at a state level, but it is not such a good approximation.

How would you deal with this problem? Are there any ways to incorporate the uncertainty of not having many data points at a block level? Are there any Bayesian frameworks that allow us to incorporate data of all levels? Is it possible that mixed models are able to do so?

If you explain any method, if you can provide a python package where that method is built, it'll be great!

Thanks!


texts: Text annotating process, quality vs quantity?

texts:
I have a question regarding annotating text data for classification.

Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.

Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).

Thoughts,

will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)


texts: (pre-trained) python package for semantic word similarity

texts:
I am searching for a python package that calculates the semantic similarity between words. I do not want to train a model (what most packages seem to offer) - the package should have been pre-trained on ideally thousands of natural language books and documents (e.g. on how often do words occur in close proximity to each other in the training material) and be simple to install/use. As for example in pseudo code below:

import XYZ

assessor = XYZ.loadPreTrainedModel("standard_text")
assessor.scoreWords("pilot", "airplane")  # returns 0.94 (I made up these numbers)
assessor.scoreWords("student", "university")  # returns 0.91
assessor.scoreWords("cat", "dog")  # returns 0.82
assessor.scoreWords("cat", "airplane")  # returns 0.13
assessor.scoreWords("student", "apple")  # returns 0.25
...


texts: Normalization around the nominal value while avoiding to get too big results

texts:
Lets say we measured a signal N times. Each measurement contains T time instance. For example for N=4 measurements and T=5 time instances the data matrix is as follows:

% N measurements for T time instance
Data =      [-1  0   1.5  1.8
             0   -3  1.9  -1.5
             1.9 1.1 0.9  1
             0   1.7 -1.3 0
             0.6 1.3 -1.8 0]

On the other hand, we know that the sum of the signal values over T time instance should be X(Nominal value).
Therefore, we can normalize each measurements at each time instance as follows:

Normalized_Data = (Data .- X/4)./abs(X/4)

The above normalization works fine as long as X is not around zero. Basically, when a Dij (The jth measurement of the signal for ith instance) is equal to its expected (basically X/ size(X)) the normalized value is 0, when its bellow/above the expected, it is negative/positive respectively.  However, when the given nominal value (X) is around zero, the Normalized_Data gets too large.
Is there any suggestion for a normalization method that avoids this issue?


texts: Weight training breakdown in machine learning

texts: I'm not sure if this exists. Is there such a situation where weights in gradient descent fail to work or break up? If so, how and when?


texts: How to train on extended data set correctly

texts:
I have trained my classifier on pictures with a mixture of several classes
on each picture, e.g. A-F. The classifier is able to (nearly) correctly segment those classes
on the images.

Now I got more data with pictures showing class G. To minimize my work, I only labeled
class G on the images and left the rest out (Invalid).

Two questions for my training arise:

If there are no examples of class G in my first dataset (because it could have been a subclass
of A-F), how do I train it correctly?
Suggested Solution: I can add my new data to the old one, but I have to tell the CNN when there is an image with class G, because I have to change the error measurement of my CNN somehow. But how?
If G was included in my first dataset (assume G is 'police car' and B is 'car', but
some images showed police car), how do I train in this case correctly?


texts: Incremental modelling of kmeans in pyspark

texts:
I have a large dataset and trained the model with kmeans for the first time. I  saved the model and pipeline used . Now again I started collecting data.  After sufficient data is collected using old model and pipeline I want to retrain the model on new collected data in Pyspark

Is it possible to use transfer learning in unsupervised learning algorithms (or clustering) like Kmeans in Pyspark? If so, how it can be done?


texts:
The term "Data Cleaning" is used to describe outlier checking, date parsing, missing value imputation to structuring datasets (organizing data values within a dataset) to facilitate analysis.

The latter is commonly referred to as "Data Tidying" but what about the former? I have seen outlier checking, date parsing and missing value imputation referred to as "soft processing", however I wonder if this is a commonly used term.

If I describe a component of data cleaning as "soft processing" is it reasonable to assume that knowledgeable people will know that I am referring to outlier checking, date parsing and missing value imputation?

In the field of Data Science, is there a commonly accepted phrase or term that umbrellas outlier checking, date parsing and missing value imputation.


texts: Why L2 norm in AdaGrad update equation not L1?

texts:
The update equation of AdaGrad is as follows:

I understand that sparse features have small updates and this is a problem. I understand that the idea of AdaGrad is to make the update speed (learning rate) of a parameter is inversely proportional to the update history of that parameter ($\eta$ / sum of previous updates). This is independent of the other parameters. This makes the small number of updates done to sparse features have a higher learning rate than dense ones.
My question is about how to implement this in the above equation. Why are we summing the squares of update history and getting its square root? I understand that we need to get rid of the negative sign. So, why not directly summing the absolute values?


texts: NLP techniques to label unlabeled data in a dataset

texts:
So I have a .xls file with negative and neutral reviews of a medicine. However, this dataset does not have labels. I converted this .xls into a dataframe and I am using the Spacy Lib.

I have to use NLP techniques to label the data. Can you guys help me out on how to use NLP techniques to label this dataset as a neutral review or a negative review. So far, I did data cleansing (remove stop words, punctuation, etc.)

What are the next steps?


texts:
I trying to get some experience by exploring this Kaggle dataset.
It consists of 1500 pictures of boats classified in 9 categories. The data is as follows :
#x_train consists of 1159 images, with 80% of images from each category

x_train.shape = (1159,200,200,3)

y_train contains the number-label for each boat

y_train.shape = (1159,)

I have tried many variations of models like the following one but without any success.
model = Sequential()

model.add( Conv2D(32, (3,3),  input_shape = x_train.shape[1:] , activation='relu') )
model.add(MaxPooling2D(pool_size=(3,3)))

model.add(Flatten())
model.add(Dense(4, activation='relu'))

model.add(Dense(2, activation='softmax'))

model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['accuracy'])

h = model.fit(x_train, y_train, epochs=50,
              batch_size = 64,
              validation_data = (x_val, y_val) )

Could you give me any advice on how to get a model with decent test_accuracy?


texts: Is there a way to visualise what/how my image classifier has identified an image?

texts:
Using tensorflow.
If I have trained my image classifier many images of a close up of melanomas and other skin cancers.

Then ask it to classify using the trained model a photo that has another element for example melanoma on a hand (with other elements that look like melanoma in the background example a hypothetical tablecloth that has melanoma patterns).

Is there a way to visualize what the image classifier has identified as melanoma ? Outputting a circled/highlighted the found melanomas in the photo?


texts: Creating a valid dataset for obtaining results

texts:
I have created a domain-specific dataset, lets say it is relating to python programming topic posts. I have taken data from various places specific to this topic to create positive examples in my dataset. For example, python related subreddits, stack exchange posts tagged with python, twitter posts hashtagged with python or python specific sites.
The data points taken from these places are considered positive data points and then I have retrieved data points from the same sources but relating to general topics, searched if they contain the word python in them and if they do discard them to create the negative examples in my dataset.
I have been told that I can use the training set from the dataset as is, but that I need to manually annotate the test set for the results to be valid, otherwise they would be biased. Is this correct? How would they be biased? To be clear the test set contains different entries to the training set.
There are close to 200,000 entries in the test set which makes manual annotation difficult. I have seen similar methods been used in papers I have previously read without mention of manual annotation. Is this technique valid or do I have to take some extra steps to ensure the validity of the test sets?


texts:
I have multi-touch attribution data like:

    medium  conversion
1   organic 0
2   (none) > referral > referral > (none) > (none) > referral   0,0,0,0,0,0
3   (none)  0
4   organic > referral > referral   0,1,0
5   referral > referral > referral > referral   0,0,1,0
6   organic > referral > referral > (none) > referral   0,1,0,1,0

I'd like to remove rows with no conversions (like rows 1, 2 and 3) and I tried grepl but couldn't make it work.

For the remaining rows how to split rows so it ends with a conversion, e.g. row 4 will be organic > referral 0,1
and row 6 will split into organic > referral 0,1 and organic > referral > referral > (none) 0,1,0,1


texts: Variance of prediction doesn't match input data with Keras model

texts:
I'm using Keras to do a regression on inputs. I've tried a lot of different models, and a lot of them converge around the same place. My problem is with the distributions of the results. In order to validate the model, I compared the distribution of the input data (test and validation) with predictions on the same data. My data is distributed normally, so when I'd see predictions with a bimodal or uniform distribution, even when the loss looked good, I knew something wasn't working with the model.
I'm able to get a normal distribution (like my ground truth outputs). My problem is that the variance of the predictions is always smaller. My only guess is the loss function, but MeanAbsoluteError seemed like a better choice than mean squared for not distorting the output.
Here's the model:
model = keras.Sequential([
    layers.Dense(32, activation="selu", kernel_initializer="lecun_normal", input_shape=(len(np_inputs[0]),)),
    layers.Dropout(0.2),
    layers.Dense(32, activation="selu", kernel_initializer="lecun_normal"),
    layers.Dropout(0.2),
    layers.BatchNormalization(),
    layers.Dense(1, activation="tanh"),
])

model.compile(
    loss=keras.losses.MeanAbsoluteError(),
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
)

And histograms of the outputs. The buckets are the same size.
Training+validation outputs

Predictions based on training+validation


texts: Can bagging ensemble consist of heterogeneous base models?

texts:
Bagging or bootstrap aggregation seems to make sense for time series forecasting using an ensemble because bagging randomizes subsets of the data with replacement. However, I've only seen bagging used for homogeneous base learners when constructing ensembles.

Stacking is another ensemble technique that uses heterogeneous base learners, but stacking employs cross-validation, which I don't view as being appropriate for economic time series forecasting, even if time series split cross-validation that retains the ordering of observations is used.

As you can see, for time series forecasting, i like the random re-sampling of bagging, and the heterogeneity of base algorithms in stacking. How to combine these qualities in ensemble learning?

Can heterogeneous algorithms be used in a bagging ensemble? Or can bootstrap re-sampling be used within stacking?


texts: How can I model the autocorrelation of objective variables under the situation where we can't observe any actual objective variable in the test phase

texts:
I'm trying to model the relationship between the declared value from a subject and stimulus.
For example, modeling a relationship between the subject's happiness and strength of stimulus so that we can predict the subject's sadness from stimuli.
(The Happiness are five scale ratings, stimuli are continuous value)
Emotions like happiness are obviously autocorrelated and  I think modeling these autocorrelations might help the model make a better prediction. However, we can only observe happiness (actual value)  in the training phase and not in the evaluation phase.
So we can't use any actual objective variables in the past to predict the current value. And unfortunately, the stimulus is not enough good estimator of objective variables so far.
What can I do to model the autocorrelation in situations like above?
In the training phase, we have:
y_1, y_2, ..., y_T
X_1, X_2, ...., X_T
where
corr(y_k, y_{k+1}) >= 0.7
In the test phase, we only have:
X_1, X_2, ...., X_N
and need to predict
y_1, y_2, ..., y_N


texts:
I am self learning data analysis using (mostly) R. I'm currently on K-nearest neighbour topic, and I went through the process of normalizing- splitting the dataset to train/test and fitting the model and evaluating it, and it all went well. But, there is one fundamental thing I just don't understand: what is the point in classifying a known data? How do I predict the value of NEW data? The outcome of knn() is a factor with labels, but after using it for evaluating my model (namely, the value of K I have chosen), what should come next? For example, in regression model I can use predict() to predict new data, but I do I predict (as opposed to just classify) using knn?
Thank!


texts: Orange data mining: Balancing data set using imblearn code

texts:
I am using an unbalanced dataset. I wanted to oversample my dataset using a python script (Scripting code for class imbalance in Biolabs Orange). However, it still gives me an error
"No module named imblearn".

How can I solve this?
Kind regards,
Emma


texts: Colouring points based on cluster on matplotlib

texts:
I have a set of points where I performed a KMeans classification. How make a plot where the color of the point is based on the cluster they belong?

EDIT: for clarification, having the set of points, I want to use the values of the array generated from KMeans.predict() ( from sklearn) to choose the color of each point.


texts:
I have trained a conditional inference decision tree in R using library party with function ctree and saved the model in an .Rda file.

I need to migrate this model from R to Java so that I can utilise the tree to make predictions in a Java environment. Can someone please point me in the right direction on how I can do this?


texts: two different attention methods for seq2seq

texts:
I see two different ways of applying attention in seq2seq:

(a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram is from here.

(b) the context vector fed into the decoder input as shown the diagram below. The diagram is from here.

What are the pros and the cons of the two different approaches? Is there any paper comparing the two?


texts: Difference between Validation data and Testing data?

texts: I am bit confused about validating data. What is this data mainly for?? Like I am seeing some tutorial and they have some training[I know it] images , they they have some validation images[donot know] and some testing[I know] images? So what is validating images mainly for??


texts:
I am trying to do a numerical time series prediction using LSTM but it seems that the LSTM just mocking previous data(Even when I use many-to-many prediction with different lengths) and when I plot the results that I get from LSTM, I a have very similar graph to real graph just with a time-delay!

The green graph is real and yellow graph is LSTM prediction(mocking).


texts: Are there any methods to detect whole multivariate time-series as anomalous from a set of multivariate time-series?

texts: Consider a scenario with Dataset D as {T1, T2, ..., Tn} and Ti is a multivariate time-series of length mi as {X1, X2, ..., Xmi}. Here each record of the time-series Xi is a vector of attribute values {Xi1, Xi2, ..., Xia} where Xij is the value of jth attribute of ith record of a time series. Given this scenario is there a method to detect the anomalous time-series(or a set of anomalous time series, i.e the T's which are anomalous) from the set D?


texts: How fit_transform, transform and TfidfVectorizer works

texts:
I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' :

When I have to vectorize my data I do not really understand what is the purpose of fit_transform and WHY 'dirty_idf_matrix' has ONLY transform argument with SAME vectorizer than 'clean_idf_matrix' which has saved the value with fit if I understood well.

Col_clean = 'fruits_normalized'
Col_dirty = 'fruits'

#read table
data_dirty={f'{Col_dirty}':['I am an apple', 'You are an apple', 'Aple', 'Appls', 'Apples']}
data_clean= {f'{Col_clean}':['apple', 'pear', 'banana', 'apricot', 'pineapple']}

df_clean = pd.DataFrame(data_clean)
df_dirty = pd.DataFrame(data_dirty)

Name_clean = df_clean[f'{Col_clean}'].unique()
Name_dirty= df_dirty[f'{Col_dirty}'].unique()

vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)
clean_idf_matrix = vectorizer.fit_transform(Name_clean)
dirty_idf_matrix = vectorizer.transform(Name_dirty)

thank you for your help !


texts: WHY or WHEN to convert numeric data to a categorical data?

texts:
This is an open ended WHY TO or WHEN TO question rather than a question on HOW TO encode numeric to categorical data.

I am currently working on  Telco Customer Churn dataset from kaggle.

This is classification problem to predict if a customer will Churn or Not.

It has total 7043 rows and 21 columns, a mix of categorical and numerial columns. One of the column is Tenure which has data distributed in the range

> df.tenure.unique()

array([ 1, 34,  2, 45,  8, 22, 10, 28, 62, 13, 16, 58, 49, 25, 69, 52, 71,
       21, 12, 30, 47, 72, 17, 27,  5, 46, 11, 70, 63, 43, 15, 60, 18, 66,
        9,  3, 31, 50, 64, 56,  7, 42, 35, 48, 29, 65, 38, 68, 32, 55, 37,
       36, 41,  6,  4, 33, 67, 23, 57, 61, 14, 20, 53, 40, 59, 24, 44, 19,
       54, 51, 26, 39])

In one of the public kernel I saw the guy changing it into a categorical column.

> pd.cut(df['tenure'], bins=[0,12,24,48, 60, df['tenure'].max()], labels=['Tenure_0-12', 'Tenure_12-24', 'Tenure_24-48','Tenure_48-60','Tenure_gt_60'])
0        Tenure_0-12
1       Tenure_24-48
2        Tenure_0-12
3       Tenure_24-48
4        Tenure_0-12
            ...
7027    Tenure_12-24
7028    Tenure_gt_60
7029     Tenure_0-12
7030     Tenure_0-12
7031    Tenure_gt_60
Name: tenure, Length: 7032, dtype: category

It is a common practice to group/bin some numeric columns (usually with date/time columns like Age, Seasonal). I am not quite sure of the benefits of doing so across various Traditional ML algorithms.

Some Algorithms which are sensitive to dimentionality may perform bad, as the number of dimensions increases by One-Hot encoding of the newly created Categorical column.

Rationally, with the columns grouped into bins it will be very difficult to cluster the target for algo like K-NearestNeighbour or SVM to find a plane.

Please! Lets discuss on the impact of this technique on different ML Algorithms (Not just limited to tradiotnal or classification algorithms).


texts:
I am a newbie to this stuff so I am sorry if my question is stupid~

I need help understanding what data leakage between X_train and X_test is and when exactly it happens.
I am currently working on a dataset where I am using KNN imputer to fill in the missing values. I need to scale the data for knn imputation and I am doing the train-test-split and applying machine learning models after the imputation process. I read that data leakage might happen during scaling so we should scale after splitting, fit_transform the train set, and only transform the test set.
I am unsure about how that would work in my case since I am scaling the data to be able to impute for missing values and I actually reach the train-test-split stage later. Should I be worried about data leakage the way I am doing things?

Here is the code:

Although here I am doing the splitting + applying DT algorithm right after imputation, I have other steps like feature selection left so I won't reach the train-test-split and decision tree part until much later.


texts: How is it possible for RNN to do sentiment analysis?

texts:
I'm wondering how RNN can be used when doing sentiment analysis.
It seems that the characteristic of RNN is to remember what appeared in the past and determine the value of the present (future), but I think this RNN's characteristic is useless when analyzing sentiment.
I am curious to see how it is possible to do emotional analysis by remembering the words that have appeared so far.
I think the way to train vectors that show how many times specific words appear is the only way to analyze sentiment.
If some RNN model train a sentence "good movie" and label "positive", can it predict a new sentence "movie good" as a "positive"?
Also, if a normal model(just fully connected deep neural network) train a sentence "good movie" and label "positive", is it hard for this model to predict a new sentence "movie good" as a "positive"?


texts: Which is better: GPT or RelGAN for text generation?

texts:
Based on my understanding, gpt or gpt-2 are using language model loss to train and generate text, which do not contains GAN.

So which is better: GPT vs RelGAN/LeakGAN/SeqGAN/TextGAN

I am so confused about this question.


texts: Using Majority Class to Predict Minority Class

texts: Suppose I want to train a binary model in order to predict the probability of who will buy a personal loan and in the dataset only 5 percent of the examples are people who marked as bought a personal loan. So, in this scenario maybe I can leverage downsampling or upsampling to balance the dataset, but if my dataset isn't big enough there may left very few examples or may be upsampling isn't appropriate. Then suppose I decided to use whole dataset, I partitioned it to the training and test sets in order to predict the probability of who won't buy a personal loan. Considering it's a binary model does it make sense to subtract this model's output probabilities from 1 and predicting who will buy a personal loan by using this result?


texts:
I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event.

The preprocessing steps i took are:
(1) turning to OneHotEncoding using pandas:

vector = pd.get_dummies(sr)

This part takes about 15 seconds.

(2) Using a sliding window of 10, i create samples and labels as follows;
i iterate the vector from (1), i take Xt as the label and Xt:t-10 as the data.

Code:

X = np.zeros((len(samples), window_size, voc_len), dtype=np.bool)
y = np.zeros((len(samples), voc_len), dtype=np.bool)

if IN_COLAB:
  loading_bar = tqdm.tqdm_notebook(enumerate(samples),desc='Build dataset',total=len(samples))
else:
  loading_bar = tqdm.tqdm(enumerate(samples),desc='Build dataset',total=len(samples))

for numpy_index, pandas_idx in loading_bar:
  x_idx = (pandas_idx, pandas_idx + window_size)
  y_idx = pandas_idx + window_size
  Xt = vector.iloc[x_idx[0]:x_idx[1]]
  yt = vector.iloc[y_idx]
  X[numpy_index] = Xt
  y[numpy_index] = yt

display(yt)

The problem is that the seconds part is VERY slow (25 minutes).
Is there a better way to do this?
a built in function of some sort?

Thx


texts: How to give a higher importance to certain features in a (k-means) clustering model?

texts:
I am clustering data with numeric and categorical variables. To process the categorical variables for the cluster model, I create dummy variables. However, I feel like this results in a higher importance for these dummy variables because multiple dummy variables represent one categorical variable.

For example, I have a categorical variable Airport that will result in multiple dummy variables: LAX, JFK, MIA and BOS. Now suppose I also have a numeric Temperature variable. I also scale all variables to be between 0 and 1. Now my Airport variable seems to be 4 times more important than the Temperature variable, and the clusters will be mostly based on the Airport variable.

My problem is that I want all variables to have the same importance. Is there a way to do this? I was thinking of scaling the variables in a different way but I don't know how to scale them in order to give them the same importance.


texts: Can convolutional network learn structural properties of one feature w.r.t to other?

texts:
I'm going through the literature on pose-estimation ( DeeperCut, OpenPose, MultiPersonPosetrack).
I'm interested in knowing whether these networks/ generally a CNN can learn properties (geometrical) like opposites, same side, diagonal etc of one feature w.r.t the other features or is it based on closest neighborhood of that feature. For example, say a CNN trained on human joints be able to predict more accurately that left and right shoulders always on the same plane/straight line, left ear is closer to left eye that it is closer to the right eye.


texts: Use both differencing and normalization in time series modeling to make it stationary?

texts: I am working on a time series dataset. Should we use both differencing and normalizing or either of the ones to make it stationary?


texts: Wrong Graph Plot using K-Means in Python

texts:
This is my first time implementing a Machine Learning Algorithm in Python. I tried implementing K-Means using Python and Sklearn for this dataset.

from sklearn.cluster import KMeans
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

# Importing the dataset
data = pd.read_csv('dataset.csv')
print("Input Data and Shape")
print(data.shape)
data.head()

# Getting the values and plotting it
f1 = data['Area'].values
f2 = data['perimeter'].values
f3 = data['Compactness'].values
f4 = data['length_kernel'].values
f5 = data['width_kernel'].values
f6 = data['asymmetry'].values
f7 = data['length_kernel_groove'].values

X = np.array(list(zip(f1,f2,f3,f4,f5,f6,f7)))
# Number of clusters
kmeans = KMeans(n_clusters=7)
kmeans = kmeans.fit(X)
# Getting the cluster labels
labels = kmeans.predict(X)
# Centroid values
centroids = kmeans.cluster_centers_

plt.scatter(X[:,0], X[:,1],cmap='rainbow')
plt.scatter(centroids[:,0], centroids[:1], color="black", marker='*')
plt.show()

The graph doesn't seem to plot the data correctly. How can I debug this issue?


texts: Pandas v. SFrame in learning data science

texts:
I'm taking a machine learning course that introduces the related assignments as such that a student may use either Pandas or SFrame in solving them. As a beginner, it's hard for me to assess which approach would be more beneficial for me in the long run. Hence the question; should I prefer Pandas or SFrame for learning purposes?

Initially I might've preferred Pandas to make sure I learn the "proper foundations" first, so to speak. However, after reading about this online, SFrame appears to be very popular after having been made open source. Can it be considered a Pandas replacement from a (future) data scientist's point of view?

Thanks a bunch! Protips always welcome in comments ;)


texts: How to implement an LSTM RNN with multiple input features

texts:
EDIT: Now I didn't convert to list.

I am training LSTM for multiple time-series in an array which has a structure: 450x801. There are 450 time series with each of 801 timesteps / time series. The labels are classes with assigned integer from 1 to 6, so the dimension of the label is 450x1. This is my implmentation:

This is my code:

def readData():
labels = pd.read_csv('label.csv', header = None)
labels = labels.values
data = pd.read_csv('data.csv', header = None)
return data, labels

data, labels = readData()

data_train, data_test, labels_train, labels_test = train_test_split(data, labels)

model = Sequential()
model.add(LSTM(units=32, input_shape = (450,801,1)))
model.add(Dense(6, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Now I got this error:

Error when checking input: expected lstm_26_input to have 3 dimensions, but got array with shape (450, 801)

Any idea how to solve it?

For instance I can try to reshape:

data_train = np.reshape(data_train, (data_train.shape[0], 1, data_train.shape[1]))
data_test = np.reshape(data_test, (data_test.shape[0], 1, data_test.shape[1]))

And now the error is:

raise ValueError('Must pass 2-d input')
ValueError: Must pass 2-d input


texts: How to deal with class imbalance in a neural network?

texts:
Suppose we have a game and its action space contains two possible actions: A and B.

We have a labelled dataset of state-action pairs but 95% of actions are A and only 5% are B

If we train a neural network, it will always output A as it will choose the most probable class to decrease its loss.

Are there ways to solve this problem?


texts:
My goal is to predict the most appropriate answer from an utterance, in a group of 21 potential answers. (I'm not sure the "question" is called utterance though. )

Example:

Utterance: How are you today?
Answers: Answer1, 2, ..., 21.

I have a training file with this format:

Utterance:
Answers: Good answer, wrong answer1, wrong answer2,..., wrong answer20.

My problem

For the first time, we have to make a prediction from a group of possible answers, and, thus, this is a MCQ form.

Any ideas how I could start the problem?

What I've done

For the moment, the only thing I did was to choose the answers from the 21 possible answers which had the highest cosine similarity with the utterance. (So, unsupervised). It's not that bad (24% against 1/21 at random), but I'm sure there are ways to make something really better.

What I don't want to do at first

Use a generative model which predicts a full sentence. I want to choose the best candidate amongs the 21 answers, and use the training file which can allow us to do supervised learning.


texts: Identifying documents similar to specific clusters

texts: Through performing clustering on a set of 1,000,000 text documents, I have identified 100 clusters. I am particularly interested in, say, 10 of the clusters. Imagine, I now have an additional set of 100,000 documents (not part of the initial 1 mil documents). I wonder if there is a way to efficiently check whether each one of the 100K new documents belongs to one of the 10 clusters.


texts: generate a report in R where the value in each Column is to be derived from different tables in sybase databasea

texts:
I work in a stress testing team. my boss has asked me to generate an interactive report every day using different tables in a Sybase database. I know the table structure and sql queries, but can't get my head around implementing in Rstudio using shiny or markdown.

Please advise.
thanks,


texts: Dataset suggestion: video dataset with accurate depth and pose info

texts:
Please suggest some datasets (can be synthetic, like rendered in blender or so) which have video frames (30fps preferred) with accurate ground truth depth maps and pose (rotation and translation for every frame).
PS: I couldn't figure out if this is the correct site to ask this question. Please let me know if it isn't and where I can ask this question.


texts: How is FP-Tree used in FP-Growth maintained for large dataset

texts:
I was studying about Association Rule Learning and got to know about FP-Growth algorithm. I am wondering that how is FP-Tree maintained for a really large dataset, for example, that of Amazon and eBay.

Is the whole dataset stored in just one FP-Tree?
If not, then how are multiple FP-Tree generated and stored? Like, is the dataset divided or something?


texts: Top n products as a kind of recommendation system

texts:
I'm looking for a paper, book or something similar that describes only the top products as a kind of recommendation in a recommender system. The top products can be determined with a simple counter. Simply count which products have had the most interactions.
Does anyone know of a good text describing this type of recommendation system?


texts: Classifies the place of birth that belongs to the person at NER

texts:
I want to classify the place of birth and date of birth for each person detected by the NER results. For example, I have a sentence like this:

This paper represents the fact that Jaden Smith was born on July 8,
1998 in California, who is the son of Will Smith who was born in
Pennsylvania on September 25, 1968.

and

Lily-Rose Depp is the daughter of Johnny Depp who was born on May 27, 1999
in France, while his father was born on June 9, 1963 in Kentucky.

then I generate a NER like this: (I take the PERSON, DATE, and GPE only)

('Jaden', 'NNP', 'B-PERSON')  ('Smith', 'NNP', 'I-PERSON')  ('July',
'NNP', 'B-DATE')  ('8', 'CD', 'I-DATE')  (',', ',', 'I-DATE')
('1998', 'CD', 'I-DATE')  ('California', 'NNP', 'B-GPE')  ('Will',
'NNP', 'B-PERSON')  ('Smith', 'NNP', 'I-PERSON')  ('Pennsylvania',
'NNP', 'B-GPE')  ('September', 'NNP', 'B-DATE')  ('25', 'CD',
'I-DATE')  (',', ',', 'I-DATE')  ('1968', 'CD', 'I-DATE')
('Lily-Rose', 'JJ', 'B-PERSON')  ('Depp', 'NNP', 'I-PERSON')
('Johnny', 'NNP', 'B-PERSON')  ('Depp', 'NNP', 'I-PERSON')  ('May',
'NNP', 'B-DATE')  ('27', 'CD', 'I-DATE')  (',', ',', 'I-DATE')
('1999', 'CD', 'I-DATE')  ('France', 'NNP', 'B-GPE')  ('June', 'NNP',
'B-DATE')  ('9', 'CD', 'I-DATE')  (',', ',', 'I-DATE')  ('1963', 'CD',
'I-DATE')  ('Kentucky', 'NNP', 'B-GPE')

Now, the problem is how to classify the birth date and place of birth for each person? for example, to produce data like the following:
{'person':'Jaden Smith', 'date':'July 8, 1998', 'gpe':'California'}
{'person':'Will Smith', 'date':'September 25, 1968', 'gpe':'Pennsylvania'}
{'person':'Lily-Rose Depp', 'date':'May 27, 1999', 'gpe':'France'}
{'person':'Johnny Depp', 'date':'June 9, 1963', 'gpe':'Kentucky'}

should this use deep learning? or is there any other method / approach in NLP to deal with this?
Is it possible to use postag and chunking as a feature to generate grouping (as label column / class column)?
Thank you!


texts: pandas: how to change the specific column as index and change index into various columns

texts:
Hi I'm new to data science. Learning data science from course-era.
I'm having pandas data frame as follows,

   time   value

A   9      5
A   8      4
A   7      3
B   9      3
B   8      2
B   7      1
C   9      3
C   8      2
C   7      1

I want to convert this as ,

       A   B  C

9      5   3  3
8      4   2  2
7      3   1  1

As I start to write query for this, it is getting complicated. Is there any easy way to do this? Thanks for the help.


texts: Find which type of log using machine learning?

texts:
I am the beginner of Machine learning.
The process is:
I have different log files (System log, MSSQL Server log, Linux log, MySQL Log, FTP log, IIS log).If any input is given, I will find out which type of log using machine learning technique. Each log has a different format. Some logs don't have structure format(Linux, MySQL log, FTP log).
   In my analysis, these are all implemented using KNN algorithm(machine learning).
  But I don't know how to implement this? please give any suggestion about this.

The log type format:

Linux-syslog:

Jan  5 08:39:01 iei-Virtual-Machine CRON[48622]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))

MySQL-error log:

2018-01-05 10:55:20 18856 [Warning] Unsafe statement is written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ between master and the slave. Statement: CALL SubmitGetChangeDetectionInfo(@_SubmitGetChangeDetectionInfo_0

Event log:

IE038,System log,Error,20/11/2017 12:47:51 PM,TerminalServices-Printers,1111,None,Driver HP Deskjet 3520 series required for printer IEC057(Mahendran) Printer is unknown. Contact the administrator to install the driver before you log in again.

FTP log:

2018-01-04 00:00:01 162.254.209.219 INFOEVOL\EC 192.168.0.13 63346
DataChannelOpened - - 0 0 c1df3130-60e6-4678-9dcd-39177cc60d06 -IISAppslog:
2017-11-06 03:25:16 192.168.0.13 GET /IEIAppsLogin.aspx
param=OwjgKJLT+ikfpFnxYbvZS/QWXTFP4GEXmT+qM7TeXTMqi5D7DKexzYjZc3aJNB0x 90 -
182.73.50.19 Mozilla/5.0+(Windows+NT+6.3;+Win64;+x64)+AppleWebKit/537.36+
(KHTML,+like+Gecko)+Chrome/61.0.3163.100+Safari/537.36
http://182.73.50.19/AppsLogin.aspx 302 0 0 234

MSSQL Server:

01/10/2018 07:07:07,Logon,Unknown,Login failed for user 'ms SQL'. Reason: Could not find a login matching the name provided. [CLIENT: 49.65.2.226]


texts: Pre-trained CNN model makes Poor Predictions on Test Images Dataset

texts:
I have tried using several a pretrained models (MobileNet) for multiclass predictions. There are 42 classes and the distributions of the images are even across the 42 classes.
This is my code:
base_model=MobileNet(weights='imagenet',include_top=False,input_shape = (224,224,3)) #imports the mobilenet model and discards the last 1000 neuron layer.
x=base_model.output
x=GlobalAveragePooling2D()(x)
x=Dense(512,activation='relu')(x) #we add dense layers so that the model can learn more complex functions and classify for better results.
preds=Dense(42,activation='softmax')(x) #final layer with softmax activation
model=Model(inputs=base_model.input,outputs=preds)
for layer in base_model.layers[:20]:
    layer.trainable=False
for layer in base_model.layers[20:]:
    layer.trainable=True

I have freezed and unfreezed some of the trainable layers.
I let it run for 5 epochs:
Epoch 1/5
1318/1318 [==============================] - 3604s 3s/step - loss: 1.5493 - accuracy: 0.5796 - val_loss: 1.7180 - val_accuracy: 0.5361
Epoch 2/5
1318/1318 [==============================] - 3272s 2s/step - loss: 1.2174 - accuracy: 0.6641 - val_loss: 1.7562 - val_accuracy: 0.5372
Epoch 3/5
1318/1318 [==============================] - 3233s 2s/step - loss: 1.0853 - accuracy: 0.6981 - val_loss: 1.2993 - val_accuracy: 0.6498
Epoch 4/5
1318/1318 [==============================] - 3223s 2s/step - loss: 0.9918 - accuracy: 0.7224 - val_loss: 1.3455 - val_accuracy: 0.6382
Epoch 5/5
1318/1318 [==============================] - 3310s 3s/step - loss: 0.9153 - accuracy: 0.7413 - val_loss: 1.2375 - val_accuracy: 0.6660

The accuracy I got is relatively good. I have tried with other pretrained models like Xception too and the accuracies were pretty good. However when I use my model to predict on the test data, and upload my submission to Kaggle, my predictions are very bad.
I am wondering if I have imported the test images and made predictions correctly?
Here is the code:
import cv2
pred_images = []
filename= []

for image_file in os.listdir('./test/test'): #Extracting the file name of the image from Class Label folder
    filename.append(image_file)
    image = cv2.imread('./test/test'+r'/'+image_file) #Reading the image (OpenCV)
    image = cv2.resize(image,(224,224)) #Resize the image, Some images are different sizes. (Resizing is very Important)
    pred_images.append(image)

category = []
for i in range(len(pred_images)):
    pred_image = np.array([pred_images[i]])
    pred = model.predict(pred_image)
    cat = np.argmax(pred, axis = 1)
    category.extend(cat)

I get a data frame looking like this, with a lot from class 38 and 20. Is there something wrong?
final = pd.DataFrame(list(zip(filename, category)), columns = ['filename', 'category'])
final.head()

    filename                               category
0   c94de2fa9b06d67848f648e33a43475c.jpg    38
1   bbb7a2da148488bb878727556aa5914c.jpg    38
2   606256bdf3636d280bfdc3def33a57e7.jpg    38
3   7f741619b952876e7c7c419a0de1ed60.jpg    20
4   05c7592b31ceb8e14d4faa30fa21794c.jpg    20


texts: Decision Tree generating leaves for only one case

texts:
I had asked a question regarding predictive analysis for marketing earlier. Prediction model for marketing to prospective customers (using pandas) Still have some doubts about it, but I have a doubt regarding the decision tree that I generated for the marketing data.
My aim is to predict if a lead will be won or lost, depending on how they were made aware of the product etc. I have a bool variable "Won", 0-sale had failed, 1-sale was made. Using a decision tree, I was able to generate a model, however, there are no leaves for cases that lead to Not Won.
Is this normal? Ive seen examples of the iris data set where all 3 features were represented in the tree, adn, hence am wondering if the approach Ive taken is correct.
In the dataset of 38000, there are roughly 1700 who have Won=1.

Im using pandas and my parameters for DecisionTreeClassifier are:
min_samples_split=2
min_samples_leaf=1
max_depth=3
I got then using grid search for parameter optimisation. I got a mean val of 0.95. If I use a bigger depth, the tree becomes too big to analyse.
Thanks
EDIT
Since Mark said to post some code, I am doing so
from sklearn.tree import DecisionTreeClassifier,export_graphviz
dt=DecisionTreeClassifier(min_samples_split=2,min_samples_leaf=1,max_depth=3)
dt.fit(x,y)
features=x.columns
#print(features)
with open("dt.dot", 'w') as f:
    export_graphviz(dt, out_file=f,feature_names=features,class_names=["Won","Lost"])
command = ["dot", "-Tpng", "dt.dot", "-o", "hello.png"]
try:
    subprocess.check_call(command)
except:
    exit("Could not run dot, ie graphviz, to produce visualization")

from sklearn.cross_validation import cross_val_score
scores=cross_val_score(dt,x,y,cv=10)
print(scores.mean())

This is the main training code, all previous lines are just munging
The cross validation score come to 0.95

Here is the csv snapshot
I use all values from "Won" onwards and am training for ""Won"
There was a single column X, which had many categorical values (20), "Won" being one of them.
Known' 'Recycled' 'Engaged' 'Prospect/MQL' 'Intern Transfer' 'MQL'
 'Working' 'Opportunity' nan 'Current Customer' 'Vendor' 'Disqualified'
 'Converted' 'SAL' 'SQL' 'engaged' 'working' 'Won' 'Web Registration'

'Inactive
However, they were all uniformly distributed ie. from 37000, all had almost the same number of observations. I used get_dummies to transform to numerical values, and dropped all the columns except "Won"
All the rest of the values value for things like designation, opp(money value scaled from 1-3) and other categories, which are all boolean


texts: A/B Testing (Binomial Distribution vs Random Distribution)

texts:
When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant click through rate.
Example:
Two Ads,
-> Ad one has 1000 impressions and 20 clicks, CTR is 2%;
-> Ad two has 900 impressions and 30 clicks, CTR is 3.3%.
Test whether there is difference between Click Through Rate (CTR) between Ad one and two.
t test for two population:
Similar to the first question, we know sample mean of each experiment is the CTR we need compare. According to CTR, sample mean is limiting normal distributed. Then we have
X¯1∼N(p1,p1(1−p1)/n1)
X¯2∼N(p2,p2(1−p2)/n2)
T-test:

If the click through rate is not constant and is in fact randomly distributed (e.g. normal distributed), how does this affect the standard deviation of the sample mean CTR and consequently the validity of the test result.
My concern is that if use the above distribution and the actual standard deviation is higher we will end up accepting the alternative hypothesis incorrectly. Anecdotally, when we have run A/A tests (i.e. the variants are the same), we appear to often accept the alternative hypothesis more often than you would expect to and I want to find out whether this could be due to the assumptions around the click through rate being constant.


texts: How to load and run feature selection on a dataset with 5,000 samples and 500,000 features?

texts:
I have a dataset with 5000 samples and 500,000 features (all categorical with a cardinality of 3).
Two problems I'm trying to solve:

Loading the dataset - I can't load it in memory despite using a computing cluster, so I'm assuming I should use a parallelization library like Dask, Spark, or Vaex. Is this the best idea?
Feature selection - how to run feature selection within a parallelization library? Can this be done with Dask, Spark, Vaex?


texts: Where can I find a >1K dataset of annotated store receipt pictures/scans?

texts:
I’m looking for a large dataset of store receipt pictures or scans. The image quality is not of the highest priority. However, some annotations would be nice:

vendor
date of purchase
total amount
(optional: tax amount)
(optional: currency)
(optional: list of each individual item with associated amount, price, discounts)


texts: How to use vectors produced by TF-IDF as an input for fuzzy c-means?

texts:
I have done text processing with TF-IDF method and as an output got a list of normalized vectors [0, 1] for each document. Such as below:

Document 1
word1:1.0, word2:0.9, ..., word_n:0

Document 2
word2:1.0, word1:0.4, ..., word_n:0
...
etc

The above is basically a list of key-values where key is a term and values  are TF-IDF values, where value 1 means that the term matches the document the most compared to other terms in the set.

My question is, to what form should I transform these vectors in order to properly use fuzzy c-means clustering on them? I feel like it should be 2D matrix of something, but can't figure it out.

At the very end I would like to have a trained model which on a given input could say to what documents (based on the membership values) it belongs with the highest chance.


texts:
Say I have a data frame in Python.
If I want to replace all values in the say, the Size column that are not 'M' or 'S' or 'L' with nan, how do I do so?
Thanks in advance


texts: Variational autoencoders - encoder-decoder neural nets relationship to maximizing evidence

texts: I am new to VAE and I do not know why if the encoder and decoder are learned by a neural networks then how is maximizing the ELBO (evidence lower bound) or maximizing evidence in general relevant to these neural networks. Can you please explain to me the exact process of finding a variational autoencoder using a toy example, outlining the steps?


texts: What are the prerequisites before running Holt Winters Model?

texts:
I just read Demand-Driven Forecasting: A Structured Approach to Forecasting(Wiley and SAS Business Series) and have a few doubts in Holt-Winters Model:

1) Unlike OLS Regression Modeling technique or ARIMA, no assumptions were checked before running Holt-Winters. For instance, in ARIMA, we first make the data stationary before running ARIMA or in OLS, we check normality, auto-correlation etc. However, as per the book, no test was conducted before and after running Holt-Winters. We just calculate MAPE and check if it is acceptable. So, can someone confirm if there are any tests that we should do before and after running Holt-Winters?

2) Can Holt-Winters model incorporate causal factors such as Price, Promotion, Marketing etc?

3) When we are running Holt-Winters multiplicative or additive model, we don't need to explicitly adjust for seasonality before?

4) How to treat for missing values or outliers in time series forecasting? We obviously cannot remove missing values rows as the time interval will break. Hence, for missing values, I have thought to use CAGR for that period(weekly, monthly or yearly) in the last 5 years and use it to estimate missing value. Can someone confirm if this correct?

Also, I believe we shouldn't remove outliers and rather look why it is there in the first place. It might be because of some external factors like Price Change or Promotion which is causing the spike. This will ofcourse be not detected by Holt-Winters. So, if it is a rare promotion, then we should remove that value and insert the normal demand again by using CAGR of past data in order to maintain the historical pattern. Hence, the spike will be gone and the historical trend will be intact, thus increasing our MAPE. Can someone confirm if this correct?


texts: Determine most important features per class in binary classification?

texts:
The only good answer i found was this  :
https://stackoverflow.com/questions/33118361/determine-most-important-feature-per-class
But the above answer doesn't work in binary classification because there is only one i, so instead i used the top most positive weights to detect most important features for class 1, and most negative ones for class 0 :
myclassifier = SGDClassifier(loss='log',  penalty='l1', l1_ratio=0.9, learning_rate='optimal' , shuffle=False, fit_intercept=True , verbose=True)
myclassifier.fit(X_train, Y_train)

top_indices_class1 = np.argsort(myclassifier.coef_[0])[-100:]

for x in top_indices_class1:

    print(feature_list[int(x)]  , myclassifier.coef_[0][int(x)]  )

top_indices_class0 = np.argsort(myclassifier.coef_[0])[:100]

for x in top_indices_class0:

    print(feature_list[int(x)] , myclassifier.coef_[0][int(x)])

Is this a good method to find the most important features for each class? is there any better way to do this?


texts: Human readable format for clusters of word vectors

texts: Let's say I have pretrained word2vec model and apply it to dataset consisting of article titles from "The Guardian". It seems pretty obvious that titles coming from "Science" section would form one cluster in latent space and titles from "Fashion" section would form another cluster in latent space. But the thing is my dataset doesn't have category label for each title. How can I come up with such human readable interpretation of cluster centers(probably coming from Kmeans)?


texts:
classes = {0: 'Other', 1: 'Thunder'}

    x_new = p.read_csv('LeeAnnDataSet.csv', sep=',').values

    y_predict = train_test_split(x_new)

    print(classes[y_predict[0]])

    print(classes[y_predict[1]])

**Every time I try using this command, I receive the error message below. I don't understand what I may be doing wrong. **
    TypeError                                 Traceback (most recent call last)
    <ipython-input-66-5abff76d706f> in <module>

    4 y_predict = train_test_split(x_new)

    5

    ----> 6 print(classes[y_predict[0]])

    7 print(classes[y_predict[1]])

    TypeError: unhashable type: 'numpy.ndarray'


texts: How to predict an approximate weekly/monthly number, when the Unique Daily Visitors for that week/month are already known

texts:
I am trying to come up with a formula or machine learning algorithm using which I can approximately predict the weekly or monthly users.

What to keep in mind is that I already have counts for the unique visitors per day for the week/months that I would like to make a near accurate prediction. Here, simply summing the daily unique users would not work, as they can be unique on one day but not on two days as they can have a session lasting over 2 days.

This method is to serve as an alternative to running a Spark job on the whole week/month data in order to save time and resources - Is this possible?

I have looked at Time Series and Linear Regression, but need more clarification on the possible approaches and also on any work-arounds?


texts: What are the challenging benchmarks for self driving cars object detection beside Kitti

texts: I am working on 3D object detection in the context of self driving cars and I was wondering if there are other challenging benchmarks for this beside Kitti.