texts:
I'm trying to predic stock values from a dataset, for example: Google stock.

I have this easy model.

regressor = Sequential()
regressor.add(LSTM(units=32, return_sequences=True, input_shape=(X_train.shape[1],1)))
regressor.add(LSTM(units=32,return_sequences=False))
regressor.add(Dense(units=1))

Layer (type)                 Output Shape              Param #
=================================================================
lstm_3 (LSTM)                (None, 60, 32)            4352
_________________________________________________________________
lstm_4 (LSTM)                (None, 32)                8320
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 33
=================================================================
Total params: 12,705
Trainable params: 12,705
Non-trainable params: 0

I'm using a timestep=60 and only 1 feature. So, when I evaluate my model with X_test data, I have this results.

But, I'm trying to make a prediction without knowing any data. I train my model and then I want to predict the future values.
I think that I will need the last sequence that I use on my train, predict the new value, and incorporate to my predict sequence.

If I want to predict the next 240 values, I think that I need to code something like this

for i in range(240)
  1. Select the last 60 values (my timestep)
  2. Use sequence generated in step 1 to predict the new value.
  3. Remove [0] item on my sequence generated in step 1, and push the value that I generated in step 2 to my sequence.

I coded it.

sequence_timestep = 60
last_sequence_train = X_train[-1]

predictions = []

def sequence_constructor():
    if len(predictions) >= sequence_timestep:
        new_sequence = predictions[-dimension_seq:]
    else:
        splitter = sequence_timestep - len(predictions)
        part_1 = last_sequence_train[-splitter:]
        new_sequence = np.append(part_1,predictions) #Concatenate 2 list
    new_sequence = np.array(new_sequence)
    return new_sequence

With that method, I use this:

for i in range(240):
    new_sequence = sequence_constructor()
    new_prediction = model.predict(new_sequence)
    predictions.append(new_prediction)

But, the output of my model is complete different that I expected.

I think that I'm forgeting something, but I can't see what happen.
Any ideas?

I tried with bigger timestep, but the results are the same.


texts: Operation composition in tensorflow-1.14,15

texts:
I am trying to make do a very simple operation in tensorflow 1.15.
I have a pretrained network that I would like to use for inference.
This network exposes a tf.global_variable let's say input and an output say output.
Now, having a session variable sess, if you want to do inference, you just run:
inference_output = sess.run(output, {input: input_numpy})

My problem is that I have already implemented a frontend to this inference (to calculate input) which is a set of tensorflow operations that I will call exp and which expects an input variable say args.
I can't find a way so the value of exp is feeded to the global variable input and thus my graph becomes:
args -> (expr -> input) -> output

Right now I am doing this as:
preprocessing_inp = sess.run(expr, {args: args_numpy})
inference_output = sess.run(output, {input: preprocessing_inp})

but what I would like to have only one operation (so everything does a single gpu pass):
inference_output = sess.run(output, {args: args_numpy})


texts: How to implement an oblique decision tree for regression?

texts: There are numerous ways to induce an oblique decision tree in the decision tree induction domain, such as using a support vector machine to determine the best hyper-plane. However, is it possible to generate an oblique decision tree for regression? I discovered that the majority of existing literature has omitted this issue. By the way, is there an open-source machine learning tool that already includes an oblique tree for regression?


texts: Downward trend in the residuals vs fit plot with constant variance interpretation?

texts: I have build a regression model where the residual vs fit plot indicates a downward trend with a constant variance. I am having trouble interpreting and understand the problem at hand. Below is the chart showing the plot. Can someone help me explain it? What does it signify and is it satisfying the assumptions of a linear regression model? Thanks


texts: Dimensionality Reduction on Cross Channel EEG data

texts:
I'm working on a project predicting seizures events using 10 minute EEG data recordings. This involves identifying the preictal (pre-seizure) phase from the interictal (normal/between seizure) phase, so binary classification.
There are 16 electrodes, therefore 16 channels in my data. I have read that calculating the cross-channel power density spectrum is a popular method for this kind of problem, but this would result in a high number of dimensions so I want to know if it is okay to use dimensionality reduction on cross-channel derived features and if so, which technique would be best.
I have tried PCA, but I read that features with maximised variance is not a good method for classifying preictal and interictal recordings. LDA would have been nice to use as this is supervised learning, but I only have two classes so I can't use this.
I'm interested if ICA would work in this instance, or would it just end up identifying the singular channel power densities? Or possibly random forests?


texts: Quick start using python and sklearn kmeans?

texts:
I started tinkering with sklearn kmeans last night out of curiosity with the goal of clustering users into groups to see what kind of user groups I can derive.  I am lost when it comes to plotting the results as most examples have nice (x,y) coordinates.  For example, the iris data set has pedal width and pedal length.  From my experimentation, I don't seem to have anything that displays very nice.  Is this assumption correct / does anyone have tips, pointers, learning resources that they could offer?

import pandas as pd
import pprint
import numpy as np
from sklearn.preprocessing import normalize
from sklearn.preprocessing import LabelEncoder
from sklearn.cluster import KMeans
from collections import defaultdict
import matplotlib.pyplot as plt
from matplotlib import style
style.use('ggplot')

I normalized the data as it had a wide variance...again, not sure if this is a correct assumption to make

X = np.array(normalize(data, axis=0, copy=False))

kmeans = KMeans(n_clusters=3)
pred = kmeans.fit_predict(X)
labels = kmeans.labels_
cent = kmeans.cluster_centers_

plt.scatter(X[:, [4]], X[:, [6]])
plt.scatter(cent[:, [4]], cent[:, [6]], marker="x", s=150, linewidths=5, zorder=10)
plt.ylabel('Count')
plt.xlabel('Department')
plt.show()

Any pointers are appreciated, I will include sample data below.  Thanks!

Sample Data:

emp_type,title,work_country,director_userid,dept_name,business_unit_name,UserCNT
0,9,7,29,20,2,2
0,13,7,8,14,6,5
0,4,3,56,29,8,3
0,15,3,36,32,2,3
0,4,3,32,16,2,0
0,4,1,40,13,6,0
0,4,3,62,12,4,1
0,13,7,61,5,13,4
2,1,3,70,35,15,2
0,4,3,64,4,13,0
2,1,3,43,43,2,0
0,13,7,50,17,16,0
2,1,3,31,26,2,1
2,1,3,65,58,17,0
0,4,3,57,63,12,0
2,1,6,7,45,18,2
2,1,3,43,42,2,0
1,1,7,65,58,17,0
2,1,3,32,16,2,0
2,1,3,29,20,2,0
0,4,0,50,17,16,2
0,5,3,20,23,9,0
0,9,3,32,16,2,2
0,4,3,5,51,12,0
2,1,7,51,53,7,0
0,13,7,37,55,12,0
2,1,4,19,62,13,0

Example clustering using entire data set:


texts: How to balance class weights correct for a CNN in Keras, given an unbalanced data set?

texts:
I want to use class weights for training a CNN with a imbalanced data set.
The question arise if the sum of the weights of all examples have to stays the same?

My previous plan was to use the function compute_class_weight('balanced,np.unique(y_train),y_train) function from scikit-learn.

But I'm totally unsure if this is even suitable for the class weights of a CNN?

Thank you in advance for each tip


texts: Click Through Rate calculation (CTR) calculation problem

texts:
So I'm doing a use case for a company interview and one of the questions is to calculate the CTR for a sorting algorithm.
My question would be: Should I remove the operations where there were no products shown (even though some of them there were clicks on products)?Example (should discard this columns?):
Number of impressions | Number of clicks
            0         |      1
            0         |      3
            0         |      0

CTR = #products clicked/#products shown
My other doubt is wether should I sum the clicks and the products shown and then divide one on other OR should I do the CTR for each operation:
Pseudo Python code (imagine that I have a dataframe with several operations as rows):
CTR = pd_df["no_clicks"].sum() / pd_df["no_impressions"].sum()

OR
pd_df["CTR"] = pd_df["no_clicks"] / pd_df["no_impressions"]
CTR = pd_df["CTR"].mean()


texts: What's the default Scorer in Sci-kit learn's GridSearchCV?

texts: Even if I don't define the scoring parameter, it scores and makes a decision for best estimator, but documentation says the default value for scoring is "None", so what is it using to score when I don't define a metric or list of metrics?


texts: Running sklearn trained classifier in a windows machine without Python

texts:
I have a classifier (and vectorizer) that I can export as a pickled model. I am using Python.

This is on my local machine. Now, in order to perform the classifications in "production" I need to be able to run this in another machine that does not have Python installed. Any idea how to do this? Is it possible to create a Python executable with the pickled model in it?


texts: Using predictive modelling for temperature data set

texts:
I am absolutely new to this area of predictive modelling in data science. I am not able to understand how and what modelling techniques do we use? Does it depend on the data type? Does it depend on size of data?

To be specific to the title, I have to predict missing values in a given temperature data set and i am unaware of anything that i can use. Could someone guide me through?


texts:
Does anyone know of any libraries I could use to convert a word into a semantic prime, given a list of approved primes?

In other words, given a sentence or a word, I would like to find the combination of "prime" words that best match the input semantically.

A good example of this would be a function that takes in a Wikipedia article and outputs something like Simple Wikipedia - it has the same semantic content, but restricts word usage to a number of prime words (in this case, simpler words).

Another example: say my primes are ["person", "move", "tool", "nice", "building"].

Then I would like to have some function, library, or NLP algorithm that converts "hammer" into ["building", "tool"], "car" into ["moving", "tool"], and  "friend" into ["nice", "person"].

My first intuition was to use word2vec: loop through every combination of the primes and find the combination of primes who's vector most closely matches the original word. However, I suspect this will not work out very well (partially because vector addition doesn't seem to be accurate enough). Is there a more robust way to do this?


texts: How to plot a 3-axis bar chart with matplotlib (and pandas + jupyter)

texts:
I'm a bit confused about how to go about plotting a 3-axis bar chart:

So my jupyter notebook reads in an excel/sheet and I have a table:

     2001  2002   2003   2004
Mar  15    16     14     18
Jun  23    25     28     24
Jul  24    23     22     26

I'm a bit confused about how to go about building the visualization and having the axes/labels set up correctly.


texts: Financial Time Series and Machine Learning question

texts:
I am working on a Machine Learning project applied to a financial time series.

Initially, I grabbed features (open, high, low, close) and implemented a Random Forest.

One of the subsequent tasks says to explore if the close-price serie is non-stationary, and if its not, to differentiate it turning it into stationary. I was able to differentiate it fine (ran the Augmented dickey fuller test to verify as well). But I have the following question: When differentiating a time-serie and working with machine learning, are you afterwards supposed to train the model again using this new differentiated serie? I'm having problems to understand the practical application of differentiating the time-serie (i.e., turning the serie into stationary)


texts: Naive about which Naive Bayes in article

texts:
Working with the Naive Bayes spam filtering article on Wikipedia
(https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering)

Is it a binary multinomial equation or a Bernoulli form or some other?

In this 2006 paper, it discusses that there are many Naive Bayes algorithms: Spam Filtering with Naive Bayes – Which Naive Bayes?
http://www.aueb.gr/users/ion/docs/ceas2006_paper.pdf

The paper states that binary Multinomial NB performs best.

If it is not multinomial, what changes are needed to make it so?


texts: How to find the position of a company given other companies in a Pandas DataFrame with Python

texts:
So I have a Pandas DataFrame. I am doing some data analysis with Python with some sets of companies based on the products they offer on my website.

For example, I have a column called ProductID and another called Company

So to find the most trending products on my website, I can do so by:

df.ProductID.value_counts(normalize=True).nlargest(10).plot(kind="bar")

And each product is offered by a company, and a company could have several products, that is a company can have several ProductID. ie Company A has ProductID 10,12,13,14

How do I find the market position of a company? Keeping in mind that out of the top 10, a company's product could appear up to 3 times or even more depending on the company's popularity


texts: TensorFlow - TFRecords load and transform images with bounding boxes

texts:
I'm trying to build a 'Car Classifier' using TensorFlow.
I have 1000 labelled JPG images, 800x800, complete with bounding boxes and associated annotations.coco.json; split into train/validate/test folders.
I've managed to load the TFRecordDataset's using the code below:
TFRecord Data Set Loading Steps
# Load TfRecord data sets
raw_train = tf.data.TFRecordDataset([training_file])
raw_validation = tf.data.TFRecordDataset([validation_file])
raw_test = tf.data.TFRecordDataset([testing_file])

# Load label map
category_index = label_map_util.create_category_index_from_labelmap(label_map_file, use_display_name=True)

------------------------------------------------------------------------------------------
def extract_features(tfrecord):
    # Extract features using the keys set during creation
    features = {
        'image/object/bbox/xmin': tf.io.VarLenFeature(dtype=tf.float32),
        'image/object/bbox/ymin': tf.io.VarLenFeature(dtype=tf.float32),
        'image/object/bbox/xmax': tf.io.VarLenFeature(dtype=tf.float32),
        'image/object/bbox/ymax': tf.io.VarLenFeature(dtype=tf.float32),
        'image/object/class/label': tf.io.VarLenFeature(dtype=tf.int64),
        'image/width': tf.io.FixedLenFeature([], tf.int64),
        'image/height': tf.io.FixedLenFeature([], tf.int64),
        'image/encoded': tf.io.FixedLenFeature([], tf.string)
    }

    # Extract the data record
    sample = tf.io.parse_single_example(tfrecord, features)

    image = tf.io.decode_image(sample['image/encoded'])
    label = sample['image/object/class/label']

    return [image, label]

raw_train = raw_train.map(extract_features)
raw_validation = raw_validation.map(extract_features)
raw_test = raw_test.map(extract_features)

Transform/Resize images for Training
ORIGINAL_IMG_SIZE = 800
RESIZE_IMG_SIZE = 160 # All images will be resized to 160x160 or 614x614 maybe for Yolo?

def format_example(image, label):
    #https://stackoverflow.com/questions/62957726/i-got-value-error-that-image-has-no-shape-while-converting-image-to-tensor-for-p
    image.set_shape([ORIGINAL_IMG_SIZE, ORIGINAL_IMG_SIZE, 3])
    image = tf.cast(image, tf.float32)
    image = (image/127.5) - 1
    image = tf.image.resize(image, (RESIZE_IMG_SIZE, RESIZE_IMG_SIZE))
    return image, label

Tensorflow examples only seem to talk about resizing the whole image and not about how to handle resizing of bounding boxes within the image, and bounding box labels.

Does anyone have any examples of how to handle the resizing of images together with bounding boxes contained within the image?

Training Pipeline

Again Tensorflow examples only seem to train with whole images, not with images with bounding boxes and associated bounding box labels.

Does anyone have any examples of TensorFlow Transfer Learning training with images with bounding boxes and associated bounding box labels?


texts: Eliminating rows in a dataframe based on specific conditions

texts: I have two columns in a dataframe namely 'origin' and 'destination'  which contains the names of different places. I need to remove the rows that don't contain 'PHX', 'JFK', 'NTU' in either of these columns. If atleast one among the 2 columns has one of these 3 places, the row can stay, else the row must be dropped altogether. Can you please help me to code this part ?


texts:
https://www.google.com/search?sxsrf=ALeKk01_SgA8G4UfNm4rOqku4yJBFvKhLw%3A1600154854621&source=hp&ei=5mxgX8ztI6KZ4-EPq-mL8Ak&q=homophones+example&oq=Homophones&gs_lcp=ChFtb2JpbGUtZ3dzLXdpei1ocBABGAEyBQgAELEDMgUIABCxAzICCAAyCAgAELEDEIMBMgUIABCxAzICCAAyAggAMgUIABCxAzoHCCMQ6gIQJzoECCMQJzoFCAAQkQI6CAguELEDEIMBOgUILhCxA1DkKliKSGDuUGgBcAB4AIAB6wGIAe8NkgEFMC44LjKYAQCgAQGwAQ8&sclient=mobile-gws-wiz-hp
Are there Machine learning algorithms for forming Homophones from input dataset word?
Homophones examples :
accessary, accessory.
ad, add.
air, heir.
all, awl.
allowed, aloud.
alms, arms.
Input : ad
Output : ad, add
Are there Machine learning algorithms for forming Homophones from input dataset word taking Indian regional languages viz Hindi, Gujarati, Bengali etc and other languages viz French, German, Italian, Spanish, Dutch etc?


texts: What models should I try with a time series database?

texts:
I've acquired and cleaned a dataset that shows statistics from every county in New York State during 2010-2019 focusing on the NYS School Aid correlating it to other growth and criminal statistics. Source

County
Year
AidAmount
Population
JailPopulation
CrimesReported

Albany
2010
253982539.0
304086
698
20890.0

Albany
2011
227093331.0
304596
614
20290.0

Albany
2012
226720094.0
305723
622
18940.0

Albany
2013
235041459.0
306589
629
17908.0

...

Yates
2016
25572768.0
25059
46
623.0

Yates
2017
26203199.0
25002
42
492.0

Yates
2018
26480853.0
24951
42
473.0

Yates
2019
27494973.0
24913
43
351.0

I'm trying to answer questions such as does a lower amount of Aid Amount in a county lead to higher illicit rates? Can we predict counties becoming high or low risk overtime?
I've done some EDA and now I'm trying to learn more about models and their applications, I want to know what kind of models should I use for this dataset.
I know I might use some classification models where I'll create a high risk label for counties that present  >0.7 per capita crimes reported and then use education aid per capita as the training samples.
As for time series models I'm lost on what ARIMA model should I try, I want to predict Aid Amount, population and criminality values
I would appreciate any input you have on this, I would like to learn as many models as possible.


texts: Two different results from seaborn and matplotlib for a kde plot. What's the reason?

texts:
I'm plotting kde plots for a set of data with seaborn and scipy.stats.gaussian_kde:

My understanding is that under the hood, seaborn uses scipy (see here).

It also seems as though the bandwidth variable is in both cases 'scott' by default, so that doesn't seem to be the cause.

Does anyone have any insight into why the plot are different?

UPDATE

So it appears that the different shape of the seaborn plot is due to the fact that instead of using scipy, seaborn uses the kde of simplestats if that library is available locally.

So the question I have now is why is the curve produced by simplestats so different than the one produced, say, by scipy (and perhaps othe libraries).

Below in the comments it was suggested that the reason has to do with the bandwidth setting, but I played around a bit with that and the differences I saw the curves was no where near as extreme. Here are two seaborn/simpletats kde curves with two different bandwidth settings (scott versus silverman):

Then here, by contrast, are two scipy-produced kde curves with, again scott versus silverman:

One possible explanation may be this note in the seaborn documentation for kdeplot (see here):

  Note that the
          underlying computational libraries have different interperetations
          for this parameter: statsmodels uses it directly, but scipy
          treats it as a scaling factor for the standard deviation of the data.

However, I'll need to do a bit more research/learning to understand if that's relevant here. Does anyone else know if this might be the explanation?


texts: Performance of Triplet loss network vs multiclass classification

texts:
I am training a triplet loss based classification network and a normal multiclass classification network on some image data. In my case, triplet loss network performs poor than multiclass network. I have tried changing layers, neurons, margin etc for triplet loss network but multiclass network performs better. Are there any cases where triplet loss network can perform worse than normal multiclass classification. If no...what are the possible things I can  look at improve or change in triplet loss network for it to work.

It is 5 class classification with some given features


texts:
Given a dataset with 30 fields and 25000 instances,

1) what are your suggestions for novel methods of rule extraction?

2) Can I use association rule mining in addition to sequential rule mining?

3) which method can be more appropriate for such a big dataset?Apriori-based like  SPIRIT, SPADE, SPAM, IBM or pattern growth ones like FreeSpan, PrefixSpan, SLPMiner?

*the output field (risk) is labelled as very low, low, medium, high, very high. Also, there is a temporal field (date) for each instance. An example is given below.

$$
\begin{array}{|c|c|c|c|}
\hline
\mathbf{date}& \mathbf{temperature} & \mathbf{density} & \mathbf{risk} \\ \hline
 2018/1/2 & 15 & 100 & \text{very high}\\ \hline
 &  & &\\
 \end{array}
$$


texts: using "OneVsRestClassifier" from sklearn in Python to tune a customized binary classification into a multi-class classification

texts:
I have binary classification method name Fclassifier I need to apply it in a multi-class classification problem, this classifier doesn't have any decision_function (or predict_proba) and its core only accept {1,-1} as class labels! I am trying to use OneVsRestClassifier multi-classifier from sklearn so that I can tune the Fclassifier into multi-class classifier:

fm = FClassification(n_iter=1000, init_stdev=0.1, rank=2, random_state=123,l2_reg_w=0, l2_reg_V=0, l2_reg=0, step_size=0.1)
OneVsRestClassifier(fm).fit(X_train,y_train)

but I am facing the following error:

  RuntimeError: Cannot clone object FClassification(init_stdev=0.1,
  l2_reg=None, l2_reg_V=0, l2_reg_w=0,
           n_iter=1000, random_state=123, rank=2, step_size=0.1), as the constructor does not seem to set parameter l2_reg_V

do you think if it is the problem of Fclassifier that can not satisfy the expectation of OneVsRestClassifier and if I should define a customized predict_proba which can fit my data to the classifier in order to solve the problem? or if the error may have other reasons and solutions?


texts: Monitoring machine learning models in production

texts:
I am looking for tools that allow me to monitor machine learning models once they are gone to production. I would like to monitor:

Long term changes: changes of distribution in the features with respect to training time, that would suggest retraining the model.
Short term changes: bugs in the features (radical changes of distribution).
Changes in the performance of the model with respect to a given metric.

I have been looking over the Internet, but I don't see any in-depth analysis of any of the cases. Can you provide me with techniques, books, references or Software?


texts: Are time series algorithms immune towards collinearity?

texts:
I have a time series dataset with 63 features and a univariate dependent variable. This is my first major time series project, so I was wondering if algorithms like ARIMA and LSTM are immune towards variables that have collinearity, similar to gradient boosted methods such as xgboost that is actually seen as a robust method. Below is the image of the correlation matrix. I have used xgboost with the dataset and have not got satisfactory results, even after removing some of the variables that are in the red category, so I'll proceed with the next step by using some time series methods.

The image can always be zoomed to get a clearer description of features.


texts: Basket items optimisation minimising constraints

texts:
I have a real problem (not home work) when I have to distribute an ordered list by position to respect some constraints

eg.

1. 11
2. 15
3. 18
4. 18
5. 1

baskets:

b1: 11, 1
b2: 18,15, 18

to respect as much as possible

b1 = 30%
b2 = 70%

I do not want to use a brute force since I'm working with long list (>10.000 items).

Can you propose a technique or a library?


texts: How does Batch Normalization in Machine Learning address covariate shift and speed up training?

texts:
In this video and this answer, it's mentioned that batch normalization doesn't allow the mean and variance of the parameters of any particular hidden layer to vary too much with change in previous layer's node values. Let's say we have a network like this (taken from the video):

As we change the mini-batch that the neural network is trained on, it's possible that the distribution of the input features  with respect to different mini-batches is different. As far as I understand, this is called covariate shift. An issue with covariate shift is that maybe the distribution of input features with respect to the validation set examples is different from the training set (ref. black cats / colored cats example in the linked video), and the distributions w.r.t. the different mini-batches are different.

Consequently, since the learned weights connecting the input layer to the next layer depend on the input features distribution, the learned weights will accordingly change with each mini-batch. Thus the activations in the next hidden layer will also vary in their distribution w.r.t. different mini-batches.

First question: Is my understanding so far correct?

Now I can understand that limiting the mean and variance of input features or that of the hidden layer activations will address the covariate shift problem only if the distributions across mini-batches (or between training and validation sets) are fundamentally the same but differ in parameters (not the neural network parameters - I mean mean and stddev). Crudely speaking, if the distribution of input features w.r.t different mini-batches is normal, but the parameters (mean and stddev) are different, then batch normalization seems fine.

Question 2: But what if the input features have normal distribution w.r.t. one mini-batch and a completely different distribution w.r.t. another? I don't see how limiting the mean and variance will help. Maybe I'm ignoring some key assumptions.

Question 3: Why does limiting the variation in the distributions of the hidden layers speed up the convergence?


texts: Running multiple times of a model is for model randomness or data randomness?

texts: When a paper report the average and std of a model on a dataset, it means that they have changed the split of training and test sets and run the model multiple times or they just run the model on constant splitting multiple times to find the randomness of model? Which one is more common?


texts: Why compressed image size is greater than original one in kmeans algorithm?

texts:
I have a png image as shown below.

And I use kmeans algorithm to compress the image by color quantization. I compressed the image to use 64 colours. The code is:

ncolor = 64
rimage = image.reshape(image.shape[0]*image.shape[1],3)
kmeans = KMeans(n_clusters = ncolor, n_init=10, max_iter=200)
kmeans.fit(rimage)
centers = kmeans.cluster_centers_

labels = np.asarray(kmeans.labels_).reshape(rows, cols)
compressed_image = np.zeros((rows, cols,3),dtype=np.uint8 )
for i in range(rows):
    for j in range(cols):
            compressed_image[i,j,:] = centers[labels[i,j],:]

fig, ax = plt.subplots(1, 2, figsize=(16, 6),
                       subplot_kw=dict(xticks=[], yticks=[]))
ax[0].imshow(image)
ax[0].set_title('Original Image', size=16)

ax[1].imshow(compressed_image)
ax[1].set_title(f'{ncolor}-color Image', size=16);

io.imsave('compressed_tiger.png',labels);

The image is the origin image. And the resulte shown on jupyter notebook is:

since I compressed the image to 64 colours but the size of the saved file compressed_tiger.png (588KB) is larger than the original one (435KB). I don't understand why it becomes larger.


texts: How to set parameters to search in scikit-learn GridSearchCV

texts:
I want to use scikit-learn's GridSearchCV to optimise a BaggingClassifier that uses a support vector classifier (SVC). I want the grid search to search over parameters for both the BaggingClassifier and the SVC.

I have tried this setup:

svc_pipe = Pipeline([
     ('svc', SVC(probability=True)),
])
pipe = Pipeline([
     ('bag', BaggingClassifier(svc_pipe, no_estimators=50)),
])

params = {
    'bag__bootstrap_features' : [True, False],
    'bag__svc__kernel': ['linear', 'rbf'],
    'bag__svc__decision_function_shape': ['ovo', 'ovr']
}

rnd_search = GridSearchCV(pipe, param_grid=params)

but I get this error:

ValueError: Invalid parameter svc for estimator BaggingClassifier(base_estimator=Pipeline(memory=None,
     steps=[('svc', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
  max_iter=-1, probability=True, shrinking=True,
  tol=0.001, verbose=False))]),
         bootstrap=True, bootstrap_features=True, max_features=1.0,
         max_samples=1.0, n_estimators=50, n_jobs=-1, oob_score=False,
         verbose=0, warm_start=False). Check the list of available parameters with `estimator.get_params().keys()`.

Can someone show me what I have done wrong?


texts: Titan Black 6GB for US$ 270 - good deal for DL?

texts:
I want to upgrade my GTX 650ti 1GB for something better suited for deep learning. I was offered a Titan Black 6 GB for US 270. I know there are several better cards today, but my budget limit is US$ 300.
Any recommendations?

Gonzalo


texts: How is loss computed for multiclass CNN with an output layer larger than the number of classes?

texts:
I have built a CNN in pytorch for classifying the Fashion-MNIST dataset (10 classes). The images are 28x28.

I have constructed the final layer in my model as an output of 50. (i.e.         $nn.Linear(100, 50)$). Also I am using cross entropy loss.

I am confused about how loss is calculated for these data sizes. From what I had known about backpropagation and loss function, the output of the neural net is compared with the expected result.

For example, using mean square error, the loss function is $(output - expected)^2$. So if I had a binary classifier, say the class labels are $({0,1})$ then the output of the neural network would need to be one dimension to compute the loss.

Now if I had three classes, how would you calculate loss? How many outputs would you need? Since the expected class label is still just a single digit, I don't see how loss can be calculated if the output of the neural network is more than one dimension.

For example, if the output is $[ x1, x2, x3]$ and the expected class label is $y$, I don't see how loss could be calculated since the dimensions don't agree.

So how is loss computed against a class label when the output of a neural network isn't a single digit?


texts: ML Approach for Getting List of Observations with Similar Features (Discrete+Continuous)

texts:
I have a dataset with 19k observations. Each has approximately 448 features:
 - Text description turned into vectors of size 300
 - 16 categorical variables represented numerically
 - The remainder are quantitative features

Each observation also has a list pointing to 10 other observations (from the 19k) that it's most similar to. I want to train an ML model that can understand how the 448 features contribute to this "similarity". Once the model understands that, it could accurately pick the 10 closest existing observations for any new observation, based on how it understands this "similarity".

I've tried clustering with Python's scikit learn (K-Means, MBKM, Affinity Propagation) but these haven't really worked out. I'm currently trying NearestNeighbors but I'm not sure I've got it right.

Please help! Thank you.


texts: How to create list of multi value in a list format of each field in a single column?

texts:
I am very new to programming and python; coming from functional background. I am currently using a tutorial: https://www.analyticsvidhya.com/blog/2019/04/predicting-movie-genres-nlp-multi-label-classification/

The purpose of this tutorial is to build a model that can predict genre of a movie just using the plot details (available in Text Form). I am mapping this concept to create tag/prediction of customer complaint register.

For example:
1 Movie --> Many Genres (horror, fantasy)

1 complain --> many Genre (Warranty, Air Conditioning)

I am currently up to the stage to make the genre into list format then get the genre tags in a list (unique) but I don't how to achieve it in Python.

i.e. line 0 i showing Warranty and Heating/Cooling as 1 value instead of having its own quote ''?


texts: How to decide the right granularity in time series?

texts: I am wondering how to decide right granularity I should use for time series if I have data for hourly as well as daily and given that I don't have business constraint which asks me to do forecasting for daily/hourly basis.


texts: Vertical and horizontal lines appearing on large confusion matrix?

texts:
I have produced a large heatmap-like confusion matrix and am seeing horizontal and vertical lines on it, so I'm trying to determine:

What they mean
Why they are there
How I can improve on this

Approach

I am relatively new to ML and in the early stages of of a multi-class text classification problem.  I may be a little verbose so you can ensure I'm on track and my question isn't due to a flaw in my approach.

I have 90,000+ samples that I'd like to be able to classify into one of 412 classes.  I've taken a basic look at the data in terms of its class distribution and the unigrams and bigrams that are selected for each class.  Continuing exploration, I trained 4 classifiers on the data, receiving the following levels of accuracy:

LinearSVC                 0.547190
LogisticRegression        0.530063
MultinomialNB             0.368121
RandomForestClassifier    0.200568

Having had a lot of trouble plotting a confusion matrix this large with Seaborn or Matplotlib, I used used the following python code to produce a confusion matrix in CSV:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC

def make_confusion_matrix(a,p,c):
    cm = pd.DataFrame(0,index=c,columns=c)
    for count in range(len(p)):
        cm[int(a[count])][int(p[count])]+=1
    return cm

tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5, norm='l2', encoding='latin-1', ngram_range=(1, 2), stop_words='english')
features = tfidf.fit_transform(df['DetailedDescription'])

model = LinearSVC()
X_train, X_test, y_train, y_test, indices_train, indices_test = train_test_split(features, df['BreakdownAgency'], df.index, test_size=0.33, random_state=0)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

cm = make_confusion_matrix(y_test.tolist(),y_pred,labels_df['TOOCS Breakdown Agency'])
cm.to_csv('ConfusionMatrix.csv')

I was finally able to view the confusion matrix in a heatmap style by using Excel conditional formatting, which produced the matrix above.

Interpretation

Given that the X axis is actual and y axis is predicted:

I interpret the horizontal lines as showing incorrect bias of predictions towards a class with a disproportionately large number of samples?

I interpret the vertical lines as showing incorrect predictions away from a class with a disproportionately large number of samples?

Does this show that the model is both overfitting and underfitting the data?
Or that the samples within my classes are overly diverse?

Action

I'm contemplating:

Manually adding samples to the classes that have very few (a minimum of 10?).
Using SMOTE to oversample small classes (knn=6).
Potentially removing some samples that are atypical or incorrect.

Any help on my Interpretation or Action would be greatly appreciated!


texts:
I'm trying to run a simple logistic regression on PyMC3. Here the code:

with pm.Model() as logistic_model:
    alpha=pm.Normal('alpha', mu=0, sd=100)
    beta1=pm.Normal('beta', mu=0, sd=100)
    beta2=pm.Normal('beta2', mu=0, sd=100)

    argtemp=alpha + beta1*data['age'].values +beta2*data['educ'].values
    prob=pm.invlogit(argtemp)
    Y_obs = pm.Bernoulli('Y_obs', p=prob, observed=data['income_more_50K'].values)
    map_estimate = pm.find_MAP()
    sampler=pm.Metropolis()

    trace= pm.sample(40000, step=sampler, start=map_estimate)

The sampler run, but at the end it gives this error:

The gelman-rubin statistic is larger than 1.4 for some parameters. The sampler did not converge.
The estimated number of effective samples is smaller than 200 for some parameters.

Can someone help me in understanding what is wrong, and why the sampler do not converge?

Thank you!


texts: Unsupervised learning for anomaly detection


texts:
I f have a time series of long time horizon (like a stock price), how do I train in the simplest way the echo state network?

If I chose the training set to be 2 years e.g. then I could chose every day in those 2 years as input.

Can I train the network for 1 month? So i divide the 2 years in parts of 1 month. And for every month in the past I train the network and do average of all neworks?

So, waht are common methods for time series analysis with echo state networks for training? Do I rather train the whole time series or do I divide it supparts to train?


texts: One Neural network with multiple outputs or multiple neural networks with a single output?

texts:
I an building a feed forward deep learning model using tabular data. The inputs are numeric features or categorical features (represented with embeddings). The outputs are the same number of numeric input features.
Is there any known research or models out there which verifies that using a single model with multiple outputs would be better/worse than multiple models, each with a single output?
In essence, with N observations and M outputs, a single model minimizes:
$$
\frac{1}{N}\sum_n^N\sum_m^M \left(y_m^{(n)} - \hat{y}_m^{(n)} \right)^2
$$
while multiple models with single output, each minimize:
$$
\frac{1}{N}\sum_n^N \left(y_m^{(n)} - \hat{y}_m^{(n)} \right)^2
$$
For a single value of $m$.
Any reason one would be preferred over the other, or do I just have to try and see for myself?


texts:
I need to try out few different machine learning methods (SVM, Logistic regression etc.), predict a value either true or false, and write down their AUC and Accuracy of these predictions.
I have allready successfully done that, now i have a two matrixes one for AUC and one for Accuracy, and they are filled with data from SVM and logistic regression (one row).
Now i need to create models for SVM and Logistic regression 10 more times (i should use bootstrapping sampling) and with that i should have 10 rows of my AUC and accuracy data. I have read multiple articles and guides/tutorials, however i can't figure out how to achieve this. I also found and tried few libraries ( one is ROSE and the other one is boot) and none worked for me. Because if i understand the  assigment correctly i need to get 10 different samples from my dataset, and then seperate the data in train and test sets so i can compare the models AUC and accuracy and see how good those models actually are.
Like i said i found multiple sources and the best thing i came up with is this:
 for (i in 1:10){
      set.seed(123)
      ##########################
      ##########################
      boot.sample = sample(n, 1000, replace = TRUE)
      bootSample = dataset[boot.sample, ]
      bootSample

      split = sample.split(bootSample$blueWins, SplitRatio= 0.80)
      training = subset(bootSample, split == TRUE,  replace=TRUE)
      test = subset(bootSample, split == FALSE,  replace=TRUE)
      print(training)
}

But with this approach i think set.seed messes up everything, because it works with the same data every time. However i think the assingment wants me to use the same seed for every machine learning model.
I maybe overcomplicated the whole thing, i am new to R.
Hope someone can clear these things up.
Thanks


texts: Decent ROC, but horrible Precision-Recall curve

texts:
I was working on a model with following process:

Split to training/validation/test sets
Try a series of different models like GBM, RF, Logistic Regressions
Optimize hyper-params on them using GridSearchCV with roc_auc as scoring metric
I applied Oversampling methods and was careful that only training folds were oversampled (Imbalance is around 99:1)
Tested model on validation set

In this example, I'm showing the result of my Random Forest model with SMOTE oversampling, but the result holds for the other models as well. The AUC is decent at around 0.72 and the ROC looks like following:

However, the Precision-Recall curve looks horrible:

I'm not sure why I'm running into this issue, but would appreciate any insight or suggestions on how I could change my approach or refine it.


texts: How to compute the loss of seq2seq model's variable length output sequence against the target sequence?

texts:
To train a seq2seq model without teacher forcing, we will use the decoder to generate a sequence, compare it against the target sequence, and compute the loss. One such example can be found here:
But I am not sure how to correctly compute the loss of the generated variable-length sequence against the target sequence. The cited code above looks like this:
    # Without teacher forcing: use its own predictions as the next input
    for di in range(target_length):
        decoder_output, decoder_hidden, decoder_attention = decoder(
            decoder_input, decoder_hidden, encoder_outputs)
        topv, topi = decoder_output.topk(1)
        decoder_input = topi.squeeze().detach()  # detach from history as input

        loss += criterion(decoder_output, target_tensor[di])
        if decoder_input.item() == EOS_token:
            break

I think there are two problems:

When the generated sequence is shorter than the target sequence, it cuts off the loss calculation. In the extreme case, if the decoder generates an empty sentence , the total loss will simply be loss(, ). This creates a bias to generate shorter sequences.

When the generated sequence is longer than the target sequence, the decoder won't pay for the extra characters it generates

Now is it a fix just to divide the loss by the generated sequence length? I think it for sure is better, but the bias is probably not completely eliminated. This is because the sequence is generated by recursively feeding its own prediction of the most likely character into the model. Because the latter part of the generated sequence depends on the early part, if the early part is right, the latter part may be right or wrong, but if the early part is wrong, the latter part is almost guaranteed to be wrong. As the probability of being wrong is larger for the latter part of the sequence, there is still a tendency to generate shorter sequences.
My question is then -- how do we correctly handle the loss of the generated sequence against the target?


texts:
Disclaimer: although I know some things about big data and am currently learning some other things about machine learning, the specific area that I wish to study is vague, or at least appears vague to me now. I'll do my best to describe it, but this question could still be categorised as too vague or not really a question. Hopefully, I'll be able to reword it more precisely once I get a reaction.

So,

I have some experience with Hadoop and the Hadoop stack (gained via using CDH), and I'm reading a book about Mahout, which is a collection of machine learning libraries. I also think I know enough statistics to be able to comprehend the math behind the machine learning algorithms, and I have some experience with R.
My ultimate goal is making a setup that would make trading predictions and deal with financial data in real time.

I wonder if there're any materials that I can further read to help me understand ways of managing that problem; books, video tutorials and exercises with example datasets are all welcome.


texts: Sentiment Analysis for Q&A based reviews

texts:
I'm a self-learning ML enthusiast and I recently started learning NLP and performing Sentiment Analysis on imdb, yelp, amazon datasets(using Python). I came across a dataset where the reviews were in the form of Q&A's for each product.
1) What do you like about the product?
2) What do you dislike about the product?
3) What would you recommend to improve?
4) What did you use this product for?

I was wondering if Sentiment Analysis can be performed on this type of data or not? If yes, any tips/pointers/links to reading material in the right direction would be much appreciated. Also, there was no overall rating given for the product, so I stuck on that aspect as well.

-Luc.


texts: Interpretation of the output from qqPlot (using car library)

texts:
Basically, I have created a linear model and am testing to verify the normality of my errors. As a result, I have used the qqPlot function from the car library and have gotten the graph that can be seen below as my output.
Additionally 4 numbers were outputed (222, 160, 78, 113). My first question is, are these numbers meant to be outliers that are not consistent with the error terms being normally distributed? Secondly, what are the dashed lines that can be seen on the graph? Is this some sort of a limit where its acceptable for the error terms to be found that allow us to not question the error terms and treat them as normally distributed?


texts: How does sklearn KNeighborsClassifier compute class probabilites?

texts:
The KNeighborsClassifier has a method for predicting class probabilities. However, I cannot find any documentation describing how these probabilities are computed.

Here is a toy example that returns class probabilites:

from sklearn.neighbors import KNeighborsClassifier
import numpy as np
N = 100
np.random.seed(1)
X = np.random.random((N,2))
Y = np.random.randint(2, size=N)
model = KNeighborsClassifier(5, weights='distance')
model.fit(X, Y)
print(model.predict_proba([[0.5, 0.5]]))
#>>> [[0.55486525 0.44513475]]


texts: Dropout dividing by compensation term = overshoots the result?

texts:
When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons?

I understand that it's to prevent signal from dying out. But I've done the test, and found that it disproportionally magnifies the resulting state.

Assume the original state is $(0.1, 0.1, 0.2, 5.0)$
and our mask is $(0, 0, 1, 1)$ (with 50% of neurons that survive).

So, the original length is $$
\begin{align*}\sqrt{(0.1\times 0.1 + 0.1\times 0.1  + 0.2\times 0.2 + 5\times 5)} &= \sqrt{25.06}
\approx 5.006.\end{align*}$$

As for the masked vector, its length is $$\sqrt{0.2\times 0.2 + 5\times 5} = \sqrt{25.04} \approx 5.004.$$

Applying the compensation gives $5.004 / 0.5 = 10.008.$

This seems incorrect: my compensation just blew up the state vector. Perhaps we should be compensating differently - more carefully? I think it would even get worse if we mask the individual weights (like DropConnect does).

In my actual test, the state-vector of $192$ elements has length of  $0.885$ and the masked vector, with compensation has length of $1.305$


texts: Better approach for handwriting recognition?

texts:
I am trying to write an ANN in python for handwriting recognition by mouse movements. ( like identify characters we draw in paint app n convert it to text)

The question might seem that I haven't done my homework, but I am *not * looking for image datasets for handwritten characters.( at least that's what I've figured out so far)

Is there any database for such (array or co-ordinates corresponding to each character) inputs? If not, how can I create this database from existing ones like MNIST?

Currently, I have written a script to generate my own database because I could not find any. But it seems a very tedious task to generate a good database.
I am a beginner in ML. Could somebody point me in right direction upon how else can we do it?


texts: How to store complex tables and structures?

texts:
I'm wondering about general approaches to storing complex tables and structures. For example, imagine I have a table like this:

               A1 A2 A3 A4 A5 B1 B2 B3 B4 B5 C1 ....Z5
 individuals
 1             .  .  .  .  . .  .  .  .  .
 2             .  .  .  .  . .  .  .  .  .
 3             .  .  .  .  . .  .  .  .  .
 4             .  .  .  .  . .  .  .  .  .
 5             .  .  .  .  . .  .  .  .  .
 ...           .  .  .  .  . .  .  .  .  .

(I'm just showing the row and column names (variables), but the table would be filled, or could be even much larger, with A1 to A999, B1 to B999 colums.)

I can want to perform some calculation on all $Ax$ columns, or on all $x3$ ones. Since it can be difficult to deal with so many columns, it may be better to write this as:

 i  subcase  A B C ... Z
 1  1      .  .  .  .
 1  2      .  .  .  .
 1  3      .  .  .  .
 1  4      .  .  .  .
 1  5      .  .  .  .
 1  6      .  .  .  .
 1  7      .  .  .  .
 1  8      .  .  .  .
 1  9      .  .  .  .
 2  1      .  .  .  .
 2  2      .  .  .  .
 ...       .  .  .  .

I remember seeing that kind of structure several times, sometimes referred to as long format. Later, one can perform arbitrary operations on all rows meeting some condition on "case". This has some advantages, but could be confusing.

One could average or crate the differences of all the $\text{subcase}==1$ rows, only from $\text{individual}==1$, or when $\text{individual}==subcase$,  for example.

But what would be the best way to store the resulting numbers--columnwise or rowise? If I wanted to store the average of all $\text{subcase}==1$ rows, I would need to repeat the number on every row meeting that condition.

Another option would be to keep the wide format structure, but having 2 rows for the headers—one for the letters and a second one for the numbers. I think it could be a natural way, but I've never seen it before.

Are there other structures people can recommend for this problem, or for more complex situations, such as trees?


texts: Reducing noisy data from non normal distribution of data with std deviation?

texts: I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that data in both row vectors are NOT normally distributed. Is there any way that I used standard deviation for reducing noise from non normally distributed data. Any guidance will be appreciated. Thanks


texts: XGBoost for binary classification: choosing the right threshold

texts:
I am working on a highly-imbalanced binary-labeled dataset, where number of true labels is just 7% from the whole dataset. But some combination of features could yield higher than average number of ones in a subset.

E.g. we have the following dataset with a single feature (color):

  180 red samples —  0

  20 red samples — 1

  300 green samples — 0

  100 green samples — 1

We can build a simple decision tree:

                      (color)

                red /       \ green

 P(1 | red) = 0.1              P(1 | green) = 0.25

  P(1) = 0.2 for the overall dataset

If I run XGBoost on this dataset it can predict probabilities no larger that 0.25. Which means, that if I make a decision at 0.5 threshold:

0 - P < 0.5
1 - P >= 0.5

Then I will always get all samples labeled as zeroes. Hope that I clearly described the problem.

Now, on the initial dataset I am getting the following plot (threshold at x-axis):

Having maximum of f1_score at threshold = 0.1. Now I have two questions:

should I even use f1_score for a dataset of such a structure?
is it always reasonable to use 0.5 threshold for mapping probabilities to labels when using XGBoost for binary classification?

Update. I see that topic draws some interest. Below is the Python code to reproduce red/green experiment using XGBoost. It actually outputs the expected probabilities:

from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
import numpy as np

X0_0 = np.zeros(180) # red - 0
Y0_0 = np.zeros(180)

X0_1 = np.zeros(20) # red - 1
Y0_1 = np.ones(20)

X1_0 = np.ones(300) # green - 0
Y1_0 = np.zeros(300)

X1_1 = np.ones(100) # green  - 1
Y1_1 = np.ones(100)

X = np.concatenate((X0_0, X0_1, X1_0, Y1_1))
Y = np.concatenate((Y0_0, Y0_1, Y1_0, Y1_1))

# reshaping into 2-dim array
X = X.reshape(-1, 1)

import xgboost as xgb

xgb_dmat = xgb.DMatrix(X_train, label=y_train)

param = {'max_depth': 1,
         'eta': 0.01,
         'objective': 'binary:logistic',
         'eval_metric': 'error',
         'nthread': 4}

model = xgb.train(param, xg_mat, 400)

X0_sample = np.array([[0]])
X1_sample = np.array([[1]])

print('P(1 | red), predicted: ' + str(model.predict(xgb.DMatrix(X0_sample))))
print('P(1 | green), predicted: ' + str(model.predict(xgb.DMatrix(X1_sample))))

Output:

P(1 | red), predicted: [ 0.1073855]
P(1 | green), predicted: [ 0.24398108]


texts: Is there any work done on reconfigurable convolutional neural networks?

texts:
Convolutional Neural networks are used in supervised learning meaning models are always "set in stone" after training (architecture and paramters) so this might not even be possible, but is there any research done on playing around with the data paths, model size (number of layers) and architecture during runtime, i.e after training is done, for instance creating a model that can be modified online to use less or more recources, skip layers or use portions of the model.

There is some work done recently on creating flexible frameworks for training and designing networks, but that's always "offline". There is also an interesting paper on training two models, "big" and "little" for the same application and using an accuracy/power trade-off policy to deploy one of the two.


texts: Method for solving problem with variable number of predictors

texts:
I've been toying with this idea for a while. I think there is probably some method in the text mining literature, but I haven't come across anything just right...

What is/are some methods for tackling a problem where the number of variables it its self a variable.  This is not a missing data problem, but one where the nature of the problem fundamentally changes. Consider the following example:

Suppose I want to predict who will win a race, a simple multinomial classification problem. I have lots of past data on races, plenty to train on.  Lets further suppose I have observed each contestant run multiple races. The problem however is that the number or racers is variable. Sometimes there are only 2 racers, sometimes there are as many as 100 racers.

One solution might be to train a separate model for each number or racers, resulting in 99 models in this case, using any method I choose.  E.g. I could have 100 random forests.

Another solution might be to include an additional variable called 'number_of_contestants' and have input field for 100 racers and simply leave them blank when no racer is present.  Intuitively, it seems that this method would have difficulties predicting the outcome of a 100 contestant race if the number of racers follows a Poisson distribution (which I didn't originally specify in the problem, but I am saying it here).

Thoughts?


texts: What is the time complexity of learning/testing phase neural network?

texts:
What is the time complexity of feed-forward neural networks? I didn't find a book or a reliable source that's talks about this. This link provides one version by breaking the structure of a simple neural network. But I need someone to confirm this, a reviewer asked me to compute a the time complexity of my model. So, I need to be sure.
Also, if two models are trained in parallel, should I sum their time complexity or take the max?


texts: Balanced vs total dataset rows, which one is better?

texts:
I work on a dataset concerning games playing results. i.e every child play an indefinite number of games and it has as output (y) two possible values "success" or "Failure".    It's about 800 000 samples, with the total number of positive output (1) is ~~ 640 000  and the total number of negative output (0) is ~~ 160 000.

id     player     game    y
1      P1         G1      1
2      P1         G2      0
3      P1         G2      1
4      P1         G3      0
5      P2         G1      0
6      P2         G1      1
7      P2         G2      0

For making a prediction I sort my data by "players" first then by "games".

I want to make some tests for improving the prediction accuracy. As I am a beginner in machine learning and data Science I don't know what is better to choose:

working with this database as it is,  even it unbalanced? ??
or selecting just games that have (approximatively) balanced output? i.e to except any game result which has a total number of positive too begger than a total number of negative  ??


texts: Will it be more computationally expensive to have multipl 2d tensors or 1 3d tensor

texts:
Odd question but I am busy creating a Genetic Algorithm that optimizes the weights on a Neural Network instead of using good old fashion 1st-order optimization (Gradient/Adam)

What I have is x as a 2d vector (rows, columns)

x = tf.placeholder(tf.float32, shape=[None, total_input])

Since I need multiple neural networks instead of my hidden layer or output layer being 2d it needs to turn into a 3d vector.

fully_connected_1 = tf.Variable(tf.truncated_normal([size_in, size_out, total_networks], stddev=0.1), name="W")
acttivision1 = tf.nn.relu(tf.matmul(input_tensor, w) + b)

I can now either have my next layer into multiple 2d vectors:

for neural_networks in layer1:
    fully_connected_2 = tf.Variable(tf.truncated_normal([size_in, size_out], stddev=0.1), name="W")
    activison2 = tf.nn.relu(tf.matmul(acttivision1 , w) + b)

or keep the 3d fashion.

fully_connected_2 = tf.Variable(tf.truncated_normal([size_in, size_out,total_neural_networks], stddev=0.1), name="W")
activison2 = tf.nn.relu(tf.matmul(acttivision1 , w) + b)

As you have noticed the code is not 100% accurate as I am not even sure whether relu activision function will output a 3d vector.
I know it is going to be resource-intensive but the question is which one will be less computationally expensive. Also if you have a better idea please share. This is the only 2 solutions I can come up with.


texts: Can I treat text review analysis as a regression problem?

texts:
I am playing with a dataset that contains tripadvisor restaurant reviews and their labels (either 1, 2, 3, 4 or 5 stars).

Initially I was thinking of using it as a classification problem, applying softmax, cross-entropy and so on, however upon a second thought it came to me the idea to actually treat it as a regression problem, as generating an output which is continuous (such as 3.897) is perfectly fine (assuming of course that the original value would have been 4).

Does this make sense? or should I just hot-encode the answers in a vector of 5 elements (on per star?).

Also, assuming that regression is a correct way to treat this, linear activation in the last layer would be the way to go, and the metrics to look at would be the mean squared error and the mean absolute error.

UPDATE:

After browsing a bit more, I found something named Ordinal Regression, actually if one thinks a bit about it, everything makes sense. If i just treat the problem as a classification one, and I have a prediction saying that a given review has 1 star when in reality it was 4 star, how would we penalize that?

If we are using a classification problem and we are using cross entropy, that is simply a missclassification error, and labeling something as 1 start when it should have been a 4 start is consider as incorrect as labeling something with 3 stars when it should have had 4 stars. As you can see this is incorrect, both predictions (1 and 3 stars) are incorrect, but one is waaay more incorrect than others.
However if we are using a regression problem, our cost function will clearly penalize more a prediction of 1 start rather than one of 3 stars (again, here we assume that the correct label would have been 4 stars).

So given all that, I would say that regression makes sense in this case, as the categories are related to each other.


texts: Non-negative Matrix Factorization for clustering

texts:
I'm learning to user NMF to do clustering.
Based on the reading What is a good explanation of Non Negative Matrix Factorization? and https://iksinc.online/2016/03/21/what-is-nmf-and-what-can-you-do-with-it/.
The first link mention for data preprocessing, normalization is necessary.
My question is if we do a normalization for the features there will be negative values in the data. Isn't non-negative necessary for this method?
For NMF,https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html , it seems you still need to define n_component before hand. Does it depend on domain knowledge or any process to help to select? There is one constraint that n_component will be  far less than matrix A(m by n) m or n.


texts: similarity measures for categorical data in data mining

texts: I am working on my assignment in which i have to mention 5 similarity measures for categorical and continuous data in data mining. As a beginner I tried my best and found SQUARE DISTANCE,EUCLIDEAN AND MANHATTAN measures for continuous data.The point where i stuck is measures for categorical data. Could any one please tell me the names of similarity measures for categorical data.


texts: Is there a quantitative way to determine if a class of algorithms tends produce low bias or low variance models?

texts: I understand that some machine learning models tend to be low bias, whereas others tend to be low variance (source).  As an example, a linear regression will tend to have low variance error and high bias error.  In contrast, a decision tree will tend to have high variance error and low bias error.  Intuitively this makes sense because a decision tree is prone to overfitting the data, whereas a linear regression is not.  However, is there a more quantitative way to determine if a class of algorithms tends to produce low bias or low variance models?


texts: Is it Possible to Use Machine Learning for Ranking Alternatives?

texts: Right now, I’m working on road and street safety analysis. I have a dataset of dangerous points in four regions of a city. Some of the available variables  are road lighting status, ITS, latitude, longitude, longitudinal protection status, type of point control, as well as the number of injuries and deaths for the past two years. Also, there are some dummy variables(road lighting status, ITS, longitudinal protection status, type of point control). I want to know whether I could rank these points from the most dangerous one to the least dangerous one based on their potential danger using machine learning methods. If possible, which methods could help? I would appreciate it if you could share relevant links or references. Thanks in advance.


texts: Why don't companies use on-device ML to serve ads to protect privacy?

texts:
Eg: The ad inventory can be stored in the cloud. Using on device ML, a process then matches the ad to the user in order to serve the best ad to the right user.

Are there any limitations to this? (I am planning to make an app that serves ads using TFLite and CoreML so the user privacy could be protected.)
Can models in TFLite be updated with user data on-device, helping models stay relevant to user behavior without compromising privacy like Apple's CoreML?


texts: Which Machine Learning model to choose for n Number recommendations based of input numeric parameters

texts:
We want to develop a Machine Learning based recommendation model. Our data-set has about 25 numeric input parameters and an output parameter (Lender). Based on the input parameters we want machine learning model to recommend n number of best lenders.

Here is the Sample Data with few input attributes and LENDER (Output parameter).

M_Year  B_Age V_Score   C_Type  F_Amount    Terms   F_Product   V_Type      LENDER
2019    20      150     1       15000       12      1           1           Lender-A
2019    30      250     2       25000       36      3           3           Lender-B
2019    20      150     3       35000       24      2           2           Lender-C
2019    20      150     1       10000       48      5           4           Lender-D

Please advise which machine learning model can fulfill our requirement?
We want to use sklearns or AWS Sagemaker. But unable to decide which algorithm to use.


texts: How to sum values grouped by two columns in pandas


texts: Is there a limit in the number of layers for neural network?

texts:
I heard the neural network has a problem with vanishing gradient problems even though the ReLU activation function is used.

In ResNet(that has a connecting function for reducing the problem), there is limit of maximum layers of 120~190(I heard).

For the complete AI performance(or general AI with strong intelligence) I believe that the limit of the number of layers must be solved.

Is there is any possibility that we find a new activation function that does not limit the number of layers? (maybe we could use exhaustive search... checking the train performance in a neural network of 200~500 layers)


texts: How to perform platt scaling for hyperparameter-optimized model?

texts:
I'm using Python and have a best estimator from a grid search. Wanted to be able to calibrate the probability output accordingly, but would like to know more about implementing platt scaling. From what I've seen, calibration can either be done through a test-set or cross-validation, and consensus seems to be that latter option is better.

When it comes to CalibratedClassifierCV, there is an input for cv which can be an int, but can also have a string "prefit" suggesting that the classifier has already been fitted. However, it seems like using "prefit" does not end up with cross-validation being performed and using an int for cv would recalibrate the model by going through the grid search twice.

CalibratedClassifierCV(grid_CV_clf.best_estimator_, cv =2)

I can still fix my grid search with a KFold object for the GridSearchCV: cv argument to ensure model isn't recalibrated when doing something like above.

So to list the options I have:

I can fix my model and run crossvalidation for calibrated classifier. However, from example above, it will run GridSearchCV each time, so total number of steps will be (GridSearchCV:cv * CalibratedClassifierCV: cv)
I can use "prefit" and it'll calibrate training data onetime
I can simply calibrate one time with my test set while holding model as "prefit"

Would appreciate if someone could help me better understand what the right approach is or let me know of another option I may not have thought of.

Thanks


texts: Can I use regression to solve a multiple equation problem

texts:
I'm working on a problem which is a multiple equation. I have a group of people and each person in the group is working on different tasks (e.g. n tasks in total). Each person in this group is working on multiple tasks and complete them. I'd like to find an estimation for the time each type of task takes.

I have equations like below:

#of days person i worked = time(task1) * #task of type 1 completed + time(task2) * #task of type 2 completed +...+ time(task n) * #task of type n completed

The only unknown in these equations are time (task i).

I have one equation per person in the group (there are 100k people). Does it make sense to use regression to find time (task i)?


texts: How do I make an interactive PCA scatterplot in Python?

texts: The matplotlib library is very capable but lacks interactiveness, especially inside Jupyter Notebook.  I would like a good offline plotting tool like plot.ly.


texts:
I am reading the original paper by Chawla and others for SMOTE. I am trying to understand how to generate these synthetic examples for over-sampling the minority class. The paper says:

"Synthetic samples are generated in the following way: Take the difference between the feature vector (sample) under consideration and its nearest neighbor. Multiply this difference by a random number between 0 and 1, and add it to the feature vector under consideration. This causes the selection of a random point along the line segment between two specific features".

I understand the idea, take your sample, the nearest neighbor, pick a random point in between, what I don't understand is how these nearest neighbors are defined.


texts: help understanding nested cross validation

texts:
From what I read online, nested CV works as follows:

I divide my whole data in k folds, where k-1 folds are the train set and one fold is the test set.
There is the inner CV loop, where we may conduct a grid search using the train set to find the best hyperparameters for a model.
There is the outer CV loop, where we measure the performance of the model that won in the inner fold, on the test set.
We repeat the above procedure for different test and train sets until, at some point, all folds got his place as a test set once.

what I cannot understand is that because we find the hyperparameters in each outer loop we run, we might have models with different hyperparameters being tested in the test set in each loop, so can we use this nested cross validation to find the best hyperparameters for a model? or the goal of this evaluation is to find the best algorithm, like SVM or Naive Bayes, for a data set? because, if we are getting different hyperparameters in each loop, we cannot say which one is the best one.

English is not my first language, if it my text is hard to understand, please tell me, so I can fix it.


texts: Problem with Logistic regression using categorial variables

texts: I have a problem with Logistic regression using categorial variables. This my training datasets :


texts: How to convert Keras h5 to PyTorch pth format?

texts: I have trained a feature extractor in Keras and saved the weights as a h5 file. Now I want to load the same weights into the same model created and initialized in PyTorch for performance comparisons. Is there any way I can convert the h5 file to pth file so I can load that into the PyTorch model?


texts: mAP scores on tensorboard (Tensorflow Object Detection API) are all 0 even though the loss value is low

texts:
I trained a faster-rcnn model on the tensorflow object detection API on a custom dataset. I found that the loss is ~2 after 3.5k steps. However, when I ran eval.py, the mAP scores are all almost 0 as shown below.

I do not understand why this is the case. However, when I look at the images at 3.5k steps, the model has captured some of the boxes as shown below

Can someone please explain why the mAP scores are close to zero, even though the model has learned to output quite a few boxes?


texts: How to match a word from column and compare with other column in pandas dataframe

texts:
I have the below dataframe

Text                                Keywords       Type
It’s a roll-on tube                 roll-on        ball
It is barrel                        barrel         barr
An unknown shape                                   others
it’s a assembly                     assembly       assembly
it’s a sealing assembly             assembly       factory
its a roll-on double                roll-on        factory

I have first found out the keywords, and based on the keyword and its corresponding type, it should true or false

For example,
when the keyword is roll-on , the type should be "ball" or "others"
when the keyword is barrel, the type should be "barr" or "others"

Output

Text                                Keywords       Type         Result
It’s a roll-on tube                 roll-on        ball         True
It is barrel                        barrel         barr         True
An unknown shape                                   others       False
it’s a assembly                     assembly       assembly     True
it’s a sealing assembly             assembly       factory      True
its a roll-on double                roll-on        factory      False


texts: Neural network options for simple data classification

texts:
I want to clear about keras neural network options for classification of simple data where there are a number of features and one target column, as in iris flower dataset (Species is target):

SL,  SW,  PL,  PW,  Species
5.1, 3.5, 1.4, 0.2, setosa
4.9, 3.0, 1.4, 0.2, setosa
4.7, 3.2, 1.3, 0.2, setosa
...
...

I am finding that in almost all examples various combinations of Dense layers Dropout are the only options:

model = Sequential()
model.add(Dense(12, input_dim=4, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='softmax'))

What other keras layers can be used in such situations, especially if the data is large, say with 50K rows and 100 features?

Edit:  My specific question is whether Dense and Dropout are the only kind of layers for this purpose and such data?


texts: Predicting which drug is most appropriate for which patient returns an accuracy of almost 0

texts:
I have a dataframe that looks like this:
data = {'age': [54, 21, 7, 18],
        'sex': [0, 1, 1, 0],
        'disease_type': ['A', 'B', 'A', 'F'],
        'change_in_pain': [-0.54, -0.89, 0.07, -0.01],
        'drug': ['drug_1', 'drug_7', 'drug_1', 'drug_89'],
        }
df = pd.DataFrame(data)

=>
   age  sex disease_type  change_in_pain     drug
0   54    0            A           -0.54   drug_1
1   21    1            B           -0.89   drug_7
2    7    1            A            0.07   drug_1
3   18    0            F           -0.01  drug_19
...

The real df has > 10000 rows (=patients) and 34 different drugs but seemingly I cant upload a csv here for a more usable example?
I would like to train a model that predicts which drug is most effective for which patient given the patient’s age, sex, disease type and how much the pain was reduced (a more negative “change_in_pain” column is better).
In this simple example “drug_1” woud work for disease A only if the patient is older and female.
I wrote the following code but the mean accuracy is returned as almost 0 :
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

# shuffle
df = df.sample(frac=1.0).reset_index(drop=True)

X = df[['age', 'sex', 'disease_type', 'change_in_pain']]
y = df['drug']

# convert categorical variable into dummy/indicator variables.
X_OHE = pd.get_dummies(X)
y_OHE = pd.get_dummies(y)

X_train, X_test, y_train, y_test = train_test_split(X_OHE, y_OHE, test_size=0.20)

scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

knn = KNeighborsClassifier(5)
knn.fit(X_train, y_train)
score = knn.score(X_test, y_test)
print('mean accuracy: {:2.2f}'.format(score))

I also tested: RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1), KNeighborsClassifier(3), DecisionTreeClassifier(max_depth=5), MLPClassifier(alpha=1, max_iter=1000) but again the mean acc is around zero.
What am I doing wrong?
EDIT:
Doing it more slowly using:
knn.fit(X_train, y_train)  # X_train: 8000x11, y_train: 8000x34
y_pred = clf.predict(X_test)  # X_test: 2000x11, y_pred: 2000x34
acc = accuracy_score(y_test, y_pred)

shows that y_pred seems to contain only zeros - but why?


texts:
Can all types of ML methods benefit from bagging? Decision Tree Classification seems always be the go-to example of bagging, what about other classifiers or regressions?

When it's suitable to do bagging, how to pick the size and number of bags?

Update:

I am looking for something mathematically more rigorous, such as, for each model (single learner) we can break its total estimation error into:

$Error^2 = Bias^2 + Variance^2 + Irreducible^2$

If we can have a rough estimation of $Variance$ and the correlation between the predictions from all the single learners, then we can know about how low we can push variance to through ensemble.


texts: Why does this implementation of SimpleNet use 3x3 kernels on it's final layer for cifar10?

texts:
Question; I'm trying to implement simplenet in tensorflow and I have a question that I can't seem to answer myself. The implementation I'm basing this off of is here: https://github.com/Coderx7/SimpleNet_Pytorch/blob/master/models/simplenet.py

Now here's the thing; cifar10 data is 32 by 32. There are 5 max pooling layers which each half the feature map output size. That means that by the final convolutional layer the feature map is 1 by 1. Yet the final convolutional layer uses 3 by 3 kernels for some reason!

This doesn't make a whole lot of sense to me. Can anyone explain this? I'm really stumped. If the feature map is literally 1 by 1 what possible use could the network have for 3 by 3 kernels?

Thanks.

EDIT: Still haven't been able to figure it out. I really am stumped. If anyone has any input it'd be appreciated.

EDIT2: I still haven't figured it out. As far as I can tell this isn't the same implementation as the original one in cafee.


texts: What's the meaning of precomputed features?

texts: When i learn about deep learning, I found dataset with precomputed features form. Link (http://cs.stanford.edu/people/karpathy/deepimagesent/coco.zip). What's the different with usual dataset?


texts:
I'm working on an implementation of a paper and I haven't been able to find a description of average unpooling. I tried using max unpooling instead and my results were not great.

Thanks!


texts: Calculating the average of gradient decent

texts:
I am currently studying the backpropagation process and gradient decent algorithm form the book Neural Networks and Deep Learning written by Michael Nielsen and 3Blue1Brown channel in YouTube.

My question is about calculating the gradient in gradient decent algorithm(the whole dataset as input).

I have drawn a picture that shows my understanding about how the algorithm works:

For example we have 1 million hand written digit images and through the first iteration we feed the network with this 1 million images. Then the gradient is calculated for each images, summed together and averaged before updating the weights.

If my understanding is not wrong, this is the same thing that I saw in 3Blue1Brown channel. In this process the average of gradient is calculated with respect to the cost of each image and not the average cost of the whole data set(1 million) in one iteration, so the formula for calculating the cost has no effect here, rather its derivative is used for calculating the gradients for each image and we do not take the average of costs here.

First I want to know if this is a correct picture about how one iteration of calculating gradient decent works, second, why don't we take the derivatives of average costs with respect to the weights and biases and then take the average sum of all gradients?

And the last question, how the number of iterations and epochs can be assigned here? can we say number of epochs is always equal to the number of iteration since the whole data is used for each iteration?


texts: Why is xgboost so much faster than sklearn GradientBoostingClassifier?

texts:
I'm trying to train a gradient boosting model over 50k examples with 100 numeric features. XGBClassifier handles 500 trees within 43 seconds on my machine, while GradientBoostingClassifier handles only 10 trees(!) in 1 minutes and 2 seconds :( I didn't bother trying to grow 500 trees as it will take hours. I'm using the same learning_rate and max_depth settings, see below.

What makes XGBoost so much faster? Does it use some novel implementation for gradient boosting that sklearn guys do not know? Or is it "cutting corners" and growing shallower trees?

p.s. I'm aware of this discussion: https://www.kaggle.com/c/higgs-boson/forums/t/10335/xgboost-post-competition-survey but couldn't get the answer there...

XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,
gamma=0, learning_rate=0.05, max_delta_step=0, max_depth=10,
min_child_weight=1, missing=None, n_estimators=500, nthread=-1,
objective='binary:logistic', reg_alpha=0, reg_lambda=1,
scale_pos_weight=1, seed=0, silent=True, subsample=1)

GradientBoostingClassifier(init=None, learning_rate=0.05, loss='deviance',
max_depth=10, max_features=None, max_leaf_nodes=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, n_estimators=10,
presort='auto', random_state=None, subsample=1.0, verbose=0,
warm_start=False)


texts: Multilabel user classification based on multiple posts

texts:
I'm currently doing a course project for a course at university and I'm tasked on doing multilabel user profile classification based on their posts in social media, for each user there's a set of posts on social media which combined make up the user's occupation, gender etc.

In the past I've done simpler classification, where each post would have its own label and you would have to predict only the post's class, whereas in my current task I would have to somehow take into account all the posts of a user, since I only have the labels for the user, and not for each individual post.

My first idea would be to concatenate all the posts for each user and create a tf-idf vector which I would use for a classifier, but that way I would lose information from each individual tweet.

My question would simple concatenation suit my needs for this task, or are there more elaborate techniques which would take into account each individual tweet's importance?


texts: Reason for generally using RMSE instead of MSE in Linear Regression

texts: In linear regression, why we generally use RMSE instead of MSE? The rationale I know is that it's easy to minimize the error in RMSE instead of MSE by Gradient Descent, but I need to know the exact reason.


texts: What is the best way to pick the optimized configuration from this dataset?

texts: I have about 8000 configurations in an excel sheet. each configuration has four scores as seen in the image below. I would like to choose the best solution that has the highest lighting level score, lowest energy consumption score, the highest visual comfort score, the highest thermal comfort score, and the lowest CO2 level score. I'm new to data science but familiar with Python. I'm hoping that someone who is kind enough to explain a possible solution to rank these configurations from best to worst. Also, I'm interested to know the mathematical algorithm behind of the possible ranking system.


texts: Library to calculate Distance Metrics on GPU

texts:
Suggestions of widely used and fast GPU libraries that calculate the Euclidean distance of a vector from a matrix of vectors (of the same dimension) ?

Preferably one that works on NVIDIA GPUs.

Need it as a part of comparing solutions that evaluate Radial Basis Function Neural Networks.


texts: Independence/correlation test between features (not feautre-label)

texts:
I'm doing a Naive Bayes prediction model where I've 666 features to select from.

I've tried the SelectKBest chi2 test but it's a features-label test whereas what I'm trying to do is to find out the independency and also correlation feature-features. As the assumption to use Naive Bayes is that variables are all independent to each other, I have to remove features that are dependent/correlated to each other.

What are the other independence tests that I should look into apart from $chi^2$? (I have continuous features)


texts:
I have recently worked with a dataset of real estate transactions with missing entries for some features. For instance, GarageYrBlt (year when a garage was built) was sometimes null. Understanding what null means here could help me, I guess, decide what approach should I take to handle missing values for this column. Simple analyzis of the dataset has shown that it is always null, when GarageArea,  GarageCars are 0, which probably means no garage exists at all for a given estate:

GarageYrBlt GarageArea  GarageCars

        NaN          0           0
       1973         45           2

I was wondering if there is a generic approach to understand missing values semantics in a given dataset.

Some approaches I could think of:

Go through feature and find those with related names. For instance, all having the word Garage. Then filter samples where those features have 0, null or some negative number that doesn't make sense in this particular context. This doesn't scale well for large number of features.
Employ Apriori algorithm for finding associative rules like the one I found above:

{GarageArea ==0 AND GarageCars == 0} ==>  {GarageYrBlt == null}
or {GarageYrBlt == null} ==> {GarageArea ==0 AND GarageCars == 0}

(although I am not sure if Apriori supports finding the latter, which have multiple elements on the right side).

Anything else?


texts:
Is it true that we assume our P(y|x;theta) to follow Bernoulli's distribution given y has binary output in Logistic Regression? Is there any specific reason why we consider Bernoulli's distribution?

If 1) is true., What happens if we consider P(y|x; theta ) as Gaussian distribution? What would be our cost function? I mean if we assume Gaussian, we can have optimal value of mean and variance by taking mean of (y|x) and its variance which is what we are maximizing the likelihood for.


texts:
I have a task in hand where I get a dirty formed sentence and need to correct it. Examples are,   "StackOverflow best question answering platform" to be converted to "StackOverflow is best question answering platform"
"John published first paper 1990"  => "John published his first paper in 1990"
"John published first paper 1990"  => "When did John published his first paper"

I am thinking of using a Seq2Seq / EncoderDecoder technique just like NMT and train it on the input output pairs as training data.
Can someone who is experienced at this, validate if this approach will work? or there is a better approach available?


texts: How to predict past test set on time series data using LSTM

texts:
I'm trying to do a regression on some inventory amounts with the following model using Keras:

model = Sequential()
model.add(LSTM(100, batch_input_shape=(BATCH_SIZE, TIME_STEPS,
X_train_timeseries.shape[2]), dropout=0.0,
           recurrent_dropout=0.0, stateful=True,
kernel_initializer='random_uniform'))
model.add(Dropout(rate=0.5))
model.add(Dense(20, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
optimizer = optimizers.RMSprop(lr=LEARNING_RATE)
model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['mae'])

I have a time step of 60, how do I predict past my test data to make actual predictions into the future?


texts: How Does EAST detector implementation with VGG16 look? How many outputs does it have?

texts:
I was reading the Efficient and Accurate Scene Text Detector paper and saw the author reference VGG-16 as a possible stem "feature extractor" network. In the paper they say:

In our experiments, we also adopted the well-known VGG16 model, where the feature maps after pooling-2 and pooling 5 are extracted.

I'am trying to re-implement this paper on my own using VGG16, and was wondering if anyone knows(or can infer) how the VGG16 implementation would look like. The author provides this diagram representing how it was done with PVANet:

Each block takes the input from the previous hi. Does unpooling on it, concatenation with the corresponding feature extractor layer output map and a 1x1 followed by 3x3 convolution (as shown in the diagram).
This works out nicely because each block will double the sizes of the feature map while also halving the channels of it; this will pass the data along until there are 32 channels! And the author has chosen pvanet layers whose output feature map sizes increase by a factor of 2x:

Four levels of feature maps, denoted as fi are extracted from the stem, whose sizes are 1/32, 1/16, 1/8, 1/4.

Going back to VGG16 though, the selected feature maps actually change by a factor of 8x: 168/21 = 320/40 = 8. block 2 also has 128 channels, which is way more than the desired end goal of 32. Here is tf.keras's topless VGG16 (I have circled the output feature map shapes of block 2 and 5 pooling layers):

(sorry about the image size, you wouldnt be able to read the numbers otherwise. I selected an input image size divisible by 32 for demonstration)
So my questions are:

how would the feature merging branch look since the selected feature maps from the feature extractor increase by 8x?
I was thinking I could just repeatedly unpool block 5's output until it matches block 2's. Seems like a peculiar way of doing it?

How would I make sure there are a total of 32 channels by the end of the feature-merging branch?
I have no idea for this one

I've seen this: https://github.com/SakuraRiven/EAST, however this implementation is using 3 concatenations.


texts: Can we predict correlation between independent variables based on dependent variables

texts:
I have a correlation matrix for dependent var Vs independent variable as below:

Year   1       Sales
Sales -0.5453  1.0000
PriceIndex    -0.6089
Income        -0.5033
Interest      -0.3842

As we can see that all independent var have negative correlation with the dependent variable. However as one of the requirement that i have for my project is that based on these values can i predict correlation between my independent variables.

Kindly suggest if we can predict the same.
Thanks, Shivi


texts: Explainable AI and unsupervised algorithms

texts: There are several packages that allow explaining ML algorithms (Lime, Shap and so on). However, it is not clear how we can explain unsupervised algorithms for example, if we use PCA for dimensionality reductions first and random forest after, how it is possible to explain the whole process?


texts: Recommendations for Motif Based Classification of Time Series with Python

texts: I was wondering if I could get recommendations for Motif-based classification packages for time series data in Python? Currently I  modified a an old python sax package to work with Python 3.5 and feed into scikit learn's tfidf vectorizor to make a 'bag of patterns'. Which seems to work but I don't have results yet. There is also this package in R https://cran.r-project.org/web/packages/TSMining/TSMining.pdf. I appreciate any thoughts or insight. Thanks!


texts: is it reasonable to consider Horse Colic dataset as a binary classification problem?

texts:
I am learning Horse Colic dataset.

Well documented attributes; 368 instances with 28 attributes (continuous, discrete, and nominal); 30% missing values

there are 3 type of outcomes, 'lived', 'died', 'was euthanized'.

since 'was euthanized' is another form of 'died', is it reasonable to consider this problem as a binary classification problem?


texts: How can I do a sequence to sequence model (RNN / LSTM) with Keras with fixed length data?

texts:
What I'm trying to do seems so simple, but I can't find any examples online. First, I'm not working in language, so all of the embedding stuff adds needless complexity to my task.

I have input, in the form of a (1, 1000) vector. They are time-series data, so I'll have 10 of them in sequence. Which, if I understand tensors correctly, gives me something of shape (10, 1, 1000), right?

I want to pass this through an RNN/LSTM and the output should be also of the same shape (10, 1, 1000). Namely, 10 vectors of 1000 dimensions each.

Can anyone guide me?

Thank you kindly.


texts:
I have a data-set comprised of a fairly large number of columns (over 1000) relative to the number of rows (370) that I am currently running a random forest regression on. I am a little confused with respect to the best way to go around various tasks such as feature selection and cross-validation.

At present, I am doing the following:

Splitting the data-set 90/10, and running the RF on all features using default hyper-parameters
Using Grid Search to fine tune hyper-parameters and then running an 'optimized' model with these tunings
Using 5-fold cross validation to evaluate the model
Obtaining feature importance scores to indicate which features are performing best

Given the large number of features I have, and the fairly low performance of my models, I would like to find a smarter workflow.

It seems sensible to me to use the feature importance scores to filter out poor performers, but would I do this before or after cross validation? For instance, would I, after completing the above steps and removing features below a certain thrsehold, then run steps 1-3 again? or would I use feature selection before parameter tuning? Do I only cross validate once? Are there any other obvious steps I should integrate?

I appreciate this isn't an exact science, but if there are some industry standards that I'm either getting wrong, or have missed entirely, then I would much appreciate the feedback.


texts: How to select features for clustering to detect the number of different unique products in a search result?

texts:
I am trying to use clustering to determine the number of products in a search of products. So far I am using kmeans clustering. I have run into a problem where I cannot determine good features to use. Here are my clustering results for one search.

As you can see, there is clearly three different groups showing up at the moment. However, when you look at the top terms of each cluster, there is a clear issue. A lot of the terms are repeating in each cluster.

There is another issue, when I try to have the model predict which clusters a few test items are in, which are clearly different products, it says that they are in the same group. Here are the test terms:

Adjustable DC-DC LM 2596 Converter Buck Step Down Regulator Power Module LW SZUS
10pcs Mini360 3A DC Voltage Step Down Power Converter Buck Module 3.3V 5V 9V 12V
5V Out, 6V to 12V In AMS1117-5.0 5.0V Step-Down Linear Voltage Regulator Module
10pcs Mini LM2596s 3A DC to DC Buck Converter Power Supply Step Down Module

I have thought about using the grammar in each title as a feature, but as there is an 80 character limit for the titles, there rarely is any grammar. I am at a loss for where to go next in terms of trying to pick more features that would get me better results. I am doing all of this in python using scikitlearn and nltk.


texts: How to approach a new data set with no dependent variable

texts:
Basically I am taking a small course in data science.
And one of our first assignments are that we have gotten a pretty huge data set (hundred of thousands of samples, and roughly 150 independent variables). There is no response variable in this, but the assignment is pretty much: "Find something that are interesting". Which I obviously kind of broad.

My questions is: How would/should you/I approach this ?

Up until now I have just cleaned the data set as good as I can. Removed variables I don't think make much sense, made sure there are no NaN variables, and stuff like that. But what would be the next step in order to "find something useful" in a data set that may or may not contain anything interesting?


texts: Machine Learning Model for Time Series Forecasting

texts:
I am using Random Forest, SVM, and XGBoost models to nowcast/forecast an economic time series variable. However, I would like to extend these models to optimize/customize them for time series forecasting. Does anybody have experience in a similar situation, or is there a paper I could review?
Any recommendations are greatly appreciated!


texts:
I am a relatively new user to Hadoop (using version 2.4.1). I installed hadoop on my first node without a hitch, but I can't seem to get the Resource Manager to start on my second node.

I cleared up some "shared library" problems by adding this to yarn-env.sh and hadoop-env.sh:

  export HADOOP_HOME="/usr/local/hadoop"

  export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

I also added this to hadoop-env.sh:

  export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native

based on the advice of this post at horton works http://hortonworks.com/community/forums/topic/hdfs-tmp-dir-issue/

That cleared up all of my error messages; when I run /sbin/start-yarn.sh I get this:

  starting yarn daemons

  starting resourcemanager,
  logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-HdNode.out

  localhost: starting nodemanager,
  logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-HdNode.out

The only problem is, JPS says that the Resource Manager isn't running.

What's going on here?