texts: Incompatible shape error when trying to fit my model

texts:
I am using keras to make a CNN model. I have a dataset of images which consists of training and validation tfrecord files, the images in the tfrecords are grayscale. I load in the my 2 datasets by following the code from this kaggle notebook
def decode_image(image_data):
    image = tf.image.decode_jpeg(image_data, channels=1)
    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range
    image = tf.reshape(image, [75,75,1]) # explicit size needed for TPU
    return image

def read_labeled_tfrecord(example):
    LABELED_TFREC_FORMAT = {
        # tf.string means bytestring
        'height': tf.io.FixedLenFeature([], tf.int64),
        'width': tf.io.FixedLenFeature([], tf.int64),
        'channels': tf.io.FixedLenFeature([], tf.int64),
        "label": tf.io.FixedLenFeature([], tf.string),
        'class': tf.io.FixedLenFeature([], tf.int64),
        "raw_img": tf.io.FixedLenFeature([], tf.string),
    }
    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)
    image = decode_image(example['raw_img'])
    label = tf.cast(example['class'], tf.int32)
    return image, label # returns a dataset of (image, label) pairs

def load_dataset(filenames, ordered=False):
    # Read from TFRecords. For optimal performance, reading from multiple files at once and
    # disregarding data order. Order does not matter since we will be shuffling the data anyway.

    ignore_order = tf.data.Options()
    if not ordered:
        ignore_order.experimental_deterministic = False # disable order, increase speed

    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files
    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order
    dataset = dataset.map(read_labeled_tfrecord, num_parallel_calls=AUTO)
    # returns a dataset of (image, label) pairs
    return dataset

def get_training_dataset():
    dataset = load_dataset(tf.io.gfile.glob(GCS_DS_PATH + '/Data/train/*.tfrec'))
    dataset = dataset.repeat() # the training dataset must repeat for several epochs
    dataset = dataset.shuffle(2048)
    dataset = dataset.batch(BATCH_SIZE)
    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)
    return dataset

def get_validation_dataset():
    dataset = load_dataset(tf.io.gfile.glob(GCS_DS_PATH + '/Data/valid*.tfrec'), ordered=False)
    dataset = dataset.batch(BATCH_SIZE)
    dataset = dataset.cache()
    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)
    return dataset

training_dataset = get_training_dataset()
validation_dataset = get_validation_dataset()

After loading my data, I define a model and fit it like so:
model = Sequential([
        Conv2D(filters=64, kernel_size=(3, 3), activation='relu',
               padding='valid', input_shape=[75,75,1]),
        MaxPool2D(pool_size=(2, 2), strides=2),
        Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding='valid'),
        MaxPool2D(pool_size=(2, 2), strides=2),
        Flatten(),
        Dense(units=585, activation='softmax')
])

model.compile(optimizer=Adam(learning_rate=0.001), loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(
    training_dataset,
    steps_per_epoch=STEPS_PER_EPOCH,
    epochs=EPOCHS,
    validation_data=validation_dataset
)

But when I try to run my code, I am faced with the following error  ValueError: Shapes (None, 1) and (None, 585) are incompatible
How can I solve this issue?


texts: Tensor flow error - conversion and peformance

texts:
Here is my code:

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0,10,20)
train_x = np.array([[i] for i in x])
train_y = np.sin(x)

regressor = tf.contrib.learn.DNNRegressor(hidden_units=[10,20, 10])
regressor.fit(x = train_x, y = train_y, steps = 2000)
predictions = regressor.predict(x = train_x)

plt.plot(x, train_y)
plt.plot(x, predictions)
plt.show()

It generates the following error message/warning:

/usr/bin/python3.4 /home/ttt/Dropbox/Programming/Python/Tensor_flow/one_dim_case.py
/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/array_ops.py:1197:
VisibleDeprecationWarning: converting an array with ndim > 0 to an
index will result in an error in the future
  result_shape.insert(dim, 1)

Process finished with exit code 0

Could you help me to understand what is wrong?

An additional remark, with so many hidden layers and neurons and number of iterations, I really surprised with poor sin approximation.

P.S. This is a toy example to help to understand tensorflow


texts: Identify the parameter causing the anomaly in a multivariate dataset

texts:
I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like Isolation Forest, kNN, Autoencoders, and One-class SVM.

I am able to identify if a payment record is an anomaly or not but I am not able to pin-point the predictor variable that is causing the anomaly.

e.g.:

Account  ||  Currency  ||  Beneficiary  ||  Amount ||  isAnomaly(target)

I want to identify if, for an anomalous record, Currency variable is causing the anomaly or Amount variable is causing the anomaly.

I have gone through the below sources amongst many others but couldn't find anything helpful.

Anomaly Detection in Database

Anomaly Detection in multiple parameters

I have recently started my journey in data science and would be glad if someone could help me with this issue.


texts:
I have 3 classes with this distribution:

Class 0: 0.1169
Class 1: 0.7668
Class 2: 0.1163

And I am using xgboost for classification. I know that there is a parameter called scale_pos_weight.

But how is it handled for 'multiclass' case, and how can I properly set it?


texts: Meaning of variance in machine learning models

texts:
I know that high variance cause overfitting, and high variance is that the model is sensitive to outliers.

But can I say Variance is that when the predicted points are too prolonged lead to high variance (overfitting) and vice versa.


texts: Knn distance plot for determining eps of DBSCAN

texts:
I would like to use the knn distance plot to be able to figure out which eps value should I choose for the DBSCAN algorithm.
Based on this page:

  The idea is to calculate, the average of the distances of every point
  to its k nearest neighbors.  The value of k will be specified by the
  user and corresponds to MinPts. Next, these k-distances are plotted in
  an ascending order. The aim is to determine the “knee”, which
  corresponds to the optimal eps parameter.

Using python with numpy/sklearn, I have the following points, with the following distance for 6-knn:

X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
nbrs = NearestNeighbors(n_neighbors=len(X)).fit(X)
distances, indices = nbrs.kneighbors(X)

# Indices

[[0 1 2 3 4 5]
 [1 0 2 3 4 5]
 [2 1 0 3 4 5]
 [3 4 5 0 1 2]
 [4 3 5 0 1 2]
 [5 4 3 0 1 2]]

# Distances
[[ 0.          1.          2.23606798  2.82842712  3.60555128  5.        ]
[ 0.          1.          1.41421356  3.60555128  4.47213595  5.83095189]
[ 0.          1.41421356  2.23606798  5.          5.83095189  7.21110255]
[ 0.          1.          2.23606798  2.82842712  3.60555128  5.        ]
[ 0.          1.          1.41421356  3.60555128  4.47213595  5.83095189]
[ 0.          1.41421356  2.23606798  5.          5.83095189  7.21110255]]

then I computed the average distance:

distances.mean()
2.9269575028354495

The problem is I don't understand how exactly could I represent the same plot as them with distances in y-axis and number of points according to the distances on the x-axis using python.

Thank for your help.


texts: How to compare and interpret results of two neural networks models?

texts:
I made two models of neural networks(different architectures), the first one gave me as result:

Training Loss: 0.4195

Training Acc: 0.8325

Training Recall: 0.7266

Validation Loss: 0.4331

Validation Acc: 0.8483

Validation Recall: 0.7400

Test Acc: 0.7864

Test Recall: 0.7657

The second one:

Training Loss: 0.3354

Training Acc: 0.8583

Training Recall: 0.7791

Validation Loss: 0.3791

Validation Acc: 0.8493

Validation Recall: 0.7773

Test Acc: 0.8254

Test Recall: 0.7602

The problem there is that although the second model gives a better loss value, the first model gives a better result on the test set, so how can we interpret and compare these results? and does it have a way to test the stability of a model?
NB: the hyper parameters are the same for two models (batch size, learning rate, optimization algorithm) and the same validation and training sets


texts: How should i build a Knowledge Graph for a custom dataset?

texts:
I'm new to machine learning and i'm trying to create a small Knowledge Graph for search purposes similar to google for a class project.

Okay, so i have been searching on this topic for few days and this is what i have found from the web and research papers.

Create RDF triples or use already existing database like Freebase, Wikidata etc.
Then train model using some algorithm like ComplEx, TransE etc.
And finally use it for the queries.

My problem is i don't want to use an already existing database. I have a set of documents with me. Is there any good library for making triples from custom data?
Also after training my model, which database should i use to store my model and how to query it back for answers.

Thank you.


texts:
I have been working on a project which i took from kaggle. I didn't get the result as mentioned in the website. What am I doing wrong here?

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

train = pd.read_csv('../data/dont-overfit/train.csv')
test = pd.read_csv('../data/dont-overfit/test.csv')

train[train.columns[2:]].std().plot('hist')

train[train.columns[2:]].mean().plot('hist')
plt.title('Distribution of means/stds of all columns')

plt.show()

print(train.isnull().any().any())

print('Distribution of first 28 columns')
plt.figure(figsize=(26,24))
for i, col in enumerate(list(train)[2:30]):
    plt.subplot(7, 4, i+1)
    plt.hist(train[col])
    plt.title(col)

plt.show()

Here is the post


texts: Is there a dataset with news articles and their headlines?

texts: I need a set of news headlines and articles to help me in a project on automatic summarization. Is there such a dataset or something similar?


texts: Normalization vs standardization for image classification problem

texts:
For day and night image classification, is it better to normalize or standardize images? In general, when should I use each method? I am interested in with example why one method is preferred over another one?
Here, by normalization, I mean dividing pixel values by 255. Standardization means subtracting the mean value of pixels and then dividing by standard deviation. See the code samples below
# normalization
datagen = ImageDataGenerator(rescale=1.0/255.0)

or
# stardartization
datagen = ImageDataGenerator(samplewise_center=True, samplewise_std_normalization=True)


texts:
I am working on sales forecasting problem.I am able to provide data about which items got sold and not sold to the algorithm.How to provide algorithm information about items that are not present in the store.Is there any way we could encode this information in data or any other algorithms accepts this kind of information.Currently, I am using Neural Networks and Random Forest to forecast
Sales.


texts: Can I tune a model after training it? (Convolutional Neural Network & Classification)

texts:
I am relatively new to Data Science and I've recently embarked on a project. Long story short, I've trained a CNN model to distinguish between Male and Female genders. However, I wish to tune my model. I've saved the weights and bias of the existing model.

Can I use the current model with weights and bias to retrain on augmented data (from the same initial dataset)? I am guessing there are 2 possible outcomes, it tunes the model or it "resets" the weights and bias.

Can I perform hyperparameter tuning with RandomSearchCV or GridSearchCV using on my existing model? I believe this will tune the model.

My aim is to reduce overfitting.


texts: Reinforcement Learning - How are these state values in MRP calculated?

texts:
This is a question from the book an Introduction to RL, page 125, example 6.2.

The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov reward process (the image is copied form the book):

In the above MRP, all episodes start in the state C then can go either left
or right by one state in each step (with equal probability).

  Episodes terminate either on the extreme left or the extreme right. When an episode terminates on the right, a reward of +1 occurs; all other rewards are zero. For example, a typical episode might consist of the following state-and-reward sequence: C, 0,B, 0, C, 0,D, 0, E, 1.
  Because this task is undiscounted, the true value of each state is the probability of terminating on the right if starting from that state.

The book mentions V(A)=1/6, V(B)=2/6, V(C)=3/6 , V(D)=4/6 and V(E)=5/6; could you please help me understand how these values were calculated? Thanks.


texts: What is purpose of the [CLS] token and why is its encoding output important?

texts:
I am reading this article on how to use BERT by Jay Alammar and I understand things up until:

  For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we select that slice of the cube and discard everything else.

I have read this topic, but still have some questions:

Isn't the [CLS] token at the very beginning of each sentence? Why is that "we are only interested in BERT's output for the [CLS] token"? Can anyone help me get my head around this? Thanks!


texts: Neural Network Architecture to approximate simple function

texts:
What neural network architecture would approximate this function? This is monotonically decreasing non negative time series, that has non-stationary mean.

$$
q(t) = \frac{qi}{(1+b*D*t)^(\frac{1}{b})}.
$$

Currently, I have a function to generate a time series of $q(t)$ as a function of $ qi , b ,D $ at any given time $ t $ . I use this function to generate many sequences varying the parameters of $ qi , b ,D $ for a set of times. So my synthetic data looks like $ q(1),q(2)...q(n)$ I then split that data in a train period and test period of different lengths: $ [q(1)...q(m)][q(m+1)...q(n)]$  I am attempting to train a neural network on this synthetic data, so I have essentially unlimited training data.

I have tried MLP, LSTM, CONV-LSTM, but don't have a clear sense of how many layers/neurons are needed for this task. I have tried a few layers with 30-60 neurons in each. It is not clear how much synthetic data I need to generate also and over 100,000 in single model fit slows down training significantly.

Does anyone have any advice how to approach this problem?


texts:
I was going through the derivation of backpropagation algorithm provided in this document (adding just for reference). I have doubt at one specific point in this derivation. The derivation goes as follows:
Notation:

The subscript $k$ denotes the output layer
The subscript $j$ denotes the hidden layer
The subscript $i$ denotes the input layer
$w_{kj}$ denotes a weight from the hidden to the output layer
$w_{ji}$ denotes a weight from the input to the hidden layer
$a$ denotes an activation value
$t$ denotes a target value
$net$ denotes the net input

The total error in a network is given by the following equation
$$E=\frac12 \sum_{k}(t_k-a_k)^2$$
We want to adjust the network’s weights to reduce this overall error:
$$\Delta W \propto -\frac{\partial E}{\partial W}$$
We will begin at the output layer with a particular weight.
$$\Delta w_{kj} \propto -\frac{\partial E}{\partial w_{kj}}$$
However error is not directly a function of a weight. We expand this as follows.
$$\Delta w_{kj} = -\epsilon \frac{\partial E}{\partial a_k} \frac{\partial a_k}{\partial net_k} \frac{\partial net_k}{\partial w_{kj}}$$
Now the document individually calculates above three partial derivatives to get following final equation for weight change rule for a hidden to output weight [Section 4.4 in the attached document]:
$$\Delta w_{kj} = \epsilon (t_k-a_k)a_k(1-a_k)a_j$$
or, $$\Delta w_{kj} = \epsilon \delta_k a_j$$
I followed till above equation.
Now, the document continues the derivation as follows (I am adding exact wordings from the document) [Section 4.5 in the document attached]

Weight change rule for an input to hidden weight
Now we have to determine the appropriate weight change for an input to hidden weight. This is more complicated because it depends on the error at all of the nodes this weighted connection can lead to.
$$\Delta w_{ji} \propto -[\sum_{k}\frac{\partial E}{\partial a_k} \frac{\partial a_k}{\partial net_k} \frac{\partial net_k}{\partial a_j}]\frac{\partial a_j}{\partial net_j}\frac{\partial net_j}{\partial w_{ji}}$$

I couldn't follow that why there is this extra $\sum_{k}$ in above equation as we have already considered sum over all $k$ when we wrote error $E=\frac12 \sum_{k}(t_k-a_k)^2$
Any lights?


texts: What is the actual output of Principal Component Analysis?

texts:
I'm trying to understand PCA, but I don't have a machine learning background.  I come from software engineering, but the literature I've tried to read so far is hard for me to digest.
As far as I understand PCA, it will take a set of datapoints from an N dimensional space and translate them to an M dimensional space, where N > M.  I don't yet understand what the actual output of PCA is.
For example, take this 5 dimensional input data with values in the range [0,10):
// dimensions:
// a  b  c  d  e

[[ 4, 1, 2, 8, 8],      // component 1
 [ 3, 0, 2, 9, 8],
 [ 4, 0, 0, 9, 1],
 ...
 [ 7, 9, 1, 2, 3],      // component 2
 [ 9, 9, 0, 2, 7],
 [ 7, 8, 1, 0, 0]]

My assumption is that PCA could be used to reduce the data from 5 dimensions to, say, 1 dimension.
Data details:
There are two "components" in the data.

One component has mid a levels, low b and c levels, high d, and nondeterministic e levels.
The other component has high a and b levels, low c and d levels, and nondeterministic e levels.

This means that the two components are most differentiated by b and d, somewhat differentiated by a, and negligibly differentiated by c and e.
Outputs?
I'm making this up, but say the (non-normalized) linear combination with the highest differentiating power is something like
5*a + 10*b + 0*c + 10*d + 0*e

The above input data translated along that single axis is:
[[110],
 [105],
 [110],
 ...etc

Is that linear combination (or a vector describing it) the output of PCA?  Or is the output the actual reduced dataset?  Or something else entirely?


texts: Dictionary based statistical NER learner

texts:
If I have to paraphrase the current NER methodologies, it generally finds patterns in strings and creates its own "vocabulary", so to speak.

Naturally, it would perform like a charm with a mammoth dataset, curated carefully with hardwork and labeled entities.

But what if, the system is first introduced with a dictionary of named entities for respective categories and then given a sample literature or simple tweets for that category, for eg, and from them it "learns" how those named entites appear in a context.

The difference is subtle from regex based, in that in regex it would try to match strings and more the dictionary size, more the rules, then more its usefulness.

But in this system, it would actually learn how "eating an apple" and "eating at Apple" both can be accurately classified with little training set, the subtleness of grammar when mentioning about fruit and about a company.

Some intuition on its implementation in CRF++, CRFsuite, Stanford or any other.

General disclaimer
Not a data scientist. Just a passing thought from an ML enthusiast.


texts: Can we use difference-in-differences with a biased A/B test?

texts:
We noticed we had a biased sample in our A/B test and was wondering if difference-in-differences would help us make valid conclusions about the data, or if there was another way to proceed.

We ran an new experiment on our site, where we offered 50% of our users a new feature. We assigned users with odd ids into the experiment group and users with even ids into the control group and then ran the experiment. However, we saw that even prior to running the experiment, there was a statistically difference between the two groups. We think this is because we ran many experiments where we segmented based on odd/even of the id, so people in the experiment group have seen many treatments.

So lesson learned, we'll flip a coin next time instead of using the id. However, we'd like to see if we can still make inferences from the current experiment. I've heard of something called difference-in-differences. Would this work in this case, or is there a different approach that would work better? Ideally, we don't want to scrap the test and start over since 50% of our users have already seen the new feature.


texts: Question mark on Correlation Matrix with RapidMiner

texts:
I'm using RapidMiner to evaluate correlation between attribute in my dataset. The problem is that a lot of values appear with '?'.

Someone can help me?

This is a sample of data


texts:
I am running logistic regression on iris dataset.  I computed thetas and this is how I draw a decision boundary line.

x_values = ([min(X_train[:,0]), max(X_train[:,0])])
y_values = - (theta[0] + np.dot(theta[1], x_values)) / theta[2]
plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=plt.cm.Set1, edgecolor='k')
plt.plot(x_values, y_values )

I tried this, but the result is odd.

X= np.c_[ X_train, np.zeros(100) ]

theta = theta.reshape(3)

d=0

xx, yy = np.meshgrid(np.arange(np.min(X_reduced[:, 0]), np.max(X_reduced[:, 0])), np.arange(np.min(X_reduced[:, 1]), np.max(X_reduced[:, 1])))

z = (-theta[0] * xx - theta[1] * yy - d) * 1. / theta[2]

fig = plt.figure(1, figsize=(8, 6))
ax = Axes3D(fig, elev=-150, azim=110)

ax.scatter(X[:100, 0], X[:100, 1], X[:100, 2], c=y_train, cmap=plt.cm.Set1, edgecolor='k', s=40)
ax.plot_surface(xx, yy, z, alpha = 0.5)
plt.show()

I guess d in plane equation (ax+by+c*z = d) shouldn't be equal to 0. So I'm completely confused about this.


texts: Add variables/features to already trained tensorflow model when new variables come in. Possible?

texts:
Let's assume I've trained a deep-learning tensorflow model with some data with n variables. At a later time, new data come in and there is an additional variable alongside the previous variables with a total of n+1 variables. If I load the previously trained model that is used to seeing n variables, is there a way to continue training and include the new variable in the model without completely retraining it using all of the data?
One way that I thought of, is to train the model on the whole dataset (assuming we know all the possible variables that may come in at a later time), with the missing values for the unseen variables/features being replaced by 0. However, this adds a lot of sparsity to the dataset, but it works.


texts: Do I need a next for loop in order to get all values?

texts:
When i print this in the module without printing to file, it prints all values. But when I print it to the designated file, it only prints the first value.

#Create the model base
from sklearn.feature_selection import chi2
import numpy as np
N = 2
for UNSPSC, category_id in sorted(category_to_id.items()):
features_chi2 = chi2(features, labels == category_id)
indices = np.argsort(features_chi2[0])
feature_names = np.array(tfidf.get_feature_names())[indices]
unigrams = [v for v in feature_names if len(v.split(' ')) == 1]
bigrams = [v for v in feature_names if len(v.split(' ')) == 2]

with open('results.csv', 'w') as f:
 print("# '{}':".format(UNSPSC), file=f)
 print("  . Most correlated unigrams:\n. {}".format('\n. '.join(unigrams[-N:])), file=f)
 print("  . Most correlated bigrams:\n. {}".format('\n. '.join(bigrams[-N:])), file=f)

How can i adjust this so that all values get printed onto the file?


texts: How to obtain the word vectors optimally

texts:
I have a list of strings as shown

sent_list = ["Carrefour is in France", "Apple pie is delicious", "Amazon has just delivered", ...]

My code to get word embeddings below

import spacy

nlp = spacy.load("en_trf_bertbaseuncased_lg")

for sent in sent_list:
    print(nlp(sent).vector)

This takes considerable time when the list is of large size (>10000). I tried disabling sentencizer within the nlp pipe but with not much improvement. How can this be optimized for shorter run time?


texts: What AI model should I build to find out how similar are the 2 audio files of musical instruments like piano, guitar, synthesizer?

texts:
I have a idea for a project in AI, and I want to know how can I build such kind of AI models. I have good understanding of ML, and some beginner level understanding of Deep learning. What should I learn in order to build such projects?
I want to compare 2 audio files, of musical instruments being recorded. I want my model to tell, what differences are there in these 2 music/songs. Differences like in reverb, and delay, instruments played in wrong key, improper mixing/mastering, different waveforms in oscillators, improper automation, wrong ADSR values, etc.
One of these file will be upload by me, and other will be uploaded by the user/student. And then, my AI model will tell the student what should be improved in his/her music to be able to sound like mine.


texts:
I have dataset of gold prices and after modifying and some preprocessing i ended up with dataframe below:

There is 50,000 record in dataset and all columns expect date are int type and date is datetime object.
i need to predict price per unit in some specific dates. but somehow i baffled with so many methods.

My question is what data mining algorithm/method is results good prediction for this kind of data ?


texts: Which between random forest or extra tree is best in a unbalance dataset?

texts:
I have an unbalanced dataset, with 3 classes, with 60% of class 1, 38% of class 2, and 2% of class 3.
I don't want to generate more examples of class 3, and I cannot get more examples of class 3.
The problem is that I need to choose between RandomForest, and ExtraTree (this is homework), and explain why I choose one of these.
So I choose the Random Forest classifier, but I am not sure if my assumptions are right or no.
I choose that, because, the split of extra tree is random, so the probabilities of the pick some examples of class 3 are low, and because I think (this is the real question) that because Random is more high-variance than Extra tree, can be more useful because the high variance can help with the dataset is unbalance.
So are this two assumption especially the last one, correct? I choose correctly random forest over extra tree?
Thanks


texts: Can we specify the number of data generated(minority class) using SMOTE?

texts: I am trying to improve classification of imbalanced dataset creditcard fraud using SMOTE imbalanced_learn. But, in this it generates the data to 50%, can we give a specific number for the data to be generated? I want to track the classifier performance with the increase in generated data. Any help will be highly appreciated.


texts: Scaling of variables considering the values of a single column or the whole dataset

texts:
I read many time that for machine learning and data mining algorithms the multi-dimensional input data should be scaled (e.g. normalized or standardized). Now my question is whether the average, min or max value shall be calculated for each row (variable) or for the dataset as a whole.
Let's say we have 3 variables
X=[x_1, x_2,.., x_n]
Y=[y_1, y_2,...,y_n]
Z= [z_1, z_2,...,z_n]
I use the following code in python
def standardize (data, train_split):
    data_mean = data[:train_split].mean(axis=0)
    data_std = data[:train_split].std(axis=0)
    return (data - data_mean)/ data_std

def normalize(data, train_split):
    data_max= data[:train_split].max(axis=0)
    data_min = data[:train_split].min(axis=0)
    return (data - data_min) / (data_max -  data_min)

If I saw it correctly, the code (that I copied from a tutorial) calculated the min,max, average and standarddeviation for the whole dataset. Meaning that the maxium value is the maxium of X,Y and Z. Is this correct to do or shall one only normalize and standardize a variable by considering its own values? So the min, max etc. should be calculated for every variable.
I'd appreciate every comment.


texts: What does it mean if a high or low number of my componenets describe a percentage of the cumulative explained variance?

texts:
In the following code run after PCA i can see that X number of components explain Y % of cumulative explained variance (CEV).
I would like to know

1- What percentage of the CEV is typically acceptable e.g. 95% or 99%? (Or is it a case by case basis?)

2- If 20 out of 200 components explain 95% of the CEV, what does this say about my data, what about when 200 out of 200 explain 95% ?

pca = PCA().fit(X_train)
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of components')
plt.ylabel('cumulative explained variance');


texts:
I'm using a shallow CNN for my current project [this one]. I have a training dataset consisting of 1000 samples and a test dataset of 400 samples. I'm using the test dataset to choose the best hyperparameters for the NN (the ones which give the smallest loss). The problem is that, when I repeat the execution (with the same hyperparameters) a few times, I obtain a very different loss each run.

There are two different sources of randomness that affect each run:

The split of the training dataset in the different minibatches used for training
The weights initialization of the network

The architecture of the CNN is the following one: a conv layer consisting of only two filters, a max pool layer, a fully connected layer with 16 neurons and the output layer, with only one unit. The activation function is ReLu.

My hypothesis is that my model depends a lot on the weights initialization (I'm currently using Xavier Initialization). I have tried using Dropout to prevent this high variance, which has helped to some extent, and also tested different batch sizes for training. My next step will be introducing batch normalization.

What else should I do to solve this problem?


texts: What is/are the default filters used by Keras Convolution2d()?

texts:
I am pretty new to neural networks, but I understand linear algebra and the mathematics of convolution pretty decently.

I am trying to understand the example code I find in various places on the net for training a Keras convolutional NN with MNIST data to recognize digits. My expectation would be that when I create a convolutional layer, I would have to specify a filter or set of filters to apply to the input. But the three samples I have found all create a convolutional layer like this:

model.add(Convolution2D(nb_filter = 32, nb_row = 3, nb_col = 3,
                        border_mode='valid',
                        input_shape=input_shape))

This seems to be applying a total of 32 3x3 filters to the images processed by the CNN. But what are those filters? How would I describe them mathematically? The keras documentation is no help.

Thanks in advance,


texts:
I have a problem. I have a data set with some users and their ratings in several movies. The movies are separated into 19 genres.

I want to cluster the users by their preferences(ratings in the movies). The problem is, that I want to find a $threshold ( θ )$ to do the clustering, but I do not know how to do this, because the data are discrete and I cannot use the statistics methods that I know.
The threshold is the maximum distance that two users can have to be in the same cluster, like 2 users that likes the same genre movies or have little differences in their tastes.

I've tried to find a threshold using simple statistics. For example, for a user sum all of his ratings in a genre and divide the result via the number of ratings and find some means in some genres, but I didn't got an answer.

Note: I must use BSAS


texts: Adding Fourier transform features to data

texts: I'm working on some timeseries data which after visualising seems to be periodic(repeating at some interval), So I planned on finding the Fourier transform of the entire and pick the top n amplitudes and frequencies to add them as features to my existing data which has features such as mean averages, exponentially weighted averages and weighted averages. I had found the top 5 amplitudes and frequencies by following this video. Now I'm not sure how do I add this to my dataset as my dataset has roughly around 100K points. Should I add the same observation to all the 100K points?? I need some help to get around this problem. Thanks in advance


texts: extracting information in presence of noise

texts: Assume I have a vector of displacements, when I calculate the derivative I obtain velocity. The problem is, there are so many noisy points and I am looking for one number as a velocity. What do you think is the best technique to obtain the velocity ? I am taking derivative and then take the mean, this gives me good result when there is not much noise however this technique is not really robust to noise also taking the median is not giving good result. Can anyone suggest a better technique to handle this problem?


texts: Keras model to focus on different metrics?

texts: When using the model.compile() attribute, does it matter what metric I place in there? For example, would model.complie(metric='accuracy') focus on increasing accuracy vs model.compile(metrics=[tf.keras.metrics.Precision()]) focusing on Precision? Or do these just state what shows during the epoch training?


texts: Naive Bayes Multinomial, independence assumption misunderstood

texts:
This is embarrassing but I think I miss understand something.

In multinomial distribution, "while the trials are independent, their outcomes X are dependent because they must be summed to n." wiki
Naive Bayes assumption is that the features are independent (given the class): "assume that each feature $x_{i}$ is conditionally independent of every other feature" wiki

Thus, how can 1&2 co-exist in what's known as the Multinomial Naive-Bayes model?

Specifically, $p(x_i|c)=p_i^{x_i}(1-p_i)^{n-x_i}\binom{n}{k}$ and $p(x_1,...,x_n)=\frac{n!}{\prod_i{x_i}!}\prod{p_i^{x_i}}$ thus $p(x_1,...,x_n) \ne \prod_ip(x_i|c) $


texts:
I am trying to predict sales quantity of an item based on their attributes. Discount is one of those attributes. The problem is I am having different discounts in same period for same item .I  need to  consider sales for period wise i.e week  .While aggregating I am facing  problem

on differnt days item has different discounts.
Case1: Some items are having large variations in discounts like(0% and 50%)
Case2: some are having small variation like(40% and 49%).

-In case1 If we aggregate the discount and divide by sales quantity then there will be  decrease of  discount in one record and  increase in another.
- In case2 If we aggregate discounts and take average of that .It seems there is no large variations in discount by taking average.Which is not correct for first case.

In my data Sales is more sensitive to discounts.

Is there any way to get generalized discount to predict sales(with Random forest algorithm)?


texts: Who is actually sharing physical RAM in a distributed sytem that has virtual shared memory? (Server and/or clients.)

texts:
There is a business with about 100 computers used by employees, and one high-powered server.  It's called a "distributed system" by the system architect.  It uses Distributed Shared Memory (DSM).  There's also middleware, and the server is hosting Virtual Machines (VMs) which are running the applications that the employees see.

The question is: does the DSM come from physical memory that the server is sharing, creating virtual shared memory, or does the memory come from those 100 computers (or both)?  Aside from the possible speculation here, could portions of the physical memories of the 100 computers even be utilized as DSM in this setup?

Thanks

(Please note that distributed systems and shared memory are course topics in data science programs. That's why I'm posting this here. Data science isn't just data mining - it's a lot more.  The nuts and bolts behind doing it are included.)


texts: I am studying the performance, over 10 years, of high school students that enrolled in a school district. My Objective is to make inferences about factors leading to poor performance in school exams. I have data and stats on all those who were dropouts. Drop outs are a big group from among the poor performers. For example if there are 1000 students, 250 are poor performers. Of these 250, 200 are drop outs and there is no data on those. I plan to run a logistic regression model and draw inferences, but how do i remedy the imbalance that this missing data is creating?


texts: SMOTE-NC does not help to oversample my mixed continuous/categorical dataset

texts:
When I use SMOTE-NC to oversample three classes of a 4-class classification problem, the Prec, Recall, and F1 metrics for minority classes are still VERY low (~3%). I have 32 categorical and 30 continuous variables in my dataset. All the categorical variables have been converted to binary columns using one-hot encoding. Also, before going for the over-sampling process, I am imputing all missing values using Iterativeimputer.

Regarding the classifiers, I am using logistic regression, random forest and XGboost. May I have your thoughts on this? Any suggestions to over-sample a multiclass and highly imbalanced dataset?


texts: NameError 'np' is not defined after importing np_utils

texts:
I am running a MNIST example in a Jupyter notebook running in an Anaconda virtual environment. I have tried to run the code below (not yet finished, I was testing it) when it comes up with an error (can be seen below the code).
(X_train, y_train), (X_test, y_test) = mnist.load_data()

#print("X_train shape", X_train.shape)
#print ("y_train shape", y_train.shape)
#print("X_test shape", X_test.shape)
#print ("y_test shape", y_test.shape)

from keras.models import Sequential #imports the sequential model
from keras.layers import Dense, Dropout, Conv2D, MaxPool2D
from keras.utils import np_utils

#flattens the images from the 28x28 pixels to 1D 787 pixels
X_train = X_train.reshape (60000, 784) #flattening the image into 28x28 pixels, so into an array of 784.
X_test = X_test.reshape (10000, 784)
X_train = X_train.astype('float32') #using a 32-bit precision when training the neural network. This is most commonly used.
X_test = X_test.astype('float32')

X_train /=255 #Used for normalisation. Each image has a 'degree' of darkness within the range of 0-255 so you need to reduce that range to 0-1 for your Neural Network
X_test /=255

#one-hot encoding using keras' numpy-related utilities
n_classes=10
print ("Shape before one-hot encoding: ", y_train.shape)
Y_train = np.utils.to_categorical(y_train, n_classes)
Y_test = np.utils.to_categorical(y_test, n_classes)
print ("Shape after one-hot encoding:", Y_train.shape)

Error:
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-0063f366d5bd> in <module>
     22 n_classes=10
     23 print ("Shape before one-hot encoding: ", y_train.shape)
---> 24 Y_train = np.utils.to_categorical(y_train, n_classes)
     25 Y_test = np.utils.to_categorical(y_test, n_classes)
     26 print ("Shape after one-hot encoding:", Y_train.shape)

NameError: name 'np' is not defined

I guess I need to define np somewhere, but the practise code on the website I am using doesn't actually define np anywhere. Any suggestions?
Many thanks


texts: Pull Random Numbers from my Data (Python)

texts: Let's imagine I have a series of numbers that represents cash flows into some account over the past 30 days in some time window. This data is non-normal but it does represent some distribution. I would like to pull "new" numbers from this distribution in an effort to create a monte-carlo simulation based on the numerical data I have. How can I accomplish this? I've seen methods where you assume the data is normal & pull numbers based on some mean and standard deviation - but what about non-normal distributions? I'm using python so any reference including python or some python libraries would be appreciated.


texts: Change values of a particular column to value_count()

texts:
Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a particular column.

data = pd.DataFrame({'Item 1': ['apple', 'potato', 'cheese', 'banana', 'cheese', 'banana', 'cheese', 'potato', 'egg'],
          'Item 2': [1,5,2,7,8,4,9,0,3],
          'Item 3': ['a','b','a','c','d','c','e','e','e']})

How do i replace the values of column 'Item 1' by the value_counts()?

I tried the following code.

data.apply(lambda x: x.map(x.value_counts()))

But this is applying to whole data. I need to do it for one column.

The resulting value should be of the form -

'Cheese':3

'Potato':2


texts: Text Classifier with multiple bag-of-words

texts: I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself.  I've pre-processed the content column in such a way that the subject and associated metadata have been completely removed.  My thinking, at this point, is that I should train the classifier on two separate bag-of-words, one for the subject column and one for the content column.  Is this the right approach?  Is it there a simple way to accomplish this using NLTK or a similar library?


texts: Why do BERT classification do worse with longer sequence length?

texts:
I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts from PUBMED. The median number of tokens from pre-processing is about 350 but I'm finding a strange result as I vary the sequence length. While using too few tokens hampers BERT in a predictable way, BERT doesn't do better with more tokens. It looks like the optimal number of tokens is about 128 and consistently performs worse as I give it more of the abstract.

What could be causing this, and how can I investigate it further?


texts: Potential speedup by applying PCA once on dataset with m rows vs. IncrementalPCA to x batches of size m/x?

texts:
I've been working on trying to perform dimensionality reduction on high-dimensional, high-volume datasets (with many rows and columns - around 100,000 - 1M rows and ~500 columns).

While the size of my dataset (m being the # of rows and n being the # of columns) doesn't prevent PCA from applying, PCA can take a fair bit of time to train and transform the data.

I was wondering whether splitting up the dataset into x batches (each of equal size m/x rows) and then partially fitting the IncrementalPCA() against each of those x batches one after the other would be faster than trying to fit a traditional PCA() against the entire dataset of size m rows?

I believe the runtime of PCA is O(m * n2) + O(n3), so effectively IncrementalPCA in this case would simply be dividing the inner factor with m/x and multiplying both terms outside by x, therefore leading to a higher overall time complexity and slower performance, with overall x * O(m/x * n2) + x * O(n3) = O(m * n2) + x * O(n3). So, naively it seems that this idea would not work.

However, could someone provide more analysis on whether such a speedup using mini-batches with IncrementalPCA is nonetheless feasible? I suspect my reasoning may be incomplete.

An alternative approach I've been trying out as well is random stratified sampling, which seems more standard, but I'm trying to combine various optimizations for achieving faster runtimes.


texts:
What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build, ...

My guess is the Data Science ones give more structure around the common ML operations and are better optimized for compute and memory needed to train, deploy, doing things in parallel and run inference on models?


texts: Calculating key and value vector in the Transformer's decoder block

texts:
I am implementing the transformer model in Pytorch by following Jay Alammar's post
and the implementation here.

My question is regarding the input to the decoder layer.
As shown in the diagram above, the encoder_decoder_attention layer in the decoder get the queries vector from the self-attention layer below, and keys and values vector from the encoder.
Here is the implementation of the decoder from above post:
class DecoderLayer(nn.Module):
        # ...
        self.self_attention = MultiHeadAttentionLayer(hid_dim, n_heads, dropout, device)
        self.encoder_attention = MultiHeadAttentionLayer(hid_dim, n_heads, dropout, device)
        #...

    def forward(self, trg, enc_src, trg_mask, src_mask):

        #trg = [batch size, trg len, hid dim]
        #enc_src = [batch size, src len, hid dim]
        #trg_mask = [batch size, trg len]
        #src_mask = [batch size, src len]

        #self attention
        _trg, _ = self.self_attention(trg, trg, trg, trg_mask)

        #encoder decoder attention
        _trg, attention = self.encoder_attention(trg, enc_src, enc_src, src_mask)

And the MultiHeadAttentionLayer looks like this:
class MultiHeadAttentionLayer(nn.Module):
    def __init__(self, hid_dim, n_heads, dropout, device):

        # ...

        self.fc_q = nn.Linear(hid_dim, hid_dim)
        self.fc_k = nn.Linear(hid_dim, hid_dim)
        self.fc_v = nn.Linear(hid_dim, hid_dim)

    def forward(self, query, key, value, mask = None):

        batch_size = query.shape[0]

        #query = [batch size, query len, hid dim]
        #key = [batch size, key len, hid dim]
        #value = [batch size, value len, hid dim]

        Q = self.fc_q(query)
        K = self.fc_k(key)
        V = self.fc_v(value)

        #Q = [batch size, query len, hid dim]
        #K = [batch size, key len, hid dim]
        #V = [batch size, value len, hid dim]

        Q = Q.view(batch_size, -1, self.n_heads, self.head_dim).permute(0, 2, 1, 3)
        K = K.view(batch_size, -1, self.n_heads, self.head_dim).permute(0, 2, 1, 3)
        V = V.view(batch_size, -1, self.n_heads, self.head_dim).permute(0, 2, 1, 3)

        # ...

What got me confused is as Jay mentions in his post:

we create a Query vector, a Key vector, and a Value vector. These vectors are created by multiplying the embedding by three matrices(Wq, Wk and Wv) that we trained during the training process.

Visually from the diagram, it looks like the key and the value vector themselves from the encoder is passed to the decoder. If so should't we be using the same weight matrices (Wq, Wk and Wv) from the encoder, in the decoder's encoder_decoder_attention layer as well ?
The code implementation has it's own Linear layers (with own weights) to calculate these vectors, and obtains the vectors by multiplying the encoder_outputs.


texts: How can I appropriately handle cleaning of gender data?

texts:
I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it.
In this dataset, the gender field is a string that could have had anything entered into it. While cleaning most entries are fairly straightforward (“f”, “F”, “female”, “cis female” and “woman” can all be coded to “F”), what I was wondering about was how to properly handle trans or queer identities (e.g. an entry that says something like “trans female” or “queer/she/they”).
Should I create a new code for trans entries for each gender, or should I just code them as though they were members of the gender they identify as?
Should I just drop them from the dataset entirely, because they might distort it? I remember reading that trans individuals suffer from much higher rates of mental illness than cis individuals.
Are there any best practices that I should follow in this regard?


texts: Neural network approach to the cocktail party effect

texts:
Imagine you have 2 people at 2 different microphones but in the same room. Each microphone is going to pick up some sound from the other person. Is there a good neural network based approach to isolating the signals so that the sound from each microphone only captures 1 person?

I remember hearing a solution to this a few years back, but Im not sure if I remember that correctly

I ask because a similar problem was mentioned to me today. During EEG brain wave data collection, each electrode can pick up signal from multiple sources in the brain. In that world they try to isolate the sources and reduce the "noise" from other brain areas, and its common to use ICA for such a task. The problem with ICA is that the post-processing stage is very time consuming, so I'm wondering if theres a better ANN/DNN approach that could solve the problem more efficiently, or maybe with better accuracy


texts: Random forest: Should OOB accuracy equals Test Set accuracy?

texts:
In random forests, we bootstrap a sample from the training set, and train a decision tree on the bootstrapped sample. Some observations are not drawn from the training set, and not used to train the decision tree: these are out-of-bag (OOB) observations that we can use to compute the performances of the model.
For random forests, I thought that the out-of-bag performance should be the same (or at least very similar) to the performance calculated on a separated
test set.
But this does not seem to be the case.
In the following R code, the accuracy computed on the out-of-bag sample is 77.81%, while the one computed on a separated test set is 81%.
What I am doing wrong?
library(randomForest)
library(ISLR)

Carseats$High <- ifelse(Carseats$Sales<=8,"No","Yes")
Carseats$High <- as.factor(Carseats$High)

train = sample(1:nrow(Carseats), 200)

rf = randomForest(High~.-Sales,
                  data=Carseats,
                  subset=train,
                  mtry=6,
                  importance=T)

acc <- (rf$confusion[1,1] + rf$confusion[2,2]) / sum(rf$confusion)
print(paste0("Accuracy OOB: ", round(acc*100,2), "%"))

yhat <- predict(rf, newdata=Carseats[-train,])
y <- Carseats[-train,]$High
conftest <- table(y, yhat)
acctest <- (conftest[1,1] + conftest[2,2]) / sum(conftest)
print(paste0("Accuracy test set: ", round(acctest*100,2), "%"))


texts: What simple machine learning algorithm to use in POS tagging

texts:
I am new to NLP. I have a dataset that has already a parts of speech included, the only problem is what algorithm to use in order to train my dataset. Sample:

{"word1": "POS",
"word2": "POS",
"word3": "POS",
"word4": "POS"}

Ambiguity is not a problem in this case.
If I input a word, it should tell what POS it is, but if the word is unknown, the POS will also be unknown.


texts: How do I make inference about test metrics for entire population from sample metrics?

texts:
Generally we calculate specific metrics for ML models on a test set (and we try to make that test set representative). I'm not clear on how to make inference about the same metrics for the population that the test set is representing - i.e., say I want to answer: if the model were to run on the whole population, what's the confidence interval of metric in question at (e.g.) 95% significance level?
Now for a simple case I can try to use my basic stats knowledge: suppose I have a binary classification model and I'm interested in reporting its precision.

I measure the precision on the test set and define the test statistic as the sample proportion $\hat p$ of correctly classified examples out of total examples
I also run the model on different folds of data to get precision on each fold, and then calculate the standard deviation of those different sample (fold) precision values - call it $\bar\sigma$: this is my proxy for the standard deviation of the sampling proportion distribution.
ALTERNATIVELY, I can measure the standard deviation for each fold $i$ as $\sigma_i=\sqrt{np_i(1-p_i)}$ where $p_i$ is the precision measured in that fold. I'm assuming Binomial distribution with sequence size $n$ and probability of "success" (correct prediction) as the precision $p_i$. Then I take the average of all these $\sigma_i$ to get an estimate of the "population standard deviation" and then divide that by $\sqrt{n}$. i.e. If the number of folds I considered was $k$, then $\bar \sigma=\Sigma_{j=1}^k\sigma_j/(k\sqrt{n})$
Using either of the methods in 2 or 3 to calculate $\bar\sigma$, we estimate the population precision as $\hat p\pm 1.96\bar\sigma$

Or I could just calculate the interval as (assuming test set size $m$)$$\hat p\pm t_{m,95\%}.\sqrt{\frac{\hat p(1-\hat p)}{m}}$$
where $t_{m,95\%}$ is the t-distribution value corresponding to 95% significance level and sample size $m$.
But what about other metrics like precision-recall combo, mean absolute percentage error, mean absolute error, RMSE, etc. etc.? Obviously I'm not expecting a recipe for each metric, but just a general idea on how we go about getting interval estimates for arbitrary metrics. Also, does the methodology described above seem correct?


texts: scikit-learn classification report's f1 accuracy?

texts:
When I run scikit-learn classification_report() on my 2-class y and yhat, I get the following:
              precision    recall  f1-score   support

           0       0.84      0.97      0.90    160319
           1       0.67      0.27      0.38     41010

    accuracy                           0.82    201329   <--- WHAT?
   macro avg       0.75      0.62      0.64    201329
weighted avg       0.80      0.82      0.79    201329

I understand F1, macro-averaged F1, etc. But what is the accuracy value of 0.82 under the f1-score column?
I already read other questions about this function:
How to interpret classification report of scikit-learn?
Sklearn classification report is not printing the micro avg score for multi class classification model
macro average and weighted average meaning in classification_report


texts: I'm interested in doing a mapping or inventory of all data science for social good and technology for social good organizations/initivaives. Is anyone aware of any similar efforts or places to start?


texts:
Can Gridsearchcv params perform worst than default RF?
RF with default values performs rmse_train=4886,r^2_train=0.84, rmse_test=11008,r^2_test=0.22.
RF after GridSearchCV tuning performs worst on train set (rmse_train=9104,r^2_train=0.45, rmse_test=11091,r^2_test=0.21). This is the code (my first ML algorithm implementation)
#data
features = pd.read_csv("dati_nn.csv")
labels = np.array(features['Cost_damage']) #regression problem
features = features.drop('Cost_damage', axis=1)
features = np.array(features)
train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=0.2, random_state = 123)

#RF default
rf = RandomForestRegressor(random_state = 123)
rf.fit(train_features, train_labels)
pred_train = rf.predict(train_features)
predictions = rf.predict(test_features)
# compute rmse and r_score
rmse_train = np.sqrt(metrics.mean_squared_error(train_labels, pred_train))
rmse_test = np.sqrt(metrics.mean_squared_error(test_labels, predictions))
r2_train = r2_score(train_labels, pred_train)
r2_test = r2_score(test_labels, predictions)

#GridSearch
param_grid = {
    'bootstrap': [True],
    'max_depth': [20, 30],
    'max_features': ['auto', 'sqrt'],
    'min_samples_leaf': [1, 2, 3],
    'min_samples_split': [5, 8, 10, 12],
    'n_estimators': [500, 600, 700, 800, 1000]
}
rf = RandomForestRegressor(random_state = 123)
grid_search = GridSearchCV(estimator = rf,
                           param_grid = param_grid,
                           scoring = 'neg_mean_squared_error',
                           cv = 10,
                           n_jobs = -1,
                           verbose = 2
                          )
grid_search.fit(train_features, train_labels)

best_grid = grid_search.best_estimator_
pred_train = best_grid.predict(train_features)
predictions = best_grid.predict(test_features)
#rmse and r2_score as above

In my opinion, the code is correct but I don't understand why rmse grows.
I searched for many days on the internet to find an answer...no success. I tried to use RandomizedSearchCV, same problem.
Any suggest? Is my workflow correct?
Thanks in advance!!


texts:
I do not understand this about GANs.

Apparently the Generator is supposed to receive a latent space vector as its input. Yet I couldn't find an example of how I can implement it in Pytorch. This is a problem for me, because different posts suggest different approaches.

Is it simply an image of Gaussian noise which is then served as an input to Generator's Convolutional Neural Network or is it a one-dimensional array passed through a Fully-Connected layer to the Generator?


texts: Reasoning for temporal difference update rule

texts:
In TD(0) learning where the value function is given by $V(s) = w^T\phi(s)$ where $w$ is a weight vector and $\phi$ is a feature map, the weight update is given by $w_{t+1} = w_{t} + \eta\delta_{t+1}\phi(s_{t})$, where $\eta$ is the learning rate and $\delta$ is the temporal difference error. The temporal difference error is given by $\delta_{t+1} = r_{t}+\gamma V_{t+1} -V_{t}$, where $r$ is the reward and $\gamma$ is the discount factor.  Note that the weight update is proportional to the old state features. This update rule can be thought of as trying to make $V_{t}$ closer to $r_t +\gamma V_{t+1}$ to make the value function more self consistent.

However, it is also possible to make the value of $\gamma V_{t+1}$ closer to $V_{t}-r_{t}$ by making the weight update rule $w_{t+1} = w_{t} - \frac{\eta}{\gamma}\delta_{t+1}\phi(s_{t+1})$. It is also possible to do something in the middle like $w_{t+1} = w_{t} + \frac{\eta}{2}\delta_{t+1}(\phi(s_{t})-\frac{1}{\gamma}\phi(s_{t+1}))$.

Are these viable weight update rules? If not, why? What properties do the learning algorithms corresponding to these update rules have?


texts: Why is the decoder not a part of BERT architecture?

texts: I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT architecture without using a decoder? How does it do away with decoders completely?


texts: Will the job of Data Science is going to be at risk?

texts:
Data Science is the sexiest job of 21st Century according to the different sources with a high payment salary . But it seems that there will be over production of this profession . According to U.S. Bureau of Labor Statistics there will be about 11.6 million jobs in 2026 but looking at the current trend there may be over production of data science professional till 2026 which may cause unemployment of Data science professionals .

According to a research , the data science syllabus is harder than non Data Science syllabus in engineering.

The skills required for a Data Scientist are:

Computer Science Skill
Mathematics (Calculus , Linear Algebra , Statistics and Probability , Optimization , Numerical Analysis , etc)
Domain Specific Skills

The above 3 points looks simpler but if we start go for mastery it takes a lot of energy and time.

I am 19 year old from a undeveloped country seeking as a professional in Data science but the above mentioned statement makes me to think whether to choose a Data Science as a profession or not ? If I am wrong then it will be my pleasure to give some guidance to a 19 year old boy who completed his 10th grade and passionated in machine learning and Data Science as a profession .

Waiting for your open minded reply  and stay safe in global pandemic☺


texts: When I saw the two results of applying convolution filter and correlation filter, the results have the same distribution and are just flipped. Why is convolution filter used instead of correlation filter in CNN?


texts: How to use Adaptive Rejection Sampling to Update Alpha in iGMM

texts:
I am wondering how can I use ARS (adaptive rejection sampling) in dirichlet process mixture model (infinite Gaussian mixture model) as described by Rasmussen (https://www.seas.harvard.edu/courses/cs281/papers/rasmussen-1999a.pdf)

Using equation (15), I have introduced the log and derive the derivative of the log of equation 15. However, I can't figure out what's the next step.


texts:
I have the Reuters RCV2 multilingual text analysis data set. I am trying to figure out the hierarchy between the topic classes. For example, in RCV1 we know that C11, C12, C13 .. C15 all are the subcategories under CCAT etc.

However I could not find such an information for RCV2 corpora. I know the classes on RCV2 are not exactly the same with RCV1 corpora. I wonder is there a way to find the list of classes in RCV2 and the hierarchical relationship between these classes?


texts: Training network with variable frame rate?

texts:
I would like to train a temporal network, but the video data available are in different frame rates(ex 7,12,15,30). How should I train this network, without down-sampling higher frame rate videos.

I tried up-sampling everything, but there is some artifacts generated.

What is the suitable approach?


texts: Feature addition/ subtraction and SVM model accuracy

texts:
I am working on a text classification problem where I would like to improve the accuracy of my model. Presently, I am using SVM with linear SVC and OneVsRestClassifier. The model should correctly predict all of the subcategories for a parent category.

For entered account name, I should get the right description for my test data file that I will use to test the model later.

If I add another feature, +1 or -1, for debit and credit to my Account_Name field, will there any improvement in the accuracy?

Or, is there anything else that will help improve the accuracy of the model?

It is currently at 74% which is low.


texts: is there a way to check if i got a "good price" on something?

texts:
I'm looking at some data. Actually, the Boston Housing dataset is probably a good proxy for it: https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html

I'm wondering if there's a way to predict if I got a "good price" given certain conditions. So something like, if I'm given a tuple such as CRIM, ZN, INDUS = (0.006320,18, 2.31), then is a house price of 50 significantly higher or lower than expected?

This isn't quite vanilla anomaly detection, because the combination of a particular CRIM, ZN, INDUS may be rare and I don't want to flag it unless price is out-of-spec for that. I guess the question is given a combination of (CRIM, ZN, INDUS) (no matter how anomalous that is), is the combination of (CRIM, ZN, INDUS, Price) rare? So, it's like a naiive bayes' problem, but that only works with categorical outputs


texts:
I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have found many tools with which I can analyse the data where each tool has its own advantages and disadvantages. Big data being really new to me, I find it very difficult to choose a platform to start with. There should be other people here who should have dealt with similar situations.

What criteria should I mainly take into account before selecting a
tool for analysing big data?

Some of the Criteria that I have taken into account : Visualization, Interaction, Security, Data Access and integration, Speed of response, Integrated Data Mining, Pattern Matching, Ease of use etc. As you can see the list that I have made for the criteria comes from the extensive reading of different articles on the topic. But I can't narrow down the list nor find the individual contribution of these criteria in the various tools available for analysis.

Let me also list some of the tools that I found after googling : Knime, Statistica 2, Rapidminer, Orange, WEKA, KEEL, R and RATTLE.

On what basis could I choose a tool from a list of tools that perform similar tasks?

UPDATE based on Comment

Aim : To develop a software that analyses the data coming from the wind mills and generate reports. The software should be able to predict when a wind mill can fail based on the analysis.

The project is still in the phase of gathering User Requirements. Maybe i am so early to come into conclusions about what tool should be used.

Someone else suggested that I should be finalise the requirements and then think about a tool that can help me get things done. So is it possible that I find what and how things should be analysed before finding a tool? And is it also possible that I find an algorithm for predictive analysis without knowing what would be the results of the tool after analysis.


texts: What is the best way to use word2vec for bilingual text similarity?

texts:
I face a problem where I need to compute similarities over bilingual (English and French) texts. The "database" looks like this:

+-+-+-+
| |F|E|
+-+-+-+
|1|X|X|
+-+-+-+
|2| |X|
+-+-+-+
|3|X| |
+-+-+-+
|4|X| |
+-+-+-+
|5| |X|
+-+-+-+
|6|X|X|
+-+-+-+
|7|X| |
+-+-+-+

which means that I have English and French texts (variable long single sentences) for each "item" with either in both version (in this case the versions are loose translations of each other) or only in one language.

The task is to find the closest item ID for any incoming new sentence irrespective the actual language of either of the sentence in the "database" or of the incoming sentence (that is, the matching sentence in the "database" needn't necessarily be in the same language as the incoming sentence as long as the meaning is the closest). I hope this goal explanation is clear.

Originally I planned to build a word2vec from scratch for both languages (the vocabulary is quite specific so I would have preferred my own word2vec) and find similarities only for the corresponding language for each new sentence but this would omit all candidates from the items where the corresponding language sentences are missing.

So I wonder if generating a common word2vec encoding for the combined corpus is viable (the word2vec method itself being language agnostic) but I cannot figure out if such a solution would be superior.

Additionally, the number of the sentences is not very large (about 10.000) maybe word2vec generation from scratch is not the best idea on one hand, but there are really specific terms in the corpora on the other hand.


texts:
I'm doing a project where a Python script used a convolutional neural network to determine if a plant is healthy, and then water it based on that. While training the CNN, it seems to get up to 100% accuracy quite early, although it isn't accurate. I only have a little less than 2000 images, and was wondering if I didn't have enough, or it was my model, which is here
        self.model = Sequential()
        self.model.add(Conv2D(numFilters, filterSize, activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 3)))
        self.model.add(Conv2D(numFilters * 2, (3, 3), activation='relu'))
        self.model.add(MaxPooling2D(pool_size=poolSize))
        self.model.add(Dropout(0.25))
        self.model.add(Flatten())
        self.model.add(Dense(numFilters * 4, activation='relu'))
        self.model.add(Dropout(0.5))
        self.model.add(Dense(2, activation='softmax'))
        self.model.compile(loss='categorical_crossentropy',
                      optimizer = 'adam',
                      metrics=['accuracy'])

I would just like to know the reason why it doesn't train well.


texts: Algorithm selection rationale (Random Forest vs Logistic Regression vs SVM)

texts:
I want to understand the criteria of selection of ML algorithms, i.e., what are the guidelines on which algorithm to be selected in which case?
The reasons I know are:

Logistic regression to be picked in case we want to advise the impact on y variable on what changes on any x variable.
Random forest works good on mixed data and very effective for categorical data. Also, it does feature selection first (so dimension reduction is not needed).
Random forest not to be picked with high featured and multiple category data due to its high processing time.
SVM works well with the closely placed data points like in image processing identification of dog vs cat.

But these are not sufficient enough to pick anyone, as I don't have any reason for why which algorithm not to be picked, like when to choose SVM over Logistic regression or RF over Logistic regression.
The only rationale I have is the performance, so I run all algorithms and who ever performs best that I select (but this is not right way).


texts: Issue while predicting multiple values which possess different order of magnitude (regression)

texts:
I have a system that take electric signal and output parameters (regression). However I run into an issue. The parameters I want to predict are not on the same magnitude. I would like also to use only one neural networks if possible.

Parameter 1:
Mean    = 12.53673
Minimum = 10.00461
Maximum = 14.98899

Parameter 2:
Mean    = 148656955394038
Minimum = 75029133522564
Maximum = 224934092847235

Parameter 3:
Mean    = 1.475720134278e+17
Minimum = 7.506184799345e+16
Maximum = 2.249190781380e+17

...

If I use any loss function such as RMSE, MSE, MAE, MSLE the losses of the big parameters takeover all the rest. The network therefore randomly guess all the smaller parameters

I am searching a good way to find those parameters using one network. I tried using "mean absolute percentage error". But the learning is exceptionally slow, and it often get stuck into extremely low value and extremely high value.

I am maybe thinking about creating one loss per target. Freeze all the network except the output layer and each epoch let one of the loss train the whole network. But I am not sure it is a good idea.

Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.

Moreover In the future I will have an issue similar except I will need to predict series that possesses different order of magnitude (and not only some distinct value). So I hope being able to reuse the technique I am learning

Any help would be very appreciated!
Thanks :)

EDIT:
For those who happens to have the same issue, this is the code I went with (keras)

class Model:
    def __init__(self, shape_features, shape_targets, mean_targets, std_targets):
        self.shape_features = shape_features
        self.shape_targets = shape_targets
        self.mean_targets = mean_targets
        self.std_targets = std_targets

    def build(self):
        inputs = Input(shape=self.shape_features, name='Input')
        x = Conv1D(filters=16, kernel_size=3, activation='relu', name='Convolution1D_1')(inputs)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_1')(x)
        x = Conv1D(filters=32, kernel_size=3, activation='relu', name='Convolution1D_2')(x)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_2')(x)
        x = Conv1D(filters=64, kernel_size=3, activation='relu', name='Convolution1D_3')(x)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_3')(x)
        x = Flatten(name='Flatten')(x)
        x = Dense(128, activation='relu', name='Dense_1')(x)
        x = Dense(64, activation='relu', name='Dense_2')(x)
        x = Dropout(0.2, name='Dropout')(x)
        outputs = Dense(self.shape_targets, activation='linear')(x)
        model = tf.keras.Model(inputs=inputs, outputs=outputs)

        def de_normalizing(tensor):
            return tensor * self.std_targets + self.mean_targets

        predictions = Lambda(de_normalizing)(outputs)
        model_prediction = tf.keras.Model(inputs=inputs, outputs=predictions)

        def custom_loss(y_true, y_pred):
            y_true = (y_true-self.mean_targets)/self.std_targets
            return K.mean(K.square(y_pred - y_true), axis=-1)

        return model, model_prediction, custom_loss

One model is used for training, the other is used for predictions


texts: Closed form of Weighted Ordinary Least Squares calculation of the trend line

texts:
I would like to know if there is a closed form version of this equation:

$\beta = \frac{n\sum{xy}-\sum{x}\sum{y}}{n\sum{x^2}-(\sum{x})^2}$

But for weighted data, where the weight $w_i$ is the value of the importance I want the individual $i$ in my database to have.


texts:
My dataset is a simple table of 20 columns and 100,000 rows.It is not a image data as commonly used in CNN. What input shape should I provide in this case?

Right now I did-

input_shape = (21,109713,1)

model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),
                 activation='relu',
                 input_shape=input_shape))

which gives the error-

ValueError: rng_mrg cpu-implementation does not support more than (2**31 -1) samples


texts: Multivariate LSTM RNN DNN returning multiple features for forecasting a time series in Python

texts:
I am using the latest installation of Keras with Python 3.6 on Linux Mint with a NVIDIA (NVDA) 2070 GPU.
I am looking up. How to get the return values of my data? How do I use all of the features, and then get a return for each one, 1 time step into the future. Also, is there a way to only get certain features. Let's say.

Date Open High Low Close 50sma Volume

How would I return only the Open High Low and Close for the next day?

I have read many past questions and tutorials, but they are not fully intact. They are usually missing parts of the code or authors or columnists (pun not intended) won't answer. (Yes, I understand. There is the most serious pandemic going on.) So, thanks ahead of time for your time. Also, if there is specific reference. I would love to see it.
I have looked over Keras and Tensorflow Documentation, too. Maybe, I just don't get it, and could someone please point out specifically. What I am missing?


texts:
I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ACE corpus. I have searched on the web a lot but I could not find any way that can be used to train NLTK's NER.

If anyone can provide me with any link/article/blog etc which can direct me to Training Datasets Format used in training NLTK's NER so I can prepare my Datasets on that particular format. And if I am directed to any link/article/blog etc which can help me TRAIN NLTK's NER for my own data.

This is a question widely searched and least answered. Might be helpful for someone in the future whose working with NER.


texts: neural network with large numbers of inputs

texts:
I learn about neural networks with a heavy excercise. I have products and want to forecast the revenue. I have 10 features. But 4 features  have large numbers of expressions. So my "group" features has over 500 different groups (of products). If I do binarization with the features I end up with vectors of over 1000 dimensions as input.
All the product classes are names with letters and numbers such as "100XA9". Now there are 500 of those. For a neural network I have to do binarization with those names. Is that right?
Then my input layer is 1000, right?

My question is this "normal"? And is this problematic for computation?

I mean this is a typical question. A store with many products (like over 500) will forecast its revenue based on the prices and other features.

I hope you can help me to untestand how to use neural networks for classification.


texts:
I am new in data science so please accept my apology in advance if my question sounds stupid. I want to do a time series forecast of outage mins in the current regulatory year. The regulatory year starts from 1 April and ends on 30 March of next year. I have data of around six months i.e. from April to September. Outage does not occur every day. So I have only 144 data points (or days out of 171 days) where the outage occurred. I have plotted the data in the following graph. The graph shows the cumulative sum of outage mins.

Now I am trying to predict the value from October to March. I wanted to forecast the value that what would be the cumulative outage mins by the end of March next year. I tried to use Exponential smoothing but it did not work, it may be because I don't have a lot of observation. Then I was reading about ARIMA but not sure whether its the right algorithm to use or not as I don't think that there would be any seasonality in this scenario and also I don't have long data points. Could anyone help with which algorithm should I use to forecast the value? I am using Python as a programming language. Any help would be really appreciated.


texts: How to Interpret Multinomial Specification in R's `mnlogit` package

texts:
The mnlogit package in R allows for the fast estimation of multinomial logit models.

The specification of forumlas is a bit different from most other regression models/packages in R, however.

Using the Fish dataset as a reproducible example,

> require(mnlogit)
Loading required package: mnlogit

Package: mnlogit
Version: 1.1.1
Multinomial Logit Choice Models.
Scientific Computing Group, Sentrana Inc, 2013.

> data(Fish, package ='mnlogit')
> head(Fish)
           mode   income     alt   price  catch chid
1.beach   FALSE 7083.332   beach 157.930 0.0678    1
1.boat    FALSE 7083.332    boat 157.930 0.2601    1
1.charter  TRUE 7083.332 charter 182.930 0.5391    1
1.pier    FALSE 7083.332    pier 157.930 0.0503    1
2.beach   FALSE 1250.000   beach  15.114 0.1049    2
2.boat    FALSE 1250.000    boat  10.534 0.1574    2

I'm trying to understand the difference between the model specification of

fm <- formula(mode ~ 0 + price | income | catch)

and

fm <- formula(mode ~ 0 + price | income + catch)

while the documentation covers the detail of such changes  in the general coeffcient area of the forumla (i.e. where price is), I don't see an explanation of how operators like + affect the alternative-specific area of the formula/code, relative to |.


texts: feature importance via random forest and linear regression are different

texts:
Applied Lasso to rank the features and got the following results:

rank feature prob.
==================================
1       a     0.1825477951589229
2       b     0.07858498115577893
3       c     0.07041793111843796

Note that the data set has 3 labels. The ranking of the features for the different labels are same.

Then applied random forest to the same data set:

rank feature score
===================================
1       b     0.17504808300002753
6       a     0.05132699243632827
8       c     0.041690685195283385

Notice that ranking are very different from the one produced by Lasso.

How to interpret the difference? Does it imply the underlying model is inherently nonlinear?


texts: Two problems with the Association Rules widget

texts: I am using the heart disease prediction dataset from UCI but analyzing it with association rules. The target class is set as "diameter narrowing" which means the presence or absence of coronary artery disease. I first used the discretize widget (set on equal width) which is connected to the Association Rules widget. When I open the widget the check box "Induce only classification rules" is not highlighted so it can't be selected. Also, when I attempt to use filter by antecedent or filter by consequent (example = diameter narrowing=1) I get no matches. I tried it with multiple different combinations of minimum support and confidence


texts: Get data from intermediate layers in a Pytorch model

texts:
I was trying to implement SRGAN in PyTorch and I have to write a Content loss function that required me to fetch activations from intermediate layers for both the Generated Image & Original Image.
I'm using pretrained VGG-19 and according to the paper I need the ReLU activations
Can anybody guide me on how can I achieve this?


texts: What ML/DL approach better suits this problem?

texts:
We have a huge dataset with us that looks like below

Factor -|- ... -|- Rank1 -|- Rank2 -|- Calls
A11-----|- ... -|-0.1234--|-3.2345--|- Cat A
A11-----|- ... -|-1.1234--|-0.2345--|- Cat B
A12-----|- ... -|-2.1234--|-3.2345--|- Cat C
A12-----|- ... -|-2.1234--|-3.2345--|- Cat C
...
A13-----|- ... -|-0.1234--|-3.2345--|- Cat A
A13-----|- ... -|-3.1234--|-0.1345--|- Cat B
A13-----|- ... -|-2.1234--|-2.2345--|- Cat C
A14-----|- ... -|-4.1234--|-4.2345--|- Cat C

and we have about 10 million of such data points.
We also have a test set with about .2 million data points where we need to accurately call them out into different classes.

At this point, we are trying a mix of K-means & Random Forest approach coded in (python2.7-sklearn) which gives us about 90% accuracy at classification but we wanna reach for more.

I am interested to apply some kind of deep learning approach to this and that's why I'm learning about TensorFlow. But every link I've gone though in deep learning talks (tf documentation + youtube videos) about CNN and image recognition (MNIST .. etc) only and I'm not finding any idea how to start solving this.

I'm looking for your suggestions and guidance on how to approach this kind of problem / What neurons to stack or How to build models for this kind of data? I'm willing to burn my share of the midnight oil on any links and suggestions offered on this but I need to know if I'm even thinking right by trying to solve it with DL or if there's any other approach for this kind of data which may work better, or to solve this what should I study or learn?

Edit 1:
So, I managed to go through the TensorFlow documentation for MNIST an perhaps I can see some correlation.

In the sense if I pass the rank1 & rank2 as input np.array for the 2 neuron input layer and each call as a one_hot np.array, where Cat A = [1,0,0],Cat B = [0,1,0],Cat C = [0,0,1] and put 3 neuron on the outermost layer each predicting one Cat(category) through a softmax function, will that work?

Even if so, what should I use as internal/hidden layers? Can I use the other factors as well in the input layers? Do I need to convert them into numeric(int/float)? Should I pass these rank 1 and rank 2 values to the input layer just as it is or should I engineer them anyway beforehand?


texts: Why is the posterior chosen to be normal in variational autoencoder?

texts: Is there any reason for choosing the posterior $q(z|x)$ as normal distribution in variational autoencoder? or is it just for convenience?


texts: Building a tag-based recommendation engine given a set of user tags?

texts: Basically, the idea is to have users following tags on the site, so each users has a set of tags they are following. And then there is a document collection where each document in the collection has a Title, Description, and a set of tags which are of relevance to the topic being discussed in the document as determined by the author. What is the best way of recommending documents to the user, given the information we have, which would also take into consideration the semantic relevance of the title and description of a document to the user's tags, whether that be a word embeddings solution or a tf-idf solution, or a mix, do tell. I still don't know what I'm going to do about tag synonyms, it might have to be a collaborative effort like on stackoverflow, but if there is a solution to this or a pseudo-solution, and I'm writing this in C# using the Lucene.NET library, if that is of any relevance to you.


texts: Can overfitting occur even with validation loss still dropping?

texts:
I have a convolutional + LSTM model in Keras, similar to this (ref 1), that I am using for a Kaggle contest. Architecture is shown below. I have trained it on my labeled set of 11000 samples (two classes, initial prevalence is ~9:1, so I upsampled the 1's to about a 1/1 ratio) for 50 epochs with 20% validation split.I was getting blatant overfitting for a while but I thought it got it under control with noise and dropout layers.

Model looked like it was training wonderfully, at the end scored 91% on the entirety of the training set, but upon testing on the test data set, absolute garbage.

Notice: the validation accuracy is higher than the training accuracy. This is the opposite of "typical" overfitting.

My intuition is, given the small-ish validation split, the model is still managing to fit too strongly to the input set and losing generalization. The other clue is that val_acc is greater than acc, that seems fishy. Is that the most likely scenario here?

If this is overfitting, would increasing the validation split mitigate this at all, or am I going to run into the same issue, since on average, each sample will see half the total epochs still?

The model:

Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
convolution1d_19 (Convolution1D) (None, None, 64)      8256        convolution1d_input_16[0][0]
____________________________________________________________________________________________________
maxpooling1d_18 (MaxPooling1D)   (None, None, 64)      0           convolution1d_19[0][0]
____________________________________________________________________________________________________
batchnormalization_8 (BatchNormal(None, None, 64)      128         maxpooling1d_18[0][0]
____________________________________________________________________________________________________
gaussiannoise_5 (GaussianNoise)  (None, None, 64)      0           batchnormalization_8[0][0]
____________________________________________________________________________________________________
lstm_16 (LSTM)                   (None, 64)            33024       gaussiannoise_5[0][0]
____________________________________________________________________________________________________
dropout_9 (Dropout)              (None, 64)            0           lstm_16[0][0]
____________________________________________________________________________________________________
batchnormalization_9 (BatchNormal(None, 64)            128         dropout_9[0][0]
____________________________________________________________________________________________________
dense_23 (Dense)                 (None, 64)            4160        batchnormalization_9[0][0]
____________________________________________________________________________________________________
dropout_10 (Dropout)             (None, 64)            0           dense_23[0][0]
____________________________________________________________________________________________________
dense_24 (Dense)                 (None, 2)             130         dropout_10[0][0]
====================================================================================================
Total params: 45826

Here is the call to fit the model (class weight is typically around 1:1 since I upsampled the input):

class_weight= {0:1./(1-ones_rate), 1:1./ones_rate} # automatically balance based on class occurence
m2.fit(X_train, y_train, nb_epoch=50, batch_size=64, shuffle=True, class_weight=class_weight, validation_split=0.2 )

SE has some silly rule that I can post no more than 2 links until my score is higher, so here is the example in case you are interested:
Ref 1: machinelearningmastery DOT com SLASH sequence-classification-lstm-recurrent-neural-networks-python-keras


texts: How to optimize client's portafolio with analytical models?

texts:
I have a model in which we want to optimize the probability of an outcome depending on a election of some product (a personalized product for every client amongst three posibilities). The product is the second product elected after the same client has a first.

I have never done an optimization of a probability with analytical models before so I am a little blind in the topic.

What I have for now is this:

I have a dataset with clients with ONE product, the optimization is intended for THEM, they have a first product and I want to model which is the next best product for every one of them - Application dataset.
I have another dataset with clients with TWO OR MORE products, the training could be done with them (I know which product was purchased first) - Training dataset.
Both datasets have exogenous variables which could help modelling: i.e. Age, gender, etc.

I have thought these posible ways to achieve the objective:

Split the training dataset depending on the first purchased product, then obtain a personalized probability on the basis the first product is purchased, the "sign" and coefficient of the second possible products would tell me which is the next best one for every first product (subset).
Model the dataset without separating anything, read the results looking for optimal combinations.
Model with interactions first-second products to look for the best combinations.
Model with white-box techniques (decision tres, logistic regression), so the combinations are traceable.
Something different.


texts:
Why when I concatenate DataFrames my code shows this error?
I trained a model to detect and classify 6 classes, and then store the coordinates and probabilities of these classes in a DataFrame, but I also want to save all the probabilities of all classes, not just the maximum probability, I wrote this Function (below the important line of my code):
def prediction():
   ...
   table=[]
   ...
   table.append(np.array([[name], [x], [x+w], [y], [y+h]], ndmin = 2).reshape((1,5)))
   table = pd.DataFrame(np.vstack(table))
   classes_probas=CNNmodel.predict(images)
   max_proba=classes_probas.argmax(axis = 1)
   classes_probas = list(classes_probas)
   max_proba = list(max_proba)
   #Add probas of each class to the table
   table['class1'] = classes_probas[0]
   table['class2'] = classes_probas[1]
   table['class3'] = classes_probas[2]
   table['class4'] = classes_probas[3]
   table['class5'] = classes_probas[4]
   table['class6'] = classes_probas[5]
   table["max_proba"] = max_proba
   return table

After I call the function "predictions":
table= pd.DataFrame(columns=['name','Xmin','Xmax','Ymin','Ymax','class1','class2','class3','class4','class5','class6','max_proba'])
table=prediction()

But when I run my code, it shows this error:
ValueError: Length of values does not match length of index


texts: What should I start with first? Python for beginners or applied data science?

texts:
I am new to data science and I would like to go deep into it.
I decided to start working with advanced competitions on kaggle, yet I should refine my knowledge in python and ML.
I am an engineer, with 4 years of programming (basically php vanilla JS, and javascript based frameworks)
I took a coirse on Udemy about ML with python. As I am already a programmer, I know most of the basics, I just need to familiarise with the script and how its written in python.
I am desperate for a help from someone, who can help me with if I should take the Python for everyone or simply just dig in directly into Applies data science with Python from University of Michigan.
I know sometimes I struggle writing a code with python but with a basic google search I can find an answer or figure out one of my own.
I know that I can skip lots of weeks on coursera as I already knows the basics. But still confused.


texts:
I am new to machine learning and it might be a bit of a stupid question.

I have implemented my model and its working. I have a question about running it on the testing data. Its a binary classification problem. If I know the proportions of classes in test data how could I use it to improve the performance of my model or prediction made by the model. So lets say 75% belong to class 1 and 25% to class 0

Any help is greatly appreciated


texts:
The tensorflow example CIFAR10 uses input pipelines to load data from the disk to a queue. I would like to implement this for my own models, but I run into an error that I can't fix somehow. My minimal example looks like the following (taken from the tutorials: https://www.tensorflow.org/programmers_guide/reading_data):

def read_sample_of_csv(filename_queue):

   # Define Reader
   reader = tf.TextLineReader()
   key,value = reader.read(filename_queue)

   # Decode CSV
   record_defaults = [[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]]
   col0, *col = tf.decode_csv(value, record_defaults=record_defaults)
   features = tf.stack([*col])
   reference = col0

   # Return
   return features, reference

def input_pipeline(filenames, batch_size, ...):
   # Reader for samples
   slipangle_filename_queue = tf.train.string_input_producer(filenames, shuffle=shuffle)
   features, reference = read_sample_of_csv(slipangle_filename_queue)

   # Batch samples
   capacity = min_after_dequeue + (num_threads+3) * batch_size
   feature_batch, reference_batch = tf.train.shuffle_batch([features,reference], batch_size=batch_size)

   # Return
   return feature_batch, reference_batch

with tf.device(MY_DEVICE):
   feature_batch, ref_batch = input_pipeline(filenames, batch_size, ...)

This is basically what is done in CIFAR10, just with a csv. The script runs with MY_DEVICE = '/cpu:0', but when I try to run it on the GPU with MY_DEVICE = '/gpu:0' I get the following error:

InvalidArgumentError (see above for traceback): Cannot assign a device to node 'shuffle_batch/random_shuffle_queue': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and devices:
QueueEnqueue: CPU
QueueSize: CPU
QueueClose: CPU
QueueDequeueMany: CPU
RandomShuffleQueue: CPU
[[Node: shuffle_batch/random_shuffle_queue = RandomShuffleQueue[capacity=1500, component_types=[DT_FLOAT, DT_FLOAT], container="", min_after_dequeue=1000, seed=0, seed2=0, shapes=[[9], []], shared_name="", _device="/device:GPU:0"]()]]

Can I not use input pipelines on GPUs? Isn't this what is done in CIFAR10?


texts:
I'm running a test on MapReduce algorithm in different environments, like Hadoop and MongoDB, and using different types of data. What are the different methods or techniques to find out the execution time of a query.

If I'm inserting a huge amount of data, consider it to be 2-3GB, what are the methods to find out the time for the process to be completed.


texts: Catboost multiclassification evaluation metric: Kappa & WKappa

texts:
I am working on an unbalanced classification problem and i want to use Kappa as my evaluation metric. Considering the classifier accepts weights (which i have given it), should i still be using weighted kappa or just use the standard kappa? I am not entirely sure of the difference to be honest.
model = CatBoostClassifier(
    iterations = 1000,
    learning_rate = 0.1,
    random_seed = 400,
    one_hot_max_size = 15,
    loss_function = 'MultiClass',
    eval_metric = 'WKappa', # weighted kappa
    #ignored_features = ignore_list,
    class_weights= weights,
    od_type = "Iter",
    od_wait = 50,
    task_type = 'GPU',
    border_count = 124,
    depth = 12
)


texts: Converting a regression problem into a classification problem

texts:
I have a regression problem where a neural network has to predict a value from $0$ to $19.999...$. I would like to convert this regression problem into a classification problem with the classes $0, 1, 2, ..., 18, 19$. Therefore I am using a $\mathrm{softmax}$-output layer.

Now, if I use a categorical crossentropy-loss, the loss is the same if the network classifies an input as $1$ instead of $14$ and if the input is classified as $13$ instead of $14$. I like to punish the loss more if the predicted value / class is farther away from the true value / class.

Is there a better loss function than categorical crossentropy for this use-case?


texts: Anomaly detection k-means in Time Series

texts:
I'm trying to use k-means to detect anomalies in the Amount column. I have the following part of my dataset:

8   2018-09-06  -2760.14
9   2018-10-04  -1149.73
10  2018-11-07  -1551.41
11  2018-12-06  -1208.17
12  2019-01-07   -244.02
13  2019-02-06    111.00
14  2019-03-06    139.24
15  2019-04-04 -27315.10
16  2019-05-07 -30326.58
17  2019-06-06  -1633.44

I used 2 as number of clusters and I get the following centroids:
[[  -237.045]
 [-28820.84 ]]
In my case the anomalies are rows 15 and 16 but now they form their own cluster so the distance to their centroid will be insignificant and thus not an anomaly
My question is, how can I detect the anomalies if they form their own cluster?


texts: What is the difference between Pachyderm and Git?

texts:
I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that:

It holds all your data in a central accessible location
It updates all depending data sets when data is added to or changed in a data set
It can run any transformation, as long as it runs in a Docker, and accepts a file as input and outputs a file as result
It versions all your data
It handles both modified data and newly added fractions of data
It can keep branches of your data sets when you are testing new transformation pipelines

It seems that Git can handle all of them. And maybe data is always larger in size than code then git-lfs was created for that purpose.
In contrast, Dolt provides a different direction that combines SQL and Git.
Do tools like Pachyderm apply nowadays in data science?


texts: ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

texts:
I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python.
If it is possible to separate Data Engineering from Data Science, on what principles is Data Engineering based, what is the result of the Data Engineering? Is it creating some data structures? If so, what these structures might be? Are there standards or best practices?