texts:
I trained an InceptionV3 model using plant images. I used Keras library. When training was started, first epoch took 29s per step and then other steps took approximately 530ms per step. So that made me doubt whether there is a bug in my code. I checked my code several times, but its logic seems right to me. I trained my model on Google Colab. I wonder whether there is a memoization mechanism or my code contains bugs. Here my code:

# Yields one image-target pair when called
def image_target_generator(files, labels):
assert len(files) == len(labels), 'Files and labels sizes don\'t match!'

for step in range(len(files)):
    img = cv2.imread(dataset_path + files[step])
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    item = (img, labels[step])
    yield item

# Generating batch
 def batch_generator(gen):

    batch_images = []
    batch_targets = []

    for item in gen:

      if len(batch_images) == BATCH_SIZE:
        yield batch_images, batch_targets
        batch_images = []
        batch_targets = []

      preprocessed_img = preprocess_image(item[0])
      batch_images.append(preprocessed_img)
      batch_targets.append(item[1])

    yield batch_images, batch_targets

# Training generator
def training_generator(files, labels):

  # So that Keras can loop it as long as required
  while True:

    for batch in batch_generator(image_target_generator(files, labels)):
      batch_images = np.stack(batch[0], axis=0)
      batch_targets = keras.utils.np_utils.to_categorical(batch[1], NUM_CLASSES)
      yield batch_images, batch_targets

# Create model
def create_model():
  model = keras.applications.InceptionV3(include_top=False, input_shape= IMG_SIZE, IMG_SIZE, 3), weights='imagenet')

  new_output = keras.layers.GlobalAveragePooling2D()(model.output)
  new_output = keras.layers.Dense(NUM_CLASSES, activation='softmax') (new_output)
  model = keras.engine.training.Model(model.inputs, new_output)

  for layer in model.layers:
    layer.Trainable = True

    if isinstance(layer, keras.layers.BatchNormalization):
      layer.momentum = 0.9

  for layer in model.layers[:-50]:
    if not isinstance(layer, keras.layers.BatchNormalization):
      layer.trainable = False

  return model

# Compiling model
model = create_model()

model.compile(loss='categorical_crossentropy',optimizer=keras.optimizers.adamax(lr=1e-2), metrics=['accuracy'])

# Fitting model
model.fit_generator(
  training_generator(train_x, train_y),
  steps_per_epoch=len(train_x) // BATCH_SIZE,
  epochs = 30,
  validation_data=training_generator(test_x, test_y),
  validation_steps=len(test_x) // BATCH_SIZE
  )


texts: Problems finding an LSTM model for classification

texts:
I am doing a study for the classification of musical genres using deep learning techniques. The work consists of making a classification using an LSTM model.

I am using GTZAN as a data set, and preprocessing them using Librosa. This way I obtain the spectral characteristics, specifically: Spectral centroid.

My input data has the following size: X_train (750, 20, 1249).

My model is:

model = Sequential ()
model.add (LSTM (units = 512, return_sequences = True, input_shape = (X_train.shape [1], X_train.shape [2])))
model.add (LSTM (units = 32, return_sequences = False))

model.add (Dense (units = 10, activation = 'softmax'))
model.compile (loss = losses.categorical_crossentropy,
                 optimizer = Adam (),
                 metrics = ['accuracy'])

model.summary ()

I get these results:

I scale also my data using MinMaxScaler.

I appreciate any help. Thank you.


texts: Decision Trees Nodes vs Leaves Definition

texts:
I am having a little trouble understanding the difference between what a "Node" of a tree and a "Leaf" of a tree.

Suppose I am trying to decide the size of coffee a person may like. There are three categories: small, medium, and large based off the peoples age, height, weight, income.

So I have four predictors and 3 possible outcomes. When looking at many gradient boosting algorithms, there are parameters that can increase the number of leaves.

My understanding of this (correct me if wrong), but I will illustrate with a picture. Assuming each yes/no split is 50/50

Does increasing the number of leaves to lets say 3 leaves change it from yes/no aka 50/50 to 33/33/33? This is a little confusing to me. Thank you for any clarification.

Sam


texts: Can I add features that are parts of another feature?

texts:
I am building a model (implementing both logistic regression and Xgboost) to understand the importance/significance of each feature in whether a customer is going to repurchase to understand what matters for customers to repurchase (More interested in inferences than the predictions)

My feature set looks like this: [income, age, price_product, discount, product_category, delivery_charge, gender, lifestyle, delivery_time_main (P_main), P1, P2, P3, P4 and so on]

P_main is something that the customer sees and may influence their decision to repurchase or not. P1 + P2 + P3 + P4 = P_main. We want to understand how many important are P1, P2, P3, P4 as well as they are stages in the process so we can make inferences about which stage is to be focused most on if we were to improve repurchases.

Can the feature set contain the parts (P1, P2, P3, P4) as well as the sum (P_main) as inputs to the model? Or does it introduce multicollinearity problems? I am removing features that have multicollinearity using the Variance Inflation Factor and then applying Lasso regression.


texts: Extracting name, date and total from a set of heterogeneous receipts

texts:
So, this is how the problem goes: I am trying to extract information from scanned receipts like this,

What I have been told is that I would get the textual data from a OCR software, so in short I will be working with a textual version of the image directly.

Problem:

The problem in hand here is that I have to extract certain information here, namely,

The Location. (for eg: New York, United States)
The complete Total (after all the discounts, tips, etc. have been involved) (Eg: 1033.42)
The Currency. ($, £, €, etc)
The Date. (Easier to guess)

The reason I want to extract the Location information is that, if for example, the currency is not explicitly mentioned here, then I can infer from the location where the receipt is generated.

Challenges:

The challenge here is that, information like Total Due could be anything like Grand Total or Total (only), or something semantically similar to Total, as I am not going to get the same receipt from the same restaurant only. (The restaurant could be anywhere in the world, but the problem is limited to only English speaking countries for now.)

The other challenge is to actually get the total information. It's very easy for us to see that 1033.42 is the total in the above receipt. But how do I get the software to know that? The way I see it, is that 1033.42 is near to the total(proximity). But there could be other numbers near to it as well.

Where I have tried and failed:

I have been told to start from NLTK(NER) but NER doesn't work for everything here. I can get the date information through it but the problem isn't just only to identify what the named entities are, imo.

What I think would work

The way I see it, I think I need to use a machine/deep learning model where the machine would be able to understand the proximity match between anything which semantically says Total and the number near to it (most probably on the right side).

Any help regarding which model would work best in terms of speed(first and foremost) and accuracy would be greatly appreciated.
I would also appreciate any help regarding where I could find any dataset or existing model which I could use for Transfer Learning.


texts: Why increasing the number of units or layers does not increase the accuracy and decrease the loss?

texts:
I have an LSTM neural network; when I increase the number of units, layers,  epochs or add dropout, it seems it has no effect and still I have persistent errors and accuracies like the following:

  loss: 3.5071 - acc: 0.0981 - val_loss: 6.7042 - val_acc: 0.0122

Why this happens and how can I fix it?


texts: Printing Feature Contributions in a Random Forest algorithm from the Treeinterpreter library leading to errors

texts:
I am working on a dataset where I predict the risks of developing pancreatic cancer with respect to a number of variables. I have created a random forest, and want to find the feature contributions. I have already used the "Treeinterpreter" library, resulting in a contributions array that is three-dimensional. I want to display the contributions in the array beside the name of the factor/variable. I have used the code below to do so, however, the code responsible for displaying the contributions does not work. I have tried multiple methods, including converting the dataframe to a numpy array, and other methods such as .all() and .any(). However, none are producing the desired result.

What can be the right way to display the feature contributions with respect to each of the feature it represents?

     # -*- coding: utf-8 -*-
"""
Created on Mon Apr 15 13:39:19 2019

@author: GoodManMcGee
"""

import pandas as pd
from sklearn.metrics import accuracy_score
from sklearn import tree
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn.metrics import confusion_matrix
from sklearn.ensemble import RandomForestClassifier
from IPython.display import Image
from sklearn.tree import export_graphviz
from treeinterpreter import treeinterpreter as ti
import matplotlib.pyplot as plt
import numpy as np
import itertools

data = pd.read_csv("pancreatic_cancer_smokers.csv")
target = data['case (1: case, 0: control)']
data.drop('case (1: case, 0: control)', axis=1, inplace=True)
x_train, x_test, y_train, y_test = train_test_split(data, target, test_size = 0.2)
clf = RandomForestClassifier(n_estimators=100)
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)
clf_accuracy = accuracy_score(y_test, y_pred)
clf_pred, clf_bias, contributions = ti.predict(clf, x_test)

#The code below was taken from DataDive's treeinterpreter tutorial.
#The aforementioned messages applies to all code between the underscores
#///////////////////////////////////////////

for i in range(len(x_test)):
    print ("Instance", i)
    print ("Bias (trainset mean)", clf_bias[i])
    print ("Feature contributions:")
    for c, feature in sorted(zip(contributions[i], data.feature_names),
                             key=lambda x: -abs(x[0])):
#An error occurs in the "data.feature_names" method in the code above:AttributeError: 'DataFrame' object has no attribute 'feature_names'. I have tried referenceing columns from datasets also, but that also leads to errors: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
        print (feature, round(c, 2))
    print ("-"*20)
#///////////////////////////////////////////


texts: Is there any way to find out execution time for classification algorithms in orange?


texts: Data pre-processing before feeding into a deep learning model

texts: Generally speaking, when training a deep learning model, like MLP,  what kind of data pre-processing operation has to be conducted when the input is a numerical sequence.


texts: Plot RDD data using a pyspark dataframe from csv file

texts:
I have just started working with pyspark on very large csv file. I am using Spark version 2.1.0. I want to read data from a .csv file and load it into a spark dataframe and then after filtering specific rows, I would like to visualize it by plotting 2 columns (latitude and longitude) using matplotlib.

This is what I've done so far:

import packages:

from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
from pyspark.sql.functions import *
import matplotlib.pyplot as plt

build spark session:

spark = SparkSession.builder \
     .master("master:IP") \
     .appName("plot track") \
     .config(conf=SparkConf()) \
     .getOrCreate()

read .csv file:

df = spark.read.csv('data.csv')

filter rows according to values in a specific column ("_c1"):

one_ID = df.filter("_c1 == 1234")

save column data? (Latitude data is in column "_c3" and longitude data is in column "_c4")

latitude = one_ID.rdd.map(lambda y: y._c3)
longitude = one_ID.rdd.map(lambda x: x._c4)

plot:

plt.plot(longitude,latitude, 'o-')
plt.show()

I thought this would work but it does not. I get the error -
- 'TypeError: float() argument must be a string or a number'

Any help would be greatly appreciated. Thanks.


texts: I get a completely different Autoencoder model everytime I run it in R

texts:
My current task is to implement time series forecasting using Autoencoders in R. After the full Autoencoder is trained, I use the Encoder part to perform the forecasting. The inputs of the autoencoder are sequences of 10 observations of my time series (which are used to predict the next observation).
The training dataset looks like this
   lag 1  lag 2  lag 3  lag 4  lag 5  lag 6  lag 7  lag 8  lag 9 lag 10
1 3897.9 3989.8 3885.0 3580.7 3784.3 3926.9 3840.3 3758.5 3862.3 3809.4
2 3944.4 3897.9 3989.8 3885.0 3580.7 3784.3 3926.9 3840.3 3758.5 3862.3
3 3829.8 3944.4 3897.9 3989.8 3885.0 3580.7 3784.3 3926.9 3840.3 3758.5
4 3637.4 3829.8 3944.4 3897.9 3989.8 3885.0 3580.7 3784.3 3926.9 3840.3
5 3556.3 3637.4 3829.8 3944.4 3897.9 3989.8 3885.0 3580.7 3784.3 3926.9
6 3401.6 3556.3 3637.4 3829.8 3944.4 3897.9 3989.8 3885.0 3580.7 3784.3

I'm using keras library to define and fit the Autoencoder
    input = layer_input(shape = 10)

    encoded = input %>%
    layer_dense(units = 1,activation = "sigmoid")

    decoded = encoded %>%
    layer_dense(units = 10,activation = "sigmoid")

    autoencoder = keras_model(input, decoded)
    encoder = keras_model(input, encoded)

    autoencoder %>% compile(
    optimizer='adam',
    loss='mean_squared_error'
  )

  callback = callback_early_stopping(
    monitor = "val_loss",
    min_delta = 0.001,
    patience = 10,
    verbose = 0,
    mode = c("auto", "min", "max"),
    baseline = NULL,
    restore_best_weights = TRUE
  )

  autoencoder %>% fit(
    x_train, x_train,
    epochs = 50,
    batch_size = 256,
    shuffle = FALSE,
    validation_data = list(x_test, x_test),
    callbacks = callback,
    verbose = TRUE
  )

The problem is that everytime I fit this model with the exact same values for epochs and batch_size it reults in completely different predictions. For instance, one time I get these predictions

But after fitting the same model again, I get these completely different predictions

Any insights on what I might be doing wrong? Thanks in advance!


texts: Learning rate in logistic regression with sklearn

texts: In sklearn, for logistic regression, you can define the penalty, the regularization rate and other variables. Is there a way to set the learning rate?


texts:
There are many resources online about how to implement MLP in tensorflow, and most of the samples do work :) But I am interested in a particular one, that I learned from https://www.coursera.org/learn/machine-learning. In which, it uses a cost function defined as follow:

$
J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K} \left[ -y_k^{(i)} \log((h_\theta(x^{(i)}))_k - (1 - y_k^{(i)}) \log(1 - (h_\theta(x^{(i)}))_k \right]
$

$h_\theta$ is the sigmoid function.

And there's my implementation:

# one hidden layer MLP

x = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])

W_h1 = tf.Variable(tf.random_normal([784, 512]))
h1 = tf.nn.sigmoid(tf.matmul(x, W_h1))

W_out = tf.Variable(tf.random_normal([512, 10]))
y_ = tf.matmul(h1, W_out)

# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)
cross_entropy = tf.reduce_sum(- y * tf.log(y_) - (1 - y) * tf.log(1 - y_), 1)
loss = tf.reduce_mean(cross_entropy)
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# train
with tf.Session() as s:
    s.run(tf.initialize_all_variables())

    for i in range(10000):
        batch_x, batch_y = mnist.train.next_batch(100)
        s.run(train_step, feed_dict={x: batch_x, y: batch_y})

        if i % 100 == 0:
            train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})
            print('step {0}, training accuracy {1}'.format(i, train_accuracy))

I think the definition for the layers are correct, but the problem is in the cross_entropy. If I use the first one, the one got commented out, the model converges quickly; but if I use the 2nd one, which I think/hope is the translation of the previous equation, the model won't converge.


texts:
I am working on a project to classify CT scan images using the CNN model, the image size is huge and I want to feed it into the network using the idea of batches, tried doing that with this pieces of code:

    # train_data size = 5460
    num_epochs = 14
    batch_size = 390
    batch = 0
    print("Starting training...")
    for epoch in range(num_epochs):
        train_batch = train_data[batch:batch_size]
        batch += batch_size
        batch_size += batch_size
        ep_loss = 0
        for data in train_batch:
            X = data[0]
            Y = data[1]
            _, c = sess.run([optimizer, cost], feed_dict={x_img: X, y_label: Y})

my questions are:

1- Is this a right way to do batching? or there is a better way?

2- with the above code I am using 'AdamOptimizer' is it a good optimization technique for the idea of batching or I should use another one?


texts: Suggestion for model performance improvement for ML competition

texts:
I am working on highly imbalanced dataset and trying to increase accuracy(metric: roc_auc) of my model which is hovering around 82-83%. This is part of an internal ML competition and people who are at the top have accuracy around 85-88%.

I am just wondering what else I can do to improve my model's accuracy. Any suggestion or tips would be appreciated.

Details of training dataset :

The training data shape is : (166573, 14)

Distribution of features :

As you can see, only the first 4 columns go to different max values.
Rest of the columns have either 1 or 0 value (max: 1, min: 0)

Scaling features :

X['scaled_distance']= sc.fit_transform(X['distance'].values.reshape(-1,1))
X['scaled_visit_count'] = sc.fit_transform(X['visit_count'].values.reshape(-1,1))
X['scaled_tier'] = sc.fit_transform(X['tier'].values.reshape(-1,1))

Null Handling :

train['tier'].fillna(round(train['tier'].mean(),2),inplace=True)

At last, I have tried different models (Xgboost, Random Forest with SMOTE, lightbgm etc..) I have got best results with lightbgm with some tuned parameters..

lgbm.fit(X_train,y_train)
LGBMClassifier(bagging_fraction=0.8, bagging_freq=15, boosting_type='gbdt',
        class_weight=None, colsample_bytree=1.0, feature_fraction=0.5,
        importance_type='split', is_unbalance=True, learning_rate=0.01,
        max_depth=7, min_child_samples=20, min_child_weight=0.001,
        min_split_gain=0.0, n_estimators=520, n_jobs=-1, num_leaves=40,
        objective=None, random_state=10, reg_alpha=0.0, reg_lambda=0.0,
        silent=True, subsample=1.0, subsample_for_bin=200000,
        subsample_freq=0)

Please refer full code here :
https://github.com/PraveenKS30/ML/blob/master/Surge2019/Surge%20Pre%20Machine%20Learning.ipynb

I am not sure what else I can do to improve my accuracy.. Should I preprocess in different way ? Should I try neural network now?
Please suggest.


texts: Autoencoder behavior with All White/Black MNIST

texts:
I am using a stock auto-encoder anomaly detector from Deeplearning4j.

I was getting unexpected results from my own variant of the auto-encoder, which looks for anomalies in my own (non-image) data, and to try and investigate, I added some additional images, 10 white and 10 black to the MNIST test data, to see the effect. See the output picture below:

.

What was unexpected was that those black images (all zero input values) have zero reconstruction error (the left column in the "best" picture), despite the auto-encoder being trained only on stereotypical digits (no black images), while the white images(i.e. all 1 input values) have the highest reconstruction error (the left column on the "worst" image). Given the generally accepted definition of an anomaly detector, I would have expected both all-black and all-white to have high reconstruction error, as neither appeared in the training set, and hence should be anomalous.

This is consistent with the unexpected output from my own variant of the auto-encoder.

I suspect there is a data science reason why this is the case with an auto-encoder, but can someone shed light, or point me to references why?


texts: Removing sensitive information from a dataset while retaining its usefulness for machine learning tasks?

texts:
I have a dataset that would be useful for training future machine learning models.

However, it contains sensitive information (personal identifiers like names and addresses) that make it difficult to store or retain longterm.

I'm wondering if there are transformations that could be applied that would remove the sensitive information, while retaining it's usefulness?

The dataset is specifically useful for training NLP related models, but I'm also interested in a general solution to this for general machine learning tasks if one exists.


texts:
Why does it seem that it's difficult to find out how people in data science create measurable value?  All I find on the internet are buzzwords like data cleaning, visualization, and writing about the data.  This is equivalent of describing a landscaper as trimming trees and grass or saying an investment banker spreads M&A.  That is insufficient.  Let my type out an insufficient answer:

  A data scientist uses database querying languages like SQL as well as statistical programming languages like R to design experiments with data and visualizing such experimental conclusions to decision-makers in the hopes of making better decisions and thus creating value for a company/cause.

How is this insufficient?  Well, how do we know that data scientists' recommendations actually create value?  If we can't measure the value of our analyses, then how can we determine if hundreds of hours of learning/working in the subject is meaningful?  Extreme example:

  George is a data scientist working for Company X.  After hundreds of hours of data cleaning/experimentation, he concludes Decision A will benefit Company X.  George convinces a Product Manager to apply Decision A that ultimately increases the revenues of Company X by $0.01

How do we know that several years of work may not amount to basically nothing?  Let me give you a depressing and real/plausible example:

  Instead of becoming a data scientist, George becomes a portfolio manager, managing investment portfolios for clients.  After 20 years of management, George has a track record of -1% per annum, compared to an S&P 500 return of 8%.

In the above example, George is a useless person and has lost value for people.  How do data scientists know that they aren't destroying value, and if we can't figure that out, what are the steps to avoid value destruction and create the most value for companies?


texts: Problem with Linear Regression and Gradient Descent

texts:
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
import numpy as np

x = np.linspace(0, 10, 1000)
points = np.genfromtxt("Dokumente/Salary.csv", delimiter=",")
points2 = points[1:, :]

def gradient_descent(current_b, current_m, learning_rate, points):
    b_gradient = 0
    m_gradient = 0
    N = float(len(points))
    for i in range(0, len(points)):
        x = points[i, 0]
        y = points[i, 1] / (10 ** 4)
        b_gradient += -(2/N) * (y - ((x * current_m) + b_gradient))
        m_gradient += -(2/N) * x * (y - ((x * current_m) + current_b))
    b_new = current_b - (learning_rate * b_gradient)
    m_new = current_m - (learning_rate * m_gradient)
    return [b_new, m_new]

def compute_error(points, b, m):
    total_error = 0
    N = len(points)
    for i in range(0, len(points)):
        x = points[i, 0]
        y = points[i, 1] / (10 ** 4)
        total_error += (1/N) * ((y - ((m * x) + b)) ** 2)
    return total_error

def run(points, staring_b, starting_m, learning_rate, num_iterations):
    b = staring_b
    m = starting_m
    for i in range(num_iterations):
        learning_rate = learning_rate
        b, m = gradient_descent(b, m, learning_rate, np.array(points))
        plt.plot(i, compute_error(points, b, m), "ro")
    return [b, m]

learning_rate = 0.000001
init_b = 0
init_m = 0
number_of_iterations = 10000
print(compute_error(points2, init_b, init_m))
[b, m] = run(points2, init_b, init_m, learning_rate, number_of_iterations)
print(compute_error(points2, b, m))

plt.plot(points2[:,1] / (10 ** 4), "ro")
x = np.linspace(0, 35, 1000)
plt.plot(x, ((x * m) + b))
plt.show()

I played with learning rate and the number of iterations, but this does not fit the data intuively.
Can i get some tips on my code please?
Thank you in advance.


texts: Can someone explain to me the structure of a plain Recurrent Neural Network?

texts:
I have seen pictures of RNNs and LTSMs, and they usually look like this:

Here the task is to take a sentence and make a prediction of some sort.  What are each of the green squares?  Are each of them layers, or does each green square have it's own set of layers and neurons?
Additionally, in such a scenario, the input data is the entire sequence of words.  When dealing with time series data, would the input be the entire data set since the entire dataset is a sequence?  If you have 1million observations, you would need to construct 1million green squares?


texts: How to interpret a drastic accuracy loss while training a neuronal net (CNN)?

texts:
How can one interpret a drastic accuracy loss after ~38 epochs? Maybe more dropout should be added to the CNN network?

(x-axis shows the number of epochs)


texts: Forecasting time series: Method Selection

texts:
Im new to forecasting time series and Im looking for some advice on selecting the best method based on the analaysis of the graph.

I have the following data and based on the little knowledge I have, Im assuming it is a stationary time series because of the shape of the data and because of the result I got when performing a Dicker Fulley Test

Test Statistic                -6.560544e+00
p-value                        8.402824e-09
#Lags Used                     2.100000e+01
Number of Observations Used    1.164000e+03
Critical Value (5%)           -2.864026e+00
Critical Value (1%)           -3.435980e+00
Critical Value (10%)          -2.568094e+00
dtype: float64

That made me choose Arima with a p and q values of 1. However the result Im getting are pretty awful. Can anyone guide on how should I choose the adequeate method for forecasting and  if case ARIMA is a good choice, how should I tweak parameters to improve my results? The end goal is to predict the next month of data, so it might not be necessary to use all of the data.


texts: Equivalence of Tidy Data and Third Normal Form

texts:
In Hadley Wickham's "Tidy Data" paper, he states that

  In tidy data:

  Each variable forms a column.
  Each observation forms a row.
  Each type of observational unit forms a table.

  This is Codd’s 3rd normal form (Codd 1990), but with the constraints
  framed in statistical language, and the focus put on a single dataset
  rather than the many connected datasets common in relational
  databases.

Codd's Third Normal Form can be described as:

A table is in 1st normal form if

It stores information in rows and columns where one or more columns, called the primary key, uniquely identify each row.
Each column contains atomic values, and there are no repeating groups of columns.

A table is in 2nd normal form if

The table is in 1st normal form, and
All the non-key columns are dependent on the table’s primary key.

A table is in 3rd normal form if

It is in 2nd normal form, and
It contains only columns that are non-transitively dependent on the primary key

I am trying to understand how these two sets of rules are equivalent.

I believe that the first Tidy Data rule maps to 1st normal form.  Specifically, "Each variable forms a column" maps to "Each column contains atomic values, and there are no repeating groups of columns".

I believe the Tidy Data rule "Each observation forms a row" maps to "All the non-key columns are dependent on the table’s primary key" and also "It contains only columns that are non-transitively dependent on the primary key".

I suspect that "Each type of observational unit forms a table" maps to "The table stores information in rows and columns where one or more columns, called the primary key, uniquely identify each row", but I'm least sure about this mapping.

I believe my above analysis is not (quite) correct, so I'm hoping someone can clarify the link between these two sets of rules.

My question is: How are the three aforementioned Tidy Data rules equivalent to Third Normal Form?


texts: Representing output label for character level speec recognition using RNN

texts:
I saw this tutorial on generating text using LSTM. In this tutorial the author trained the network by taking 100 previous characters as input and the next character as the output label.

I am interested to try some simple speech recognition using LSTM. I may use mfcc features of the audio signal as input data, but what's confusing me most is how to represent the output label.

The dataset I have is the VCTK corpus which contains sentence level audio recording and its transcription.

In the tutorial, next character that comes after the input vector was used as output label. But for speech it's impractical to know which part of speech produced which character without transcribing the audio for every second. So, how would I represent the output labels for this problem?


texts: What's the most effective way to measure the accuracy of my multi-class classification NN?

texts:
I'm relatively new to data science, and am confused about how to measure the accuracy of a multi-class neural network. The model I'm building is attempting to predict the outcome of a given event with 20 different outcomes - there is some imbalance in the labels, the most frequency label is around 15% of the total, and the least frequent is around 1%. The event I'm attempting to classify is difficult to predict, and I'll be happy with around 20% accuracy over all events - however I'm worried that when I achieve that, then my model will simply be predicting the most frequent class all the time, and just getting lucky on enough of the others to bring overall accuracy up to 20%. I'm using keras' 'categorical cross-entropy' loss function and 'accuracy' metric - are these the most appropriate for the problem? Which other metrics are best suited to a problem like this? Very happy to be referred to literature to help build my understanding of this problem.
I understand the confusion matrix for binary classification, however for multi-class classification, however I'm at a loss (no pun intended) as to how to apply these principles to a multi-class problem, whilst ensuring my model isn't just predicting the most popular class and getting lucky on enough others to meet my target.
Thanks in advance for any assistance!


texts: What does the phrase 'underlying mapping' mean?

texts: In a paper by KaimingHe entitled Deep Residual Learning for Image Recognition, there is a phrase 'underlying layer'. What does this mean?


texts: After plotting my predicted values against the true label values, I didnt quite get the answer I was looking for

texts:
I downloaded data on wine quality and tried to run a regression model to predict the quality, However I did not receive the plot I was expecting. The mean absolute error for the wine quality was approximately 0.5, as a result of this I thought that the True vs predicted value plot would look very similar however I got a graph that Isn't really what I was expecting. Is this what happens when you work with discrete parameters?

As I'm new to this, I was under the impression that this final plot would always resemble your standard linear graph if your predicted and true values were very similar.

Here is the link to my code if it helps: https://colab.research.google.com/drive/1mxRIx5ufVsA0ljdTpL0Ud0qm2y39eyGX


texts: Looking for second opinion on checking that attribute in data frame is unique using r

texts:
Check whether attribute name in city is unique (i.e. each city has a different name).  data_frame is city with a column called name.

length(unique(city$name))
rapply(city,function(name)length(unique(name)))
unique(city$name)


texts: How can I count the number of matching zero elements between two numpy arrays?

texts:
I have a function that returns the predicted accuracy of a time-series model. I have two equally-sized numpy arrays, one for the actual direction and one for the predicted direction. I'm classifying whether there is a change in the data's direction from the previous t-1 step.  '1' stands for an increase from t-1, '0' stands for no change in direction, and '-1' stands for a decrease from t-1. I'm trying to compare the elements of both arrays to determine if they both contain the same data to determine accuracy.
I can match indexes and count the number of '1s' and '-1s' that match,  but I cannot count the number of matching zeros. (It's kinda hard summing zeros). :-)  Anyway, I've tried the numpy sum function specifying '0' as the argument for both arrays but it only returns an array of zeros but no count.  I'm not trying to create a confusion matrix...the goal here is to create an accuracy score.  I plan to take all the matching ones, negative ones and zeros and divide that by the total length to get an accuracy score.
Thanks in advance.


texts: What's the name of data points that trained model gets on input in production?

texts:
I need some machine learning vocabulary advice.

What do you call input data that a trained model gets in production?

I know that labelled or unlabelled data it gets during training is called training set or training data.

When it's ready, after initial training, you deploy it in production to solve real problems. How do you call the input data that the model receives at that point?


texts: How does multicollinearity affect neural networks?

texts:
Multicollinearity is a problem for linear regression because the results become unstable / depend too much on single elements (source).

(Also, the inverse of $X^TX$ doesn't exist so the standard OLS estimator does not exist ... I have no idea how, but sklearn deals with it just fine)

Is (perfect) multicollinearity also a problem for neural networks?


texts: Financial SEC Filling Table data Extraction Using Machine Learning

texts:
I am trying to build a ML system which extracts financial data from Tables.

Sample table is like below.

From the above image you can see Net sales is an attribute along with two values 2707.1 and 1994.5 for 2 different  quarters. So I want to extract those data with meta data like The date, Quarter,  Currency value etc.,. Like below image.

Normally if it is in Free Text I use Entity extraction methods using Spacy or other modules.
We can not apply rules because there will be many variety of table from which we have to extract from.

Edit:
The data are from HTML file.

A sample website


texts: Hi I am working on a movie recommendation system and I have like 80.000 movies in my data set. The problem is there are like thousands of directors which is a problem when I represent a movie as a vector. One of the features of a movie is its director and I do not want my movie vector to be like 5000 in length. One hot encoding is advised everywhere but I want to do something else that will make my movie vectors small in length. I heard integer encoding but I have never seen an example of it nor it is advised. One thing that comes to my mind is to create a small interval like from 0 to 5 and let's say I have 6000 directors and I will assing first director to 0 and the next director will be assigned to 0+6/6000 and third one will be assigned to 0+2×6/6000 etc. However, I have a lot of doubts. what can you suggest? I am a rookie and not well-versed on the topic.


texts:
I've got a question about embedding. I've got dataset of items of which I have a name, description and image. I want to make vector representation of them using these three features. How should I do that? I made three different models (using neural network) which are using these features to classify these items (to some categories) and I have representation of these features from embedding layers, but how can I concatenate these representations into one vector which will be the best representation of each of item? These representations have to be similar if the items are similar.
Do you have some advice to my problem?

Best regards


texts: Does the transformer decoder reuse previous tokens' intermediate states like GPT2?

texts:
I recently read Jay Alammar's blogpost about GPT-2 (http://jalammar.github.io/illustrated-gpt2/) which I found quite clear appart from one point :
He explains that the decoder of GPT-2 processes input tokens one at a time, only actively processing the last input token, the past tokens being saved in memory already and "passively" reused without reevaluation.

From my understanding of the transformer architecture, I had the impression that the decoder reevaluates every token generated at each generation. Is this then a difference between the decoder from GPT-2 or does the decoder from a "classical' transformer also work this way ?

Intuitively I would think that it would make more sense the reevaluate everything at each iteration since new dependencies between words can appear that weren't there at the beginning and would then not be taken into account if past processed words are passively reused.

I hope I am making sense, can someone with knowledge about the GPT2 architecture help me clarify this ?


texts: AB testing : When AA testing doesn't work

texts:
After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old AA testing.
AA testing consists in dividing randomly the users between two branches, making both branches have the exact same experience, and testing that conversion rates on both branches are not significantly different, which would mean that both branches are not actually treated the same (or that the population is not uniformly distribued between two branches)
As you can imagine, this did not happen at all. We sent roughly 3500 mails to each branch, and one of these exhibits a 40% conversion while the other only has 32%, which is a difference that has a p-value of the order of 10^-12
Furthermore, we partitionned the time in 4 periods, each one of these exhibit a very significant difference with the other one.

Now what am I supposed to do? I would like to have a theoretical discussion about this with Oracle's support, but of course, everybody that sees an AA testing go wrong will believe you did not respect a proper methodology.
My question : I would like some pointers of people with more experience than me in data science to help me to :
- Express my problem in the clearest possible way to get the company that sells me this solution to take an interest in the issue here
- Understand if I am forgetting to check anything on my side to explain this discrepancy
The exact process that is used from our server to the final client is :

we send every x hours a list of a few hundred people to Responsys

Responsys is supposed to affect them in an independent and identically distributed way using random numbers generated with java.util.random.nextfloat()

then the list is filtered to remove undeliverable people, and each branch is sent the exact same email

Oracle is pretty laconic in his description, but I think the following lines from their documentation is supposed to describe an i.i.d. uniform distribution.

During launch, for each email recipient:
a. generate a random number ranging in [0, 1] using java.util.Random.nextFloat()
b. decide the bucket this random number is in
c. send the campaign in that bucket to the recipient


texts: Identifying subset of feature set to represent remaining features using pca or any other techniques

texts: How can you use PCA to identify a subset of feature set to represent remaining features in a data set? Suppose there are 10 features given by F={ f1,f2......f10}. How to identify a subset of F such that remaining features can be predicted using this subset? is PCA enough for this or are there any other techniques to perform this on a high dimensional data set?


texts:
For practicing linear regression, I am generating some synthetic data samples as follows.
First it generates 2000 samples with 3 features (represented by x_data). Then it generates y_data (results as real y) by a small simulation. i.e. by assuming a linear dependence model: imaginary weights (represented by w_real), bias (represented by b_real), and adding some noise.
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

#create some test data and simulate results
x_data = np.random.randn(2000,3)
w_real = [0.3,0.5,0.1]
b_real = -0.2

noise = np.random.randn(1,2000)*0.1
y_data = np.matmul(w_real,x_data.T) + b_real + noise

print(len(x_data))
print(len(y_data[0]))

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

x1 = x_data[:,0]
x2 = x_data[:,1]
x3 = x_data[:,2]
ax.scatter3D(x1, x2, x3, c=x3, cmap='Greens');

plt.show()

#actual implementation of liner regression
#compute y_pred, compare with y_data above etc etc
#assume more code here

exit()

I am trying to visualize above simulated samples (x_data and y_data) using matplotlib. I was able to plot x_data as displayed in following image. I would also like to visualize simulated results (y_data) on this plot, may be with a different color. Motivation behind it is to visualize relation between x & y. How can I plot it so?
Data dimensions:

x_data: $2000 \times 3$
y_data: $2000 \times 1$

Here are how sample data is displayed by above sample,

Here is an example of what I would like to achieve. The image shows two different hypothesis represented by straight lines, in my case I would like to draw a single line representing y_data.


texts: Multiclass classification with Neural Networks

texts:
Let’s suppose I wanted to classify some input as one of three categories using a simple neural network.
The output of my network are three columns (one for each possible category I assume) with values between 0 and 1. Moreover, the single rows add up to precisely one when adding the three columns together. Is is possible to interpret the output as the probability of my input belonging to each single category?


texts: Does Karl Pearson correlation indicate linear relationship between two variables ? Or it indicates nonlinear relationship or simply the correlation?

texts: Wikipedia and literature do not seem to convey correct interpretation of Karl Pearson correlation. A few authors interpret it as a linear correlation or association. To me it simply tells direction and extent of relation between two variables (one being categories based).


texts: Merge one label with one information for classification problem or multi-label classification

texts:
I want to build a model to support decision making in order to propose or not loan insurance to clients. Because sometimes clients asking loan and loan insurance have less chance to have their loan accepted by a bank and sometimes more chances.

There are three actors in the problem: a bank, a loaner applicant (someone who ask for a loan) and a counselor. The counselor studies the loaner application and if it has a good profile it will propose to him loan from banks that fits his profile. Then the application is sent to the bank but the bank could refuse the applicant (based on criteria we don't know).

The counselor has also to decide whether or not he will propose to the loaner applicant a loan insurance. I want to build a model for that decision.

The risk is that some banks reject loan applicant who accepts a loan insurance and other banks accept more applicants with a loan insurance. But there aren't rules regarding banks since some banks accept or reject applicants with loan insurance according of the profile of the applicants and the type of acquisition.

Thus, the profile of the applicant and the bank he is applying to can matter in their rejection from banks but all criteria influencing the decision are quite uncertain.

Even though it is a classification problem, in my dataset I don't have a good label for loan insurance proposal. I have a features that says if the insurance was proposed or not and to less than 1% the insurance was proposed. I have a label that says if the clients application for a loan was accepted or not.

Thus, the data I have is former applicants profile - and banks who propose loan according to what the applicant wants - and if they were accepted or not by the bank and if they wanted a loan insurance or not.

I thought of combining the label with the information but I don't really know how or maybe doing multi-label classification but also I don't really know if it does fit to the problem.


texts: Google data studio: How to order a dimension?

texts:
I'd like to visualize the number of bookings per month, but I can't understand how to order the months (they are a dimension and not a metric, maybe that's the problem?).
When I try to do so it asks me the type of aggregation but I need no aggregation to do so. Am I missing something?


texts: RL Sutton book, initial estimate of q*(a) for 10 arm testbed

texts:
The Sutton book does not mention what the initial estimate is for q*(a) before the first reward is received.  In this code repo that seems to go along with the book:  Sutton code repo

They have initialized it with 0 per snippet below:

def __init__(self, kArm=10, epsilon=0., initial=0., stepSize=0.1, sampleAverages=False, UCBParam=None,
                 gradient=False, gradientBaseline=False, trueReward=0.):

But the explanation for Figure 2.1 that shows the distribution of rewards for the 10 arms of the bandit says,

  Figure 2.1: An example bandit problem from the 10-armed testbed. The
  true value q ⇤ (a) of each of the ten actions was selected according
  to a normal distribution with mean zero and unit variance, and then
  the actual rewards were selected according to a mean q ⇤ (a) unit
  variance normal distribution, as suggested by these gray
  distributions.

So should I initialize instead with np.random.randn()?

Edit:  The distribution


texts: Which supervised learning algorithm to classify customers buying items

texts:
I am very new to machine learning, and it's hard for me to know in what direction I should go. Thanks in advance for your help.

I have a set of data showing what customers bought:

A customer may have bought 1 or more items
Each item is discribed by :

A quality level which is an integer between 1 and 7
Its content from a first point of view (the item has x1% of content 1, x2% of content 2, ...)
Its content from a second point of view (the item has xA% of content A, xB% of content B, ...)

A customer says that he belongs to the group G1, G2 or G3

I would like to be able to guess the group a new customer belongs to, given the items he bought.

What algorithm should I ideally use ?
What algorithm would you recommend if I can only use numpy, without scipy or scikit (it's on a server which only has numpy) ?

Thank you very much

Edit : I'm adding the appropriate words to the title and to the tags


texts: Why does my training loss oscillate while training the final layer of AlexNet with pre-trained weights?

texts:
I am working on texture classification and based on previous works, I am trying to modify the final layer of AlexNET to have 20 classes, and train only that layer for my multi class classification problem.
I am using Tensorflow-GPU on an NVIDIA GTX 1080, Python3.6 on Ubuntu 16.04.
I am using the Gradient Descent Optimiser and the class Estimator to build this. I am also using two dropout layers for regularization. Therefore, my hyper parameters are the learning rate, batch_size, and weight_decay. I have tried using batch_size of 50,100,200,weight_decays of 0.005 and 0.0005, and learning rates of 1e-3,1e-4,and 1e-5. All the training loss curves for the above values follow similar trends.

My training loss curve does not monotonically decrease and instead seems to oscillate. I have provided a tensorboard visualization for learning rate=1e-5, weight decay=0.0005, and batch_size=200.

Please assist in understanding what went wrong and how I could possibly rectify it.
The Tensorboard Visualization for the case I specified

  # Create the Estimator
  classifier = tf.estimator.Estimator(model_fn=cnn_model)
  # Set up logging for predictions
  tensors_to_log = {"probabilities": "softmax_tensor"}
  logging_hook = tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=10)
 # Train the model
  train_input_fn = tf.estimator.inputs.numpy_input_fn(x={"x": train_data},y=train_labels,batch_size=batch_size,num_epochs=None,shuffle=True)
  classifier.train(input_fn=train_input_fn, steps=200000, hooks=[logging_hook])
  # Evaluate the model and print results
  eval_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": eval_data},
  y=eval_labels,
  num_epochs=1,
  shuffle=False)
  eval_results = classifier.evaluate(input_fn=eval_input_fn)
  print(eval_results)

#Sections of the cnn_model
 #Output Config
 predictions = { "classes": tf.argmax(input=logits, axis=1),# Generate predictions (for PREDICT and EVAL mode)
"probabilities": tf.nn.softmax(logits, name="softmax_tensor")}  # Add `softmax_tensor` to the graph. It is used for PREDICT and by the `logging_hook`.
if mode == tf.estimator.ModeKeys.PREDICT:
  return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)

 # Calculate Loss (for both TRAIN and EVAL modes)
  onehot_labels = tf.one_hot(indices=tf.cast(labels,tf.int32),depth=20)
  loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=logits)

#Training Config
  if mode == tf.estimator.ModeKeys.TRAIN:
    optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
    tf.summary.scalar('training_loss',loss)
    summary_hook = tf.train.SummarySaverHook(save_steps=10,output_dir='outputs',summary_op=tf.summary.merge_all())
    train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op,training_hooks=[summary_hook])

  # Evaluation Metric- Accuracy
  eval_metric_ops = {"accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])}
   print(time.time()-t)
   tf.summary.scalar('eval_loss',loss)
   ac=tf.metrics.accuracy(labels=labels,predictions=predictions["classes"])
   tf.summary.scalar('eval_accuracy',ac)
   evaluation_hook= tf.train.SummarySaverHook(save_steps=10,output_dir='outputseval',summary_op=tf.summary.merge_all())
  return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops,evaluation_hooks=[evaluation_hook])


texts:
Here is part of my code:

class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(2, 1,  bias=False)
        self.linear2 = nn.Linear(1, 2,  bias=False)

    def forward(self, x):
        z = self.linear1(x)
        y_pred = self.linear2(z)

        return y_pred, z

model = SimpleNet().cuda()

for epoch in range(1):
    model.train()
    for i, dt in enumerate(data.trn_dl):
        optimizer.zero_grad()
        output = model(dt[0])

        loss2 = 0
        for j in range(0,len(output[0])):
            l1 = torch.autograd.grad(output[0][j][0], output[1], create_graph=True)[0][j]
            l2 = torch.autograd.grad(output[0][j][1], output[1], create_graph=True)[0][j]
            loss2 = loss2 + abs(torch.sqrt(l1**2+l2**2)-1)
        loss1 = F.mse_loss(output[0], dt[1])
        loss = loss1+loss2
        loss.backward()
        optimizer.step()
    if epoch%100==0:
        print(loss1,loss2,loss)

So I need the gradient of the output layer with respect to some node (this is a simple example, the real one has more layers in between), which I calculate using torch.autograd.grad(output[0][j][0], output[1], create_graph=True)[0][j]. However the way I do it now requires that for loop, over each element of the batch which is very slow. Is there a way to take this gradient all at once for a batch? Thank you!


texts: Role of mode in bottleneck function in unet network

texts:
I have come across the script that belongs to a person in kaggle. The snippet is given below.

 def bottleneck(x,filters_bottleneck,mode='cascade', depth=6,kernel_size=(3,3),activation='relu'):
   dilated_layers = []
   if mode == 'cascade':
      for i in range(depth):
        x = Conv2D(filters_bottleneck,kernel_size,activation=activation,padding='same',dilation_rate=2**i)(x)
        dilated_layers.append(x)
      return add(dilated_layers)
  elif mode == 'parallel':
     for i in range(depth):
       dilated_layers.append(Conv2D(filters_bottleneck,kernel_size,
                            activation=activation,padding='same',dilation_rate=2**i)(x))
       return add(dilated_layers)

To understand what the function bottleneck does, A pictorial explanation is given below. The highlighted portion is the bottleneck. This part of the network is between the contracting and expanding paths.

The function bottleneck accepts a parameter mode. Am confused with the parameter mode. Is the parameter mode part of the vocabulary of deep learning, if so, can you help me by providing additional resources to understand. And the same applies to the cascade in mode parameter too.


texts: Extracting embeddings of Categorical Variables

texts:
I am trying to build a regression model, for which I have a nominal variable with very high cardinality. I am trying to get the categorical embedding of the column.

Input:

df["nominal_column"]

Output:

the embeddings of the column.

I want to use the op of the embedding column alone since I would require that as a input to my traditional regression model. Is there a way to extract that output alone.

P.S I am not asking for code, any suggestion on the approach would be great.


texts: Understanding cosine distance with word vectors

texts:
I'm a new DL4J user, and I'm running all the works of Shakespeare through a Word2Vec neural net. I've got a pretty basic question about how to understand the results so far. In the below example, there's an obvious association with the "ing" in king and the "ing" in other words that probably don't have much to do with king. Am I missing something about how a word2vec formula uses the characters inside the words it is mapping? Or is my net just really untrained?

Also, what does the cosine distance between those example words say to you about the results, if anything? Thank you for your advice!

   List<String> abc = vec.similarWordsInVocabTo("king", 0.8); //80% similar
   System.out.println(abc);

   String[] words = {"woman", "king", "boy", "child", "human"};
   for (String word : words) {
       System.out.println(vec.similarity("man", word));
   }

Output - Similar words to king:

[taking, drinking, kingly, picking, waking, singing, wringing, knight, feigning, beginning, ink, thinking, kin, knocking, making, bringing, knowing, lingring, winking, neighing, king-, kings, asking, stinking, king, liking]

Output - Vector similarity between "man" and woman, king, boy, child, human:

woman:   0.8305895924568176
king:    0.00203840178437531
boy:     0.2974374294281006
child:   0.4752597510814667
human:  -0.10414568334817886


texts: Why is my loss function for DQN converging too quickly?

texts:
I'm still relatively new to deep learning and am experiencing an issue that I can't seem to find a solution/explanation for. I've developed a DQN model in tensorflow, as described by DeepMind, and am attempting to train it on a game that I wrote myself. From what I can see, the only difference between the DeepMind model and the model I've implemented is that I am not using a target network. From my understanding of DQN, I don't think this has an impact on the issue that I am facing.

Issue:
For some reason, the loss function seems to converge too quickly before the network has learned anything valuable.

Loss Functions: I've attempted several loss functions and optimizers with the same result. Initially, I attempted using an L2 loss function. Unfortunately, this caused my gradients to explode. I tried clipping the gradients between [-1,1]. Under this condition, the loss functions still converged but did so relatively quickly without the network learning anything valuable. I've looked more into the issue of exploding gradients and realized that the DeepMind describes loss clipping as a means of stabilizing the network. My understanding is that the paper essentially describes using a Huber loss function instead of an L2 loss. After implementing a Huber loss, the loss function still converges but does so more quickly.

My Impression: My impression of the issue is that the the network is falling into a local minimum and is unable to escape. However, from reading more into deep networks, it seems that falling to local minimums is not an issue given the large number of parameters present in the model.

Reward Function: Also I have some doubts about the reward function I am using and believe it may be causing this issue. The game's score depends on the length of the survival time. In order to account for this my reward function assigns reward by taking the length of survival time into account. The way this is done is by first collecting a game play iteration of experiences while playing the game. However, instead of the experiences noted as [state,action,reward,state], they are collected as [state,action,timestamp,state]. Note that these timestamps are collected in descending order. For example, if a game iteration survived a total of 15 seconds, then the earlier states are given timestamps closer to 15. Meanwhile later states, which lead to the inevitable termination, are given timestamps closer to zero. At this point, I apply a shifted and scaled sigmoid function to these timestamps to generate the reward for each experience. This way, states that lead to termination will be given values ranging between [-1,1) while all other states are given rewards of approximately 1. Note that I am still using experience replay to break the correlation between experiences.

Relevant Code:dist_cnn.py,dist_tf.py. I am using a distributed tensorflow model. Please excuse the any messy code.

If you have any insight into what maybe causing this, please let me know!


texts: Understanding LSTM input shape for keras

texts:
I am learning about the LSTM network. The input needs to be 3D. So I have a CSV file which has 9999 data with one feature only. So it is only one file.

So usually it is (9999,1) then I  reshape with time steps 20 steps

timesteps = 20
dim = data.shape[1]
data.reshape(len(data),timesteps,dim)

but I am getting following error

  ValueError: cannot reshape array of size 9999 into shape (9999,20,1)

and the input in LSTM

model.add(LSTM(50,input_shape=(timesteps,dim),return_sequences=True, activation="sigmoid"))


texts:
Quite new to NLP and especially NER. I'm trying to train a NER model on a custom dataset. This is a dataset of houses for sale. As part of the entities I'm training the model to extract are reference numbers. These are of variable length (but usually between 4-9) and look like G55L7 or LPP01Z1-32.

How can I give these entities a new "POS tag", as from what I'm aware of, I can't find any in SpaCy's default list that would match these?

Ideally, I'd like to train this alongside a pre-existing NER model so that I can also extract ORGs which SpaCy already has support for.


texts: How to plot number of Trees and OOBs score with Grid Search

texts:
I searched to find the answer but I don´t find something with Grid Search.
I create a random forest and gradient boosting regressor with grid search. Now I want to make a visualization to see if the number of trees is alright.
# Grid-Search erstellen auf der Basis eineer Random Search
param_grid = {
    'bootstrap': [True],
    'max_depth': [50, None],
    'max_features': ['auto', 50],
    'min_samples_leaf': [5],
    'min_samples_split': [2],
    'n_estimators': [100, 150, 200, 250, 300, 350]
}

# Base-Model zum Verbessern erstellen
rf_v1 = RandomForestRegressor(random_state=42)

# Grid-Search initisieren
rf_grid_search_v1 = GridSearchCV(estimator = rf_v1, param_grid = param_grid,
                           scoring = 'neg_mean_absolute_error', cv = 3,
                           n_jobs = -1, verbose = 2)

# Die Grid-Search für das Model durchführen
rf_grid_search_v1.fit(X_train_v1, y_train_v1)

But how to plot the splits of trees with gridsearch? Is there any way to make an RF/xgb trees split visualization?
I am really grateful if anybody has a tip for me.


texts: Preparing, Scaling and Selecting from a combination of numerical and categorical features

texts:
I'm currently working on the Titanic dataset from Kaggle. The features consist of both numerical and categorical variables and I've also engineered a few categorical variables using original features. I want to know what might be the best practice in terms of encoding the categorical variables and scaling features.

ENCODING

I know there are a few ways to encode categorical variables including:

Ordinal Encoding
One Hot Encoding
Binary Encoding
Hash Encoding

I know that Ordinal Encoding is undesirable sometimes as it makes the assumes that there's an ordered relationship between feature classes. I haven't gotten around to trying Binary and Hash encoding yet, and have done most of my work using One Hot Encoding.

Now I run into another issue while using OHE. I have a pipeline that handles data preparation, feature selection/extraction and prediction that I run grid search on. So what happens is for different folds, the number of features generated through OHE are different as in certain folds one or more features' classes are likely being missed. I wanted to ask how this impacts performance and is there any other way I can go about OHE to avoid this? One way I was thinking is to handle it before data is split into a train and test split- would this work? Since in preprocessing like imputation the fitting is to be done using only train data and then used to transform the validation and test data, I've constructed the pipeline as such (train test split -> fit using train data -> transform test/val data when predicting).

cat_pipeline = Pipeline([
    ('Selector', DataFrameSelector(cat_attribs)),
    ('Imputer', DataFrameImputer()),
    ('CategoryEncoder', cat_enc),
])

num_pipeline = Pipeline([
    ('Selector', DataFrameSelector(num_attribs)),
    ('Imputer', Imputer(strategy='median')),
    ('Log', log_apply()),
    ('MinMaxScaler', MinMaxScaler()),

])
full_pipeline = FeatureUnion([
    ('NumPipeline', num_pipeline),
    ('CatPipeline', cat_pipeline),
])

Prepare = Pipeline([
    ('AttributesAdder', CustomAttributesAdder()),
    ('Scaler', full_pipeline),
])

PrepSelPred = Pipeline([
    ("Prepare", Prepare),
    ("TopFeatureSelector", KernelPCA(n_components=3)),
    ("Classifier", RandomForestClassifier()),
])

SCALING

As you can see from above I use a MinMaxScaler and normalize my numerical attributes rather than standardize them. This is so that they match the scale of the OH features. I've read that standardization is preferable for ML algorithms but also that it is inadvisable on OH vectors. What would be the best approach here?

FEATURE SELECTION/EXTRACTION

I've tried both Feature Selection (using SelectFromKBest() and SelectFromModel()) as well as dimensionality reduction using PCA, though not together. Is one preferable to another, and do they work well in succession?

PREDICTING

Given the nature of the problem and the kind of features I am dealing with, are there any algorithms that are better suited to the task. If I stick with OHE for example, are there algorithms that perform better when the majority of features are sparse?


texts: How to tag Personal Name, Place and Organisation using Natural Language Processing or Fuzzy string matching

texts:
I have a data-set that has 2 columns in it:
Name                        Description
university of piitsburg    personal name
Jacob                      personal name
JEFF Snob                  organisation name
LG                         company name
ABC Pvt Ltd                personal name

The column Name can have name of Person, Company or Organisation name and the column Description tells in which category they fall ( Person, Company or Organisation name).
Most of the time there is data-defect as names are not correctly mapped. If you see the above example row 1 and row 5 are wrongly mapped.
So, somehow I want to use NLP on column Name to tag it to appropraiate description.
Please let me know if I am thinking in the right direction or I need to take anyother approach.
NLP NER has its own limitation like it would trear JEFF Snob has 2 different names and tag it differently.
Here is the code that I am looking to run on every row:
#Import all the required libraries.
import os
from nltk.tag import StanfordNERTagger
import pandas as pd

#Build NER tagger object
st = StanfordNERTagger(stanford_classifier)

#A sample text for NER tagging
text = 'srinivas ramanujan went to the united kingdom. There he studied at cambridge university.'

#Tag the sentence and print output
tagged = st.tag(str(text).split())
print(tagged)

Please let me know how to solve the problem statement using NLP or any-other logic?


texts: Neural network, Support Vector Machine or something else to classify into 7 groups


texts:
Transfer learning: Take a trained neural network and use it for a new classification task.

When we want to use transfer learning with a convolutional neural network, we don't have to use the same image size as input than the image size used for training. But if we change the input size we will have to re-train fully connected layers. See this post on Stackoverflow.

I don't undertand why changing input shape will not affect convolutional layer weight and why it will affect fully connected layer weights.

Tell me if my question is not understanble.


texts: Random Forest in R: Error in eval(expr, envir, enclos): object not found

texts:
I use the randomForest package in R but I'm getting the following error:

Error in eval(expr, envir, enclos) : object 'a' not found

The line that causes the problem is the following:

model = randomForest(class ~ a + b + c, data=train, ntree=100

I looked up the error and apparently it has to do with the fact that train does not contain a, but I manually checked train and it does contain a.

As a matter of fact, when I try to build a decision tree instead, it works just fine! (with the same formula and dataset)

model = rpart(class ~ a + b + c, data=train, method="class")

So I don't understand where this error is coming from. Any ideas?

EDIT: There seems to be a problem with the fact that I use bigrams for my text. If I just use unigrams, it works fine. If I use this code for example:

model = randomForest(class ~ `stack overflow`, data=train, ntree=100)

It returns the error named above: object 'stack overflow' cannot be found.


texts: How to classify neutral sentiments using BERT

texts:
We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also?

https://colab.research.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb

Actually, I want to know like what kind of changes do we need to make in the above notebook so that it can classify neutral sentiments also besides positive and negative.

Thanks in advance.


texts: Could we use image hasing techniques for images classification tasks?

texts:
I have read some articles about image hashing, and I would like to know if we could apply this technique for general purpose images classification tasks.

Especially I would like to know which could be the drawbacks in using image hashing for this kind of tasks.


texts: Deep reinforcement learning on changing data sizes

texts:
I have a game that I want to build a model that will learn to play the game. Yet, the environment output is two lists that represent the location and number of soldiers of the user and Opponent.

The lists lengths are changing with each step, so to know the input size to the network is a problem.

How can this be dealt with?


texts:
In orthogonal regression, we are trying to minimize the distance from each data point $(x,y)$ to the fitted model.

My question is, how come that there is a distinction between independent and dependent variables in orthogonal regression?

In my naive understanding, the fit we are trying to achieve does not distinguish between $x,y$ - we want to find the line that minimizes the sum of distance to each $(x_i,y_i)$.


texts: I am working to augment my data using Generative Adversarial Networks, I have used Deep Convolutional GAN's for this purpose but they are not learning the right data distribution, so please suggest me which GAN's to use for this purpose ?


texts: In Recommendation systems, Does we need to build each model for each product if we are using Logistic regression?

texts:
I am reading this paper wide and deep learning recommedation system paper and haven't understood one thing.

To serve the latency, they actually first get the 100 best apps according to user query and rank them according to the probability score.

Does this mean that while buidling the model we have to build individual model for each app?

Are there any other Deep learning frameworks similar to this ?


texts: Does self driving technology gain more from the data or the state-of-the-art algorithm?

texts:
When people start to figure the self driving cars will replace some vehicles on the road in the very near future, it somehow means the learner in the robot software could reach a very low empirical error to ensue the safety of passengers. The great deal of driving data collected and the deep reinforcement learning algorithms contribute mostly to the success of the self driving practice as we see it.

When it comes to many cases of our own practice, we yet cannot build up a predictive model to such a high accuracy to bring the huge difference to our own business.

My question is, when I saw the machine learning techniques can make revolution on the car driving due to its successful algorithms to implement behavioral cloning tasks well, what is the foremost reason for this success, the big training data set or the deep reinforcement learning techniques or any particular reason in self driving problems?

Furthermore, can this success be copied in most of the practical problems we solve in machine learning? In other words, if today's machine learning techniques can help a car drive by itself, how can I reach the same success when developing my own predictive models to boost my business? If we cannot, what are the constraints, not enough data in the particular case or not a smart model yet? The reason the self driving thrives can help answer this question.


texts: redundancy or functional dependency of two columns

texts:
Being a beginner in Data Analysis I ask you not to be much judgemental about this question.

I was tring to find if there is a standard way (function?) in Pandas module to identify redundancy of a column. For example, notably for categorical data types,  if a column is a function of another it could be considered as redundant and thereof could be disregarded without any loss of prediction power. I would prefer not to implement any heavy machinery myself, but use common knowledge methods, therefore is my question.

I guess correlation is a good metric, but it has a different meaning and it is not a good fit for two categorical columns.

Any answers would be much appreciated, cheers.


texts:
I have 102 observations. I made standard scale for dataset. I have found the accuracy training and accuracy testing values, but training score is 1.00 and testing score is -217.541. I have run with MLPRegressor score. What does negative accuracy testing score mean?

FileX_train, FileX_test,FileY_train,FileY_test = train_test_split(FileX,FileY,test_size=0.33, random_state=0)

sc = StandardScaler()
X_train = sc.fit_transform(FileX_train)
X_test = sc.transform(FileX_test)

sc = StandardScaler()
Y_train = sc.fit_transform(FileY_train)
Y_test = sc.transform(FileY_test)

from sklearn.neural_network import MLPRegressor
from sklearn.metrics import accuracy_score
mlp = MLPRegressor(solver='lbfgs',activation='tanh', hidden_layer_sizes=(100,), max_iter=1000000000, learning_rate='constant')
mlp.fit(X_train, Y_train.ravel())
print('Accuracy training : {:.3f}'.format(mlp.score(X_train, Y_train)))
print('Accuracy testing : {:.3f}'.format(mlp.score(X_test, Y_test)))

Results:

Accuracy training : 1.000
Accuracy testing : -217.541


texts: Open source alternative to google's Auto ML

texts:
Auto ML receives raw(ish) data and generates a supervised learning model from it. That is, it automates feature engineering and model selection/tuning.

What open source alternatives currently exist that when given raw data and told which variable to target, and will produce an ML model autonomously?


texts: Speed for different kernels in scikit-learn's SVM

texts:
I'm using scikitlearn in Python to create some models while trying different kernels. I was surprised to see that rbf was fit in under a second, whereas linear took a minute and poly took hours. Can anyone explain why to help me?

My code:

import numpy as np
from sklearn.svm import SVR
import matplotlib.pyplot as plt
from sklearn.metrics
import mean_squared_error, r2_score
f = open(datafile)
X = np.empty([0,1], dtype = int)
y = np.array([])
for i in range(100):
    my_lines = f.readline().split(" ")
    t = np.array([[int(my_lines[0])]])
    n = np.array([int(my_lines[1])])
    X = np.concatenate((X, t), axis =0 )
    y = np.concatenate((y, np.array([int(my_lines[1])])), axis = 0)
f.close()
svr_rbf = SVR(kernel='rbf', C=1e3, gamma=0.1)
svr_lin = SVR(kernel='linear', C=1e3)
svr_poly = SVR(kernel='poly', C=1e3, degree=3)
svr_rbf.fit(X, y)
svr_lin.fit(X, y)
svr_poly.fit(X, y)

The data:

0 22
1 23
2 23
3 23
4 25
5 24
6 25
7 26
8 26
9 30
10 29
11 29
12 30
13 30
14 30
15 31
16 29
17 28
18 29
19 31
20 30
21 30
22 31
23 31
24 30
25 31
26 31
27 30
28 29
29 31
30 33
31 31
32 32
33 30
34 30
35 30
36 29
37 30
38 29
39 29
40 27
41 28
42 28
43 27
44 27
45 28
46 28
47 29
48 29
49 28
50 28
51 29
52 27
53 27
54 28
55 28
56 28
57 30
58 32
59 31
60 30
61 28
62 28
63 30
64 27
65 27
66 28
67 27
68 29
69 33
70 39
71 35
72 34
73 29
74 30
75 28
76 28
77 29
78 27
79 26
80 25
81 17
82 0
83 0
84 29
85 21
86 20
87 18
88 19
89 19
90 19
91 18
92 17
93 18
94 20
95 19
96 20
97 19
98 18
99 18


texts:
I have a data set where customers who are non-delinquent with us have defaulted with others. So from the total data set, 30% are those who defaulted with others but 70% are those who are good with us and other others as well.
With the help of 30% customers who defaulted with others (being good/non-delinquent with us), can we identify the similar customers (matching in profile) from the remaining 70% customers who are currently good with us as well as with others, and also to any new customer?
This is required so that we can have an early indication that who are the customers (from remaining 70% & any new addition) most likely turn bad in near future?
I am not sure how to apply any machine learning algorithm (as we just have 1 class defaulted with others) to identify similar customers.


texts: Can I apply an independent test t on two sets of 30 samples each , even of the normality test fails

texts:
In the link
t-Test
 there is a nice explanation about the use of the t-test and how to observe the assumptions of the test: normality and equal variance. If the normality check fails (Shapiro test), it recommends to use the Mann-Whitney U test. However it also says " before turning to non-parametric controls you should consult a statistician. It might be the case that your sample data is large enough to apply parametric t-test although the normality criterion is violated"

I would like to know if 30 is large enough or I will have to go with the  Mann-Whitney U test?


texts: LSTM for prediction of next location step - help with standardization

texts:
I have a few questions regarding the topic and I hope someone might have experience with any of them.

What I am trying to do is train an LSTM network, whose input is a sequence of N steps in a XYZ space (i.e 3 features over N point per sample, each point is part of some coordinate in space) and i want to predict the next point in the same XYZ space.

Note: Not all samples are from the same DB, meaning the XYZ space varies between some samples/

My questions are:

Say I want to use min-max scaling, do I scale across ALL samples at once or scale per XYZ space? say i have 100 samples from XYZ_A and 50 from XYZ_B, do i take my min/max from 150 point of both space or not?
should I also scale my output labels? and if so should I use them in the initial scale or should I only scale on my train inputs and use that scaler on my outputs?
When I want to make a prediction after training, should I scale the data according to my training data?

OR since my features are relatively from the same domain should I skip scaling all together?


texts:
I have been trying to apply a simple neural network using keras to predict a sequence of numbers and the rule is if the input integer is odd it should be 4 and if its even it should be 2. Yet the neural network gets stuck at a 60% accuracy rate. Anyone know a solution to this?

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

from keras.layers.normalization import BatchNormalization

from sklearn.model_selection import cross_val_score
from keras.wrappers.scikit_learn import KerasClassifier

import numpy as np

def gen(x):
    if (x%2==0):
        return 2;
    else:
        return 4;

a = []
for i in range(1,100001):
    a.append([i,gen(i)])

a = np.array(a)

x = a[:,0:1]
y = a[:,1:2]

def MakeClassifier():
    network_classifier = Sequential()
    network_classifier.add(Dense(units=2,kernel_initializer="uniform",activation="relu",input_dim=1)) #Hidden Layer1 taking into account number of inputs(independant variables(x)
    network_classifier.add(BatchNormalization())
    network_classifier.add(Dense(units=1,kernel_initializer="uniform",activation="sigmoid"))#OutPutLayer

    network_classifier.compile(optimizer="adam",loss="binary_crossentropy",metrics=["accuracy"])#If multicategorical then categorical_crossentropy
    return network_classifier

classifier = KerasClassifier(build_fn= MakeClassifier , batch_size = 10 , epochs = 1000)

classifier.fit(x,y,epochs=100,batch_size=1000)

print(classifier.predict([[6],[7]])) #Should Predict 2 and 4


texts: Can VAEs be used to generate multivariate data?

texts:
Most of the tutorials online seem to use VAEs to generate images and use CNNs to generate data. I am working on a game with multivariate data consisting of character position and the character attributes to train a VAE.

I was wondering if it was possible because I have tried training the network with pytorch but I continue to get poor results with KL Divergence term decreasing to 0.

Any help would be really appreciated.


texts:
I'm learning how U-NET network works to do semantic segmentation.

I think I have understood everything but features. What are those image features?

I read that convolutional layers extract features from the images using their filters, but what are they? Are they corners? edges? colours?

I have read this article, "Finding Features", but I think I need more information about them.


texts: Different representations of dendrograms

texts:
I have a dendrogram represented in a format I don't understand:
(K_5:1.000030e+00,((K_1:2.000000e-05,(K_2:1.000000e-05,K_3:1.000000e-05):1.000000e-05):1.000000e-05,K_4:3.000000e-05)0.806:1.000000e+00):0.000000e+00;
I am not sure how to interpret the above.
It is an output of hierarchical clustering.
K_1, K_2, K_3, K_4, K_5 are the data points.
I have other dendrograms represented in the following format:
[x_1,x_2,x_3,x_4,x_5] (we start with one big cluster and split a cluster at each step)
[x_1,x_2][x_3,x_4,x_5]
[x_1,x_2][x_3,x_5][x_4]
[x_1][x_2][x_3,x_5][x_4]
[x_1][x_2][x_3][x_5][x_4]
I want a way to convert between these two representations.


texts: how to see the correlation graph when there is lot of variables?

texts:
my variable has more then 4000

my corr() graph looks like this

is there better way to look at this graph more clearly?


texts: Are all 110 million parameter in bert are trainable

texts:
I am trying to understand are all these 110 million parameters trainable of bert uncased model. Is there any non trainable parameters in this image below?
By trainable I understand they are initialized with random weight and during pretraining these weights are backpropagated and updated.


texts: XGBoost most important features appear in multiple trees multiple times

texts:
I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 classes. I have now about 300 features in the model.

Then I take an output model and count for each feature

In how many trees a feature was present
In how many splits a feature was present

Then I sort the features from the ones with most trees to least trees.
I get some results that I am not sure of. Features at the top of the list with most trees and splits basically appear in each single xgboost tree multiple times. For example in xgboost with 100 rounds and colsample_bytree=1.0 and max_depth=6 I would see a feature A appear in 100 trees and in ~400 splits. Feature B appears in 98 trees and 350 splits etc... Basically it seems that all my trees are based on all the same same top features in different configurations.

Question:

Does it mean:

It normal
Do those features overfit my model
I tried to force model to take other features by decreasing colsample_bytree or colsample_bylevel and it helps but somewhat, but model performance does not dramattically improve.

Any other suggestions?

Update Observations (Feb 2019)

The features are continuous not categorical
When I sort all features based on their total gain (sum of gain in all nodes that split on the feature), the features that appear in all trees multiple times can have highest total gain or quite low total gain. There is not rule here.


texts: Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation

texts:
I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode:

for e in  epochs:
    for i in idx:
     loss_batch = 0
     for b in batch_size:
         input = data[i+b]
         output = nnlm(input)
         loss_window = neg_log_lhd(output, target)
         loss_batch += loss_window

     loss_batch /=batch_size
     loss_batch.backward()

idx here is the index of the first input word in the batch. Loss is accumulated over the whole batch (if batch_size=128, it means there are 128 sliding windows for which the loss is calculated) and backpropagated before moving to the next one.

The model seems to converge, judging by the curve of the loss function, but I'm concerned about the vector representation of words. I computed cosine similarity for 4 different words: 'bilbo','baggins', 'gandalf', 'thorin'. I expected the similarity between the first two to be very high, as they are used interchangeably, or very frequently together. Nevertheless, it starts with -0.042 and is -0.04 after 40 epochs (~400 batches/epoch). Obviously something's wrong here.

What could possibly be the reason for the situation when the model converges, but the distance between word embeddings doesn't change?


texts: r2 for regression models it is a score or error?

texts: some places I have seen it is called as score and some other place as error. Lets suppose r2=0.83 means that score = 83% and Error= 17% or vise versa


texts: Understanding portfolio-level risk models

texts:
I have a tremendous amount of experience training supervised machine learning models. However, I recently became a data scientist at a small financial services company, and I've been asked to build some sort of model that assesses portfolio-level risk. I'm not sure how to go about this task. Is this a supervised learning problem? It doesn't sound like one.

I used to work as a software/data engineer at a different financial services company, and while there I was briefly exposed to the concept of Basel credit risk management. Is Basel used for assessing portfolio-level risk? As far as I could tell (I could be wrong), Basel seemed to aggregate charge-off predictions by running every single client through each of the company's relevant credit risk models (which were trained in a supervised fashion). If my memory is right, how does this aggregation work? Is it effective?


texts: regsubsets function in R for feature selection

texts:
I am trying to implement feature selection using regsubsets function.
I would like to know which algorithm is used in the background in order to build the model when we use regsubsets?


texts: Does high accuracy metrics with small (but equally sampled) dataset means a good model?

texts: I have been training my CNN with 200 images per class for a classification problem.  There problem is a binary classification one. And with the amount of test data ( 25 per class) I am getting good accuracy, precision and recall values. Does that mean my model is actually good?


texts: LSTM forecasting on multivariate time series

texts:
I'm new to RNNs and LSTM and would like some direction with a problem I have. I have a data set containing system metrics (like CPU utilization, disk operations, memory use) of an AWS EC2 instance with a total of 7 columns and around 8000 rows. Each row also has a timestamp with a 5 min interval between each row.

I want to build a LSTM model to forecast the performance for let's say the next hour based on my data. What would be the best approach for solving a problem like this? I know this can be done in many different ways but I would really appreciate some input how to go about this.


texts: i am trying to one hot the inputs for the encoder and decoder layers in lstm? is this logical?

texts:
the follwoing code is a generator I am using to pass the training, for this generator i am passing questions and answers. the first 200 Q and A I am reserving for testing. The relevant part of the code is after the padding line for ans_pad and ques_pad. Should I also one hot this data. I am trying to build a chatbot, but so far I ma getting very poor results and I feel like the issue lies in this part. I one hoted the output data, is it logical to do the same for the input as well? I tried doing it but it led to errors in my input layer of the model

def generator(questions,answers, batch_size=32):
num_samples = len(questions)
while True:
    for offset in range(200, num_samples, batch_size):
        question_samples = questions[offset:offset+batch_size]
        answer_samples = answers[offset:offset+batch_size]
        ques_train = []
        ans_train = []
        for i in question_samples:
            ques_train.append(i)
        for i in question_samples:
            ans_train.append(i)
        ques_train = np.array(ques_train)
        ans_train = np.array(ans_train)
        ans_pad = sequence.pad_sequences(ans_train, maxlen = maxLen, dtype = 'int32', padding = 'post', truncating = 'post')
        ques_pad = sequence.pad_sequences(ques_train, maxlen = maxLen, dtype = 'int32', padding = 'post', truncating = 'post')
        encoder_input_data = np.array( ques_pad)
        decoder_input_data = np.array( ans_pad)
        for i in range(len(ans_train)) :
            ans_train[i] = ans_train[i][1:]
        padded_answers = sequence.pad_sequences( ans_train , maxlen=maxLen , padding='post' )
        onehot_answers = utils.to_categorical( padded_answers , vocab_size )
        decoder_output_data = np.array( onehot_answers )
        yield(encoder_input_data,decoder_input_data),decoder_output_data

should I one hot the input question and answers, or just the output?


texts: How to use Non Negative Matric Factorization (NMF)'s transform method to project topics on new text data

texts:
I have created a NMF topic model in python the code snippet for which is as follows:

def select_vectorizer(req_ngram_range=[1,2]):
    ngram_lengths = req_ngram_range
    vectorizer = TfidfVectorizer(analyzer='word', ngram_range=(ngram_lengths), stop_words='english', min_df=2)
    #print("User specified custom stopwords: {} ...".format(str(custom_stopwords)[1:-1]))
    return vectorizer

vectorizer = select_vectorizer([2,5])
X = vectorizer.fit_transform(new_review_list)

clf = decomposition.NMF(n_components=20, random_state=3, alpha = .1).fit(X)
vocab = vectorizer.get_feature_names()
print_top_words(clf, vocab, num_top_words)

which created 20 topics like the following:

Topic #0:
[u'blocks available', u'delivery blocks available', u'notifications blocks', u'notifications blocks available', u'new blocks', u'know blocks available', u'new blocks available', u'know blocks', u'open blocks available', u'available work', u'zero blocks', u'like blocks', u'notification blocks', u'day blocks', u'slow blocks', u'10 blocks', u'option set', u'logged 10', u'notification blocks available', u'day blocks available']
Topic #1:
['something', 'something'] and so on..

Now I want to supply new text data to this fitted NMF model such that the model tells us which topic this new text belongs to. I have searched a lot, somebody suggested using "transform", but I do not know how to use it. For eg: if I have a new text, like:

new_feedback = "This is a new tester feedback"

Then how do I find which topic out of the 20 topics created earlier, does this new_feedback fall into? Please help!!

UPDATE: On trying to do what's mentioned in the below comment, I receive the following error:

clf.transform(vectorizer.transform(new_feedback))
  File "/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 1380, in transform
    X = super(TfidfVectorizer, self).transform(raw_documents)
  File "/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 884, in transform
    "Iterable over raw text documents expected, "
ValueError: Iterable over raw text documents expected, string object received.


texts: Optimizing parameters for CNN autoencoder based on training and validation loss

texts:
I have designed an autoencoder with a encoder and decoder consiting of 2D convolutational layers (the input are 40'000 2D images). I train the autoencoder using adam optimizer. The autoencoders has the following hyperparameters which I would like to tune (in brackets are my default values):

Number of layers in encoder and decoder (I start with 2 in decoder and encoder)
Filter size for convolutional layers (I start with 32 and 64)
Convolutional kernel size (I start with 3x3)
Stride size (I start with 2x2)
Dropout (I start with 0.25 after each layer)
Learning rate (0.001)
learning_rate_decay (0)
Latent dimension (I start with 8)
Number of units in the dense layer (layer before creating latent space, I start with 16)
Batch size (I start with 128)

One possibility would be to use just grid or random search but this is very inefficient and takes a long time with so many hyperparameters. Instead, I would like to observe the training and validation loss (using tensorboard) and adjust the parameters accordingly. For example when observing the training and validation loss there could be overfitting or underfitting (or also an increase in loss etc.).

Are there some general rules or hints how the hyperparameters could be adjusted based on the observed losses or based on other criterions?


texts: How to understand what each layer is learning in a Deep learning neural network?

texts:
In a recent answer I read on Stack Exchange, I read about a possible way to understand more clearly what happens in each hidden layer of a neural network.

Here's the excerpt-

  You should watch what makes neuron activated in each layer depending
  on the input. As you know each neuron will be activated (once the DNN
  is trained) for specific input combinations. By visualizing that you
  can get an idea about what exactly each layer has learned in term of
  high-low level features.

Source - High-level features of a neural network

I wanted to know if there are any papers who have tried doing this (links would be really helpful). Meanwhile are there any other ways to understand what happens in each hidden layers?


texts: Overfitted model produces similar AUC on test set, so which model do I go with?

texts:
I was trying to compare the effect of running GridSearchCV on a dataset which was oversampled prior and oversampled after the training folds are selected. The oversampling approach I used was random oversampling.

Understand that the first approach is wrong since observations that the model has seen bleed into the test set. Was just curious about how much of a difference this causes.

I generated a binary classification dataset with following:

# Generate binary classification dataset with 5% minority class,
# 3 informative features, introduce noise with flip_y = 15%

X, y = make_classification(n_samples=5000, n_features=3, n_informative=3,
                            n_redundant=0, n_repeated=0, n_classes=2,
                            n_clusters_per_class=1,
                            weights=[0.95, 0.05],
                            flip_y = 0.15,
                            class_sep=0.8)

I split this into 60/40% train/test split and performed GridSearchCV with both approaches on a random forest model. Ended up with following output based on best_estimator_ from both approaches:

Best Params from Post-Oversampled Grid CV:  {'n_estimators': 1000}
Best Params from Pre-Oversampled Grid CV:  {'classifier__n_estimators': 500}
AUC of Post-Oversampled Grid CV - training set:  0.9996723239846446
AUC of Post-Oversampled Grid CV - test set:  0.6060618701968091
AUC of Pre-Oversampled Grid CV - training set:  0.6578310812852065
AUC of Pre-Oversampled Grid CV - test set:  0.6112671617024038

As expected, the Post-Oversampled Grid CV AUC is very high due to overfitting. However, evaluating both models on the test set lead to very similar results on AUC (60.6% vs 61.1%).

I had two questions. Why is this observed? I didn't assign a random_state to any of these steps and retried it many times, but still end up with the same results. In such a case, what becomes the better model to progress with since they're producing similar results on test set?

For oversampling and handling it through the pipeline, I made use of imblearn:

# imblearn functions
from imblearn.over_sampling import RandomOverSampler
from imblearn.pipeline import Pipeline as Imb_Pipeline

Happy to share my code if needed. Thanks.


texts: Orange Geo Map does not show background map

texts:
We are running Orange with our students and most of them are having problems with running the Geo Map widget. It does not show any background map. In the bottom an error message is displayed: "Cannot fetch map from the Internet. Displaying only cached parts."
I have found this Github page, but it does not explain how to update or add any JSON files that might be missing:
https://github.com/biolab/orange3-geo/blob/master/CONTRIBUTING.md
What do we have to do to get the background maps to appear again in the Geo Map widget?


texts: What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not?

texts:
I spent the past year learning Python. As a person who thought coding was impossible to learn for those outside of the CS/IT sphere, I was obviously gobsmacked by the power of a few lines of Python code!

Having arrived at an intermediate level overall, I was pretty proud of myself as it greatly expands my possibilities in data analysis and visualization compared to Excel (aside from the millions of other uses there are for Python).

Purely in terms of data analysis and visualization:

what does approaching the same data set with pandas/matplotlib/seaborn/numpy bring to the table as opposed to using Tableau?

(sidenote: I was greatly disappointed to see all my hard-earned Python data wrangling skills were available in such a user-friendly GUI... :'( )


texts: How to draw multiple matrices (with grid, custom color per cell) in 3D with raycast?

texts:
I would like to draw multiple matrix with ray-casting in 3D.
More specific like this (source)
I have seen similar figure in some paper (I forgot which one). I wonder how they can draw like this. If there are specific tool or Latex Source code that I can customize to draw my own.