texts: Converting video into frames using openCV

texts:
I am converting video into video-frames using the given code

converting video into frames

import cv2
import numpy as np
import os

# Playing video from file:
cap = cv2.VideoCapture('/home/administrator/Desktop/Projects/gait-     recognition/video/p008-n05.avi')
frames = []
try:
        if not os.path.exists('data'):
        os.makedirs('data')
except OSError:
    print ('Error: Creating directory of data')
currentFrame = 0
while(True):
# Capture frame-by-frame
    ret, frame = cap.read()
   # Saves image of the current frame in jpg file
   name = './data/frame' + str(currentFrame) + '.jpg'
print ('Creating...' + name)
cv2.imwrite(name, frame)
frames.append(frame)
# To stop duplicate images
currentFrame += 1
if ret == False:
   break
# When everything done, release the capture
cap.release()

    imgs = np.array(frames)

which is giving me output as images in the form of a single multi-dimensional array. This is required here as

Create features from input frames in shape (TIME, HEIGHT, WIDTH, CHANNELS)

spatial_features = net_pose.feed_forward_features(imgs)

When I feed the images as video_frames in this, I get this error as follows:

Please tell me where and what modifications are required to get the desired tensor.


texts:
I happily created my first NN and performed hyperparameter optimization through GridSearchCV. I just don't know what to do next.
Do I have to fit it again with the best parameters GridSearchCV() revealed? is there an elegant way to do so?
Otherwise, how to proceed?

def create_model(...

    model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['accuracy'])
    return model

model = KerasRegressor(build_fn=create_model, verbose=0)

> hypparas
{'batch_size': [2, 6], 'optimizer': ['Adam', 'sgd'], 'opt_par': [0.5, 0.8]}

GridSearchCV(estimator=model
                    , param_distributions=hypparas
                    , n_jobs=1
                    , n_iter=20
                    , cv=3
                    )

grid_result = grid_obj.fit(X_train1, y_train1, callbacks = [time_callback])

print("Best: %f using %s" %  (grid_result.best_score_, grid_result.best_params_), "\n")

means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']

for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

    Best: -0.941568 using {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 2}

    -1.725617 (0.620383) with: {'optimizer': 'Adam', 'opt_par': 0.5, 'batch_size': 2}
    -1.595137 (0.224487) with: {'optimizer': 'sgd', 'opt_par': 0.5, 'batch_size': 2}
    -0.941568 (0.149151) with: {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 2}
    -1.338372 (0.523434) with: {'optimizer': 'sgd', 'opt_par': 0.8, 'batch_size': 2}
    -1.094907 (0.121018) with: {'optimizer': 'Adam', 'opt_par': 0.5, 'batch_size': 6}
    -1.588476 (0.569475) with: {'optimizer': 'sgd', 'opt_par': 0.5, 'batch_size': 6}
    -1.443133 (0.342028) with: {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 6}
    -1.275414 (0.331939) with: {'optimizer': 'sgd', 'opt_par': 0.8, 'batch_size': 6}


texts: How to use neural network's hidden layer output for feature engineering?

texts:
I was wondering how can we use trained neural network model's weights or hidden layer output for simple classification problem, and then use those for feature engineering and implement some boosting algorithm on the new engineered features.

Suppose,if we have 100 rows with 5 features (100x5) matrix.

data:
      X                 Y
x1,x2,x3,x4,x5          y1  y2
0,1,2,3,4               0   1
3,2,5,6,4               1   0

Network:
2 layers, input and softmax output, compile using cross entropy  .

Can we utilise trained weights or hidden layer output of above network and use it for feature engineering on original dataset and then apply some boosting algo on modified dataset and will it increase accuracy ?


texts: I have data of some movies and their subtitles.I want to classify them based on their ratings

texts: I will convert the subtitles into vectors and use them as features to classify the movies into different categories based on their ratings.The problem that I am facing is my feature vector is much much larger compared to the number of examples I have. I would like to know what should the size of my tailing data set be to use LDA,PCA,SVM and naive Bayes.Would 10 movies per category be sufficient?


texts: Predict real world data after modelling with scaled features

texts:
I trained and test a model with scaled features.
Now, I want to predict a single real world sample. If I have one sample alone, I can't scale it to fit into the model like I did with the test data.
I imagine it's a pretty common situation.
What are the steps I need to do to predict this single sample?


texts: How do I encode time in high dimensional space?

texts:
I have a dataset of form text, text, category, category, time, text and I would like to apply the attention mechanism to it. This requires that all inputs be in the same vector space. I am using a particular encoding method (from BERT) for the text-type data and I can build a custom trainable embedding for the category features. However, I don't have a good way of embedding time data.

Currently, my time feature is normalised on [0,1], and represents when over the time period (one year) the post was created. Naively, I would split this up into month, day of week, and time of day features to do feature engineering, but I don't have a good way to embed this in very high dimensional space (say, 500+ dimensional space).

What's the best option here? I would like to avoid tiling or repeating the same feature set to reach the requisite dimensionality -- is there a better way? I could put a trainable embedding layer on top of those three features, too, but this seems suboptimal.


texts: Tools for visualisation: making a video of a sliding windows moving on the time series

texts:
I have a Neural Network (an autoencoder) that gets as the input a time-window of length M from a time series of length N, (M << N), and transforms it to another same dimensional time-window.

In purpose of visualization, I want to make a video that shows input of the NN in the top of the frame and the corresponding output in the bottom of the frame, runnig the NN on consecutive windows.

I was wondering if you could introduce me some tools or give me some hints on this.


texts: How to distinguish informative and non-informative feature - Feature importance?

texts:
I have a dataset with 5K records focused on binary classification problem. I have more than 60 features in my dataset. When I used Xgboost, I got the below Feature Importance plot. However I am not sure how to find out whether all of these are informative?

Questions

1) Yes, I can select top 15/20/25 etc. But is that how it is done? Is there any minimum F-score that we ought to look for?

2) Or is it like I select top 10 features, check for accuracy and again add 2-3 features in every round and verify accuracy manually. Is this how it is done?

3) How would you people go about it? I tried with full dataset, the accuracy is only about 86%. when I tried with 15-20 features, it's only about 84. So manual feature selection is the only way to improve further?

Can you help me please?


texts: How predictions of level 1 models become training set of a new model in stacked generalization.

texts: In stacked generalization, if I understood well, we divide the training set into train/test set. We use train set to train M models, and make predictions on test set. Then we use the predictions as input of a new model. Thus, the new training set will have M features corresponding to the M models predictions. Finally, we use the last model to make the final predictions. First, is my understanding correct ? If so, how is it possible to use the last model to make predictions as it has different features.


texts:
I have a two-class prediction model; it has n configurable (numeric) parameters. The model can work pretty well if you tune those parameters properly, but the specific values for those parameters are hard to find. I used grid search for that (providing, say, m values for each parameter). This yields m ^ n times to learn, and it is very time-consuming even when run in parallel on a machine with 24 cores.

I tried fixing all parameters but one and changing this only one parameter (which yields m × n times), but it's not obvious for me what to do with the results I got. This is a sample plot of precision (triangles) and recall (dots) for negative (red) and positive (blue) samples:

Simply taking the "winner" values for each parameter obtained this way and combining them doesn't lead to best (or even good) prediction results. I thought about building regression on parameter sets with precision/recall as dependent variable, but I don't think that regression with more than 5 independent variables will be much faster than grid search scenario.

What would you propose to find good parameter values, but with reasonable estimation time?


texts: Feature selection using feature importances in random forests with scikit-learn

texts:
I have plotted the feature importances in random forests with scikit-learn. In order to improve the prediction using random forests, how can I use the plot information to remove features? I.e. how to spot whether a feature is useless or even worse decrease of the random forests performance, based on the plot information? The plot is based on the attribute feature_importances_ and I use the classifier sklearn.ensemble.RandomForestClassifier.

I am aware that there exist other techniques for feature selection, but in this question I want to focus on how to use feature feature_importances_.

Examples of such feature importance plots:


texts: Is there any domain where Spiking Neural Networks outperform other algorithms (non-spiking)?

texts:
I'm reading about reservoir computing techniques like Echo State Networks and Liquid State Machines. Both of the methods involve feeding inputs to a population of randomly (or not) connected spiking neurons, and a relatively simple readout algorithm that produces the output (e.g. linear regression). The neuron population weights are either fixed, or trained via a Hebbian-like local activity rule like STDP.

These techniques perform well when modelling multi-dimensional inputs that have significant temporal components. However, computing the spiking neuron membrane potentials involves differential equation integration and can be computationally expensive.

Are there any examples of where the additional computational complexity of reservoir computing techniques is outweighed by gains in a prediction or classification task?

For example, are there any cases of SNN techniques outperforming comparably complex architectures based on RNNs, ANNs, SVMs, DNNs, CNNs, or other algorithms?


texts:
I am new to Data Science and I want to make Customer Product Analytics for my company(bank). I can have a data of customers, their income, daily transactions, average balance etc and what product(saving certificates etc) they have taken according to their account balance. Can i have a prediction for new customers or existing customers that what product will be suitable for them according to their average balance , income etc? Can the machine learning algorithm predict each product to a particular customer?
I got to know cluster analysis and predictive analysis can be useful for such task. But i want to recommend a particular product to a particular customer. Which algorithm can be useful? And from where I have to begin?


texts: What are the different ways to feature engineer webpage data for input into a webpage classification model?

texts:
Looking for resources on the different ways that one can manipulate webpage data to input as features into a neural net.
I'm aware of a service called diffbot that claims to use a CV based method to "look" at the website while still being contextually aware of the data content embedded in the page.
What I'm struggling with is figuring out how to embed the webpage content so that a model can consume it.
Is there some standard way of identifying text/image elements on a page and their relative locations to one another?
If there's any particular modeling tricks that have been proven to be successful in this field that would also be appreciated.
Context:
I have a set of about 12 million sites that I plan on scraping and it would be economically ideal to know what data I need for training the model so I can optimize what resources need to be downloaded, stored, and processed.
The goal of the model is to classify websites by what kinds of services the site offers.


texts: Grouping numeric data into efficient group/pool

texts:
I have devices assigned with different data plan. But based on device behaviour amount of data used by device changes during the month. I need to put the device into appropriate data plan based on predicted data usages before end of the month to avoid data overage.

There are 5MB,25MB,50MB,150MB,250MB data pool plans per device. If I put 2 devices into 25MB plan then both device can collectively use 50MB of total data among each, if there are 5 device in 25MB plan then all 5 devices collectively can use 125MB data.

My first simple approach was to based on total data usages of the device, put it into higher data plan. It was not effective because it's shared pool, and pool size varies based on how many devices are in the pool, I need to optimised the grouping of the devices into any particular data plan to optimised pool data usages. I can have any number of group/pool (Max 5 because there are 5 different data plans).

another way I tried was to cluster the device based on data usages and add them into particular group/pool, but with this cluster gets created based on how the usages data is distributed, which is not optimally close to data plan.

Anyone know how to approach this problem to put the devices in a pool to optimally use the pools data limit and not go beyond it.


texts: Sentiment analysis for multiple entry in one text

texts:
I would like to do sentiment analysis on a set of financial news from the S&P 500 for given entities (organization names). However, each news (rows in my dataset) may have more than one entity and I have to do analysis for each entity separately the same news. First, my algorithm should find which entities are present in a specific news and then do sentiment analysis for them.

I would appreciate if someone could help me.

Example of output:


texts:
I want to try K-fold cross validation in R for C5.0 algorithm,

The following is the code i use. Can someone suggest me how can i include k-fold as well?

Classifi_C5.0 <- C5.0(TARGET ~., , data = training_data_SMOTED, trails = 500,
                      control = C5.0Control(minCases = mincases_count,
                                            noGlobalPruning = FALSE))

Is it required to do k-fold cross validation for Random forest?


texts: Should I run a learning curve before or after grid search?

texts:
I would like to find out how much data I should use for training my model. I also want to test what parameters I need using grid search.
My question is should I first run a learning curve on a baseline i.e. no parameters / default ones in the model first (xgboost) or should I run learning curve straight after best model is output from grid search?
But I need to find out first the training size but am thinking maybe the learning curve will change with different parameters.


texts: What is the purpose of standardization in machine learning?

texts:
I'm just getting started with learning about K-nearest neighbor and am having a hard time understanding why standardization is required. Reading through, I came across a section saying

  When independent variables in training data are measured in different
  units, it is important to standardize variables before calculating
  distance. For example, if one variable is based on height in cms, and
  the other is based on weight in kgs then height will influence more on
  the distance calculation.

Since K nearest neighbor is just a comparison of distances apart, why does it matter if one of the variables has values of a larger range since it is what it is.

Considering 3 points A,B & C with x,y co-ordinates (x in cm, y in grams) A(2,2000), B(8,9000) and C(10,20000), the ranking of the points as distance from origin for example (or any other point), will be the same whether the y values are in grams,pounds, tonnes or any other combinations of units for both x and y so where's the need to standardise. Every example or QA i see brushes through with the same statement of 'one variable influencing the other' without a real example of how this might occur. Again, how does one know when this influence is too much as to call for standardization.

Also,what exactly does standardization do to the values? One of the formulas does it by Xs = (X-mean)/(max-min) Where does such a formula come from and what is it really doing? Hopefully someone can offer me a simplified explanation or give me a link to a site or book that explains this in simple terms for beginners.


texts: Which algorithm is perfect to determine best fit analyst based on multiple factors?

texts:
Assume I have a team of 200 analyst who work on different IT tickets. I want a better ticket assignment system which considers multiple factors before ticket assignment.

Outstanding Ticket with each analyst - availability
A skill score on the type of ticket
Customer satisfaction Score
Communication score, etc

Now, as soon as a new ticket lands in the queue I'll determine how critical the ticket is, what kind of skill set is required and based on the customer/business area what kind of satisfaction and communication score the ticket is expecting.

Can someone suggest any algorithm which fits for this kind of problem - i.e. mapping the right analyst with right ticket based on multiple variables?


texts: How to pre-process frequency of a series of signals?

texts:
I use a neural net to generate predictions based on a time series of signals. I use a sliding window to feed the data to an LSTM model. The input signals have a random frequency that - I believe - is a valuable input. Time measure is restricted to seconds. By random frequency I mean the time between the signals varies between 0 (delta not measurable in seconds but order is known) and an extreme of 1-2 hours of rough maximum. I struggle with the following questions:

How to convert the frequency to an input feature for a neural net? My best idea is to measure the time elapsed since the last signal. This gives me a feature for each new signal in seconds. Approximate range is 0 through 7200 seconds. Is there any better way?
Depending on the answer on point 1 how should I normalize the data? I can have anything between 0 seconds and 1-2 hours. Providing my assumption on point 1 is right then I can add: the median of the entire sliding window is sometimes well below 30 seconds, sometimes well over 300 seconds. Problems I have:

I think I should normalize by each of the sliding window separately.

If I apply Z-score exclusively on each sliding window scope then the overall magnitude is eliminated. It would be interesting to keep an indicator of the overall input signal frequency measured on the entire dataset not only on the given window. Shall I create a second feature for compensating this?
If I apply a min-max approach on the given window then the small values will be too close to 0 due to having only a few very high values.

If I apply a min-max or the Z-score on the entire dataset then the small values will be too close to 0 due to having only a few very high values.
Any ideas about how to overcome these problems?


texts: Interpreting IEPY documentation commands

texts:
I have some troubles in interpreting the documentation (http://iepy.readthedocs.io/en/latest/tutorial.html) commands of IEPY package for Natural Language Processing. The first step to work with IEPY is to create an "instance" (as a non-programmer I have little idea what is it all about). They also provide the command to do it:

iepy --create <project_name>

My problem is that when I type the exact sentence into my command line, I get a syntax error. I suppose that I do not follow some coding conventions and misinterpret what I should really type. Would be glad to hear what I do wrong.


texts: Should I concat multiple stock timeseries datasets into one?

texts:
I have several timeseries datasets of stock data, with fundamental indicators. I would like to build a model that selects stocks for buy and hold.
I understand that to perform this task I have two options:

Train a model for each stock: This way, I understand that it is the most practical, however, the amount of data for each model will be very reduced (Each dataset has less than 1000 lines).

Putting all the data together in a single dataset: I didn't find anything on the internet to support this idea, however, I understand that the model would be more robust and would have a much larger amount of data to be trained.

So, what would be the correct way to perform this type of analysis? Any of you would suggest another way?
Thannk you in advance!


texts:
I have huge car photos.

I want to predict car's "brand-model-body type and production year"

First, I splitted data into train and validation, and I categorized them like this.

Every category has about 1000 train and 900 validation images.

My plan was: I train my keras model with these categories after training, model can predict labels like below:

audi a3 sedan 2008  => %25

audi a3 sedan 2009  => %25

audi a3 sedan 2010  => %25

audi a3 sedan 2011  => %25

And I can tell user that: "This car is Audi A3 Sedan 2008-2011"

My problem is, some of these categories have very similar photos. For example: audi a3 2009 and audi a3 2010 have same body type and there is not much difference between photos (No difference in reality).
Because of that, train accuracy has improved to about 0.9 but validation accuracy hasn't improved above 0.55

When I try some predictions, it usually gives same label, "Ford Focus sedan 2009" :)

Here is my output:

epoch,          acc,              loss,              val_acc,       val_loss
27,    0.7965514530544776, 0.56618134500483, 0.5192149643316993, 1.729015349846447

28,    0.8058803490480816, 0.5408204138258657, 0.5176764522193236, 1.778763979018732

29,    0.8167710489770164, 0.5116128672937693, 0.523258489762041, 1.7806432932022545

30,    0.8256544639818643, 0.4872381848016096, 0.5207534764479939, 1.8059904007678271

31,    0.8355546238309248, 0.4629556378035959, 0.5237253032663666, 1.8191414148756815

32,    0.8424464767701014, 0.4444190686917562, 0.5242512903147193, 1.8496954914466912

33,    0.8508739288802705, 0.422022156655134, 0.5303593149032422, 1.8565427863780883

34,    0.8576819265745635, 0.40545297008116027, 0.5262894901236571, 1.909881308499735

My train code is here:

Image_width, Image_height = 224, 224
num_epoch = 5000
batch_size = 16
learning_rate = 0.0001
model = ResNet50(weights='imagenet', include_top=False, input_shape=(Image_width, Image_height, 3))
fc_neuron_count = 1024
output = model.output
output = GlobalAveragePooling2D()(output)
output = Dense(fc_neuron_count, activation='relu')(output)
predictions = Dense(num_classes, activation='softmax')(output)
model = Model(inputs=model.input, outputs=predictions)

model.compile(optimizer=opt.Adam(lr=learning_rate), loss=losses.categorical_crossentropy,
              metrics=['accuracy'])

history_transfer_learning = model.fit_generator(
    train_generator,
    epochs=num_epoch,
    steps_per_epoch=num_train_samples // batch_size,
    validation_data=validation_generator,
    validation_steps=num_validate_samples // batch_size,
    class_weight='auto',
    callbacks=callbacks_list)

Am I doing something wrong? How can I achieve this result?
Should I change validation accuracy calculation, or should I give more photos per category?


texts: Automatically generating images for an art show with deep learning software

texts: My pursuit is to generate something like a grottesque(a kind of painting producing human-animals and plants hybrids).  I need to do something like this painting in order to create an art exhibition. I don't need that produced images are ordinate or beautiful, just monstrous images produced automatically by a computer. Can someone help me? And is it possible to do it in a month?


texts: LSTM Model for predicting the minutely seasonal data of the day

texts:
I am making a model for predicting the network traffic volume for our data center.
Let me describe my dataset first. At this time, we have the model of 90 days, on each day, we record the network traffic volume every minute. And this data also has the seasonality pattern: the network traffic volume fluctuates and remains at night, decreases in the morning and increases in the afternoon (as I could observe, this pattern happens on all the data in my dataset)

What I want at this time is a model for predicting the network traffic volume in some next minutes(25 minutes, for example), given that the data is given for all the previous minutes. The new predicted value will be contributed for the next predicting. For example, the value at minute i will be added to a window of data with a specific length to predict the value at minute i+1.

At this time, I have tried LSTM-RNN and the feature I use is the minute of the day and the network traffic volume in that minute (with normalization before inputting to the LSTM network).
However, my problem is: my model could catch the wrong pattern of the data: when the network traffic volume should increase in the afternoon, my model predicts it decreases. I have tried with different structures of LSTM network (increases the LSTMs layers, change the number of nodes in the fully-connected layers after the LSTM layers...); and also the length of the window, but that problem still remains.

So I want to ask if there is any problem with LSTM model for predicting the data with seasonal trend like my data? If no, could anybody suggest me an LSTM model or any other suitable model for my data?

Thank you in advance :-)


texts: Disparity between training and testing errors with deep learning: the bias-variance tradeoff and model selection

texts:
I am developing a convolutional neural network and have a dataset with 13,000 datapoints that is split 80%/10%/10% train/validation/test. In tuning the model architecture, I found the following, after averaging results over several runs with different random seeds:

3 conv layers: training MAE = 0.255, val MAE = 0.340
4 conv layers: training MAE = 0.232, val MAE = 0.337
5 conv layers: training MAE = 0.172, val MAE = 0.328.

Normally, I'd pick the model with the best validation MAE (the trends are the same for the testing MAE, for what it's worth). However, the architecture with the best validation MAE also has the largest difference between training and validation MAE. Why is what I'd normally think of as overfitting giving better results? Would you also go with 5 convolutional layers here, or are there concerns with a large difference in training and validation/testing performance?

On what I imagine is a related note, I am familiar with the article "Reconciling modern machine-learning practice and the classical bias–variance trade-off" in PNAS, which has the thought-provoking image below. Is this something that's actually observed in practice -- that you can have minimal training error but good out-of-sample, generalizable performance, as shown in subpanel B?


texts: Need help with how to diagnose training and validation metrics

texts:
I'm currently learning to build a DNN binary classification model on some dataset. But when I analysed the results, I found it counter-intuitive that the training accuracy changed almost at the same rate as the validation accuracy. So did the losses. And they all reached a plateau at some point. How could I possibly improve the model performance at this stage? Here's a snippet of the results:

fold: 0 epoch: 0 batch: 0
training loss: 0.674389 validation loss: 0.67371
training accuracy: 0.656331 validation accuracy: 0.656968
Fold: 0 epoch: 0 batch: 500
training loss: 0.527997 validation loss: 0.527813
training accuracy: 0.734021 validation accuracy: 0.733639
Fold: 0 epoch: 0 batch: 1000
training loss: 0.526019 validation loss: 0.526277
training accuracy: 0.736067 validation accuracy: 0.735111
Fold: 0 epoch: 0 batch: 1500
training loss: 0.525466 validation loss: 0.525721
training accuracy: 0.736672 validation accuracy: 0.73532
Fold: 1 epoch: 0 batch: 0
training loss: 0.525416 validation loss: 0.525261
training accuracy: 0.735794 validation accuracy: 0.736488
Fold: 1 epoch: 0 batch: 500
training loss: 0.525751 validation loss: 0.525623
training accuracy: 0.735472 validation accuracy: 0.735887
Fold: 1 epoch: 0 batch: 1000
training loss: 0.525579 validation loss: 0.525459
training accuracy: 0.735538 validation accuracy: 0.736077


texts: Does central limit theorem work well for Pareto distribution?

texts:
I am new to Data Science. Recently I was studying a course about statistics. One of the tasks there was to check the central limit theorem in practice.

The idea was quite simple: take a continuous random variable; generate, say, 1000 samples from it, each of size n; draw a histogram of the samples. Then find the parameters of the normal distribution using the central limit theorem and draw the PDF of the distribution. As a result, the histogram and the PDF should be, roughly speaking, "similar" (and become more "similar" as n grows).

I chose Pareto distribution and, with this Python code,

import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as sts
from math import sqrt

n = 10

b = 3.0
random_value = sts.pareto(b)
mean = random_value.mean()
variance = random_value.var()

plt.hist(samples(random_value, n, 1000), bins=20, normed=True)

x = np.linspace(0, 6, 100)
pdf = sts.norm(mean, sqrt(variance / n)).pdf(x)
plt.plot(x, pdf, color='r', label='theoretical PDF')

where the samples function was like this

def samples(random_value, sample_size, number_of_samples):
    result = np.asarray([])
    for i in range(number_of_samples):
        result = np.append(result, np.mean(random_value.rvs(sample_size)))
    return result

I got the next histogram and PDF as a result

The result looked suspicious to me: there was no expected similarity. I checked the same code with another continuous distribution (with a uniform distribution), and the graphs looked much more similar.

Is it a feature of the Pareto distribution which makes it a bad choice to demonstrate how the central limit theorem works? Or something is wrong with the code itself (e.g., the parameters of the normal distribution are calculated in a wrong way)? Thank you in advance.


texts: Tensorflow regression model giving same prediction every time

texts:
import tensorflow as tf

x = tf.placeholder(tf.float32, [None,4])    # input vector
w1 = tf.Variable(tf.random_normal([4,2]))   # weights between first and second layers
b1 = tf.Variable(tf.zeros([2]))             # biases added to hidden layer
w2 = tf.Variable(tf.random_normal([2,1]))   # weights between second and third layer
b2 = tf.Variable(tf.zeros([1]))             # biases added to third (output) layer

def feedForward(x,w,b):                     # function for forward propagation
    Input = tf.add(tf.matmul(x,w), b)
    Output = tf.sigmoid(Input)
    return Output

Out1 = feedForward(x,w1,b1)                # output of first layer
Out2 = feedForward(Out1,w2,b2)             # output of second layer
MHat = 50*Out2                             # final prediction is in the range (0,50)

M = tf.placeholder(tf.float32, [None,1])   # placeholder for actual (target value of marks)
J = tf.reduce_mean(tf.square(MHat - M))    # cost function -- mean square errors
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(J)     # minimize J using Gradient Descent
sess = tf.InteractiveSession()             # create interactive session

tf.global_variables_initializer().run()    # initialize all weight and bias variables with specified values
xs = [[1,3,9,7],
      [7,9,8,2],                           # x training data
      [2,4,6,5]]

Ms = [[47],
      [43],                                # M training data
      [39]]

for _ in range(1000):                      # performing learning process on training data 1000 times
    sess.run(train_step, feed_dict = {x:xs, M:Ms})

>>> print(sess.run(MHat, feed_dict = {x:[[1,15,9,7]]}))
[[50.]]

>>> print(sess.run(MHat, feed_dict = {x:[[3,8,1,2]]}))
[[50.]]

>>> print(sess.run(MHat, feed_dict = {x:[[6,7,10,9]]}))
[[50.]]

In this code, I am trying to predict the marks M obtained by a student in a test out of 50 given how many hours he/she slept, studied, used electronics and played the day before the test. These 4 features come under the input feature vector x.

To solve this regression problem, I am using a deep neural network with an input layer with 4 perceptrons (the input features), a hidden layer with two perceptrons and an output layer with one perceptron.

I have used sigmoid as the activation function. But, I am getting the exact same prediction([[50.0]]) for M for all possible input vectors I feed in.

Can someone please tell me what is wrong with the code above, and why I get the same result each time?


texts: Convolutional Neural Networks for Sensor Data

texts:
I am reading this paper regarding action recognition based on sensor data. And this paper first transforms the input to spectrogram (time vs frequency, it's like the Fourier Transform of each segment of the input time series but square it). So instead of the time series you are given a matrix of st x sf (st is number of time windows, sf is number of frequency components considered) as input to you CNN.

There is this algorithm:

Can someone explain this? Especially the formula for CNN, I don't get why you have to convolve the (t-1) and dw.


texts: What are some of the things that Policy Gradients Method can solve?

texts: What are some of the things that Policy Gradients Method can solve that other Methods (like Q Learning and Genetic Algorithms) can't solve?


texts: Libraries for (label propagation algorithms/frequent subgraph mining) for graphs in R

texts:
General description of the problem

I have a graph where some vertices are labeled with a type with 3 or 4 possible values. For the other vertices, the type is unknown.
My goal is to use the graph to predict the type for vertices that are unlabeled.

Possible framework

I suspect this fits into the general framework of label propagation problems, based on my reading of the literature (e.g., see this paper and this paper)

Another method that is mentioned often is Frequent Subgraph Mining, which includes algorithms like SUBDUE,SLEUTH, and gSpan.

Found in R

The only label propagation implementation I managed to find in R is label.propagation.community() from the igraph library.
However, as the name suggests, it is mostly used to find communities, not for classifying unlabeled vertices.

There also seems to be several references to a subgraphMining library (here for example), but it looks like it is missing from CRAN.

Question

Do you know of a library or framework for the task described?


texts: How to get rid of the expectation in Monte Carlo Policy Gradient method?

texts:
Taken from Policy Gradient lecture notes, slide 16 onward:

Here in the David Silver Lecture series I perfectly understand how he took the expectation value in the theorem part by combining the following equation .

But how he removed the expectation part when talking stochastic gradient decent ?


texts: What is the best way to replace NaN values for ranked columns

texts:
I have a column named BsmntQual that gives a ranking on the height of the basement per each house.  These are all of the unique values in this column:

print(train['BsmtQual'].unique().tolist())

>>> ['Gd', 'TA', 'Ex', nan, 'Fa']

This is the legend of this particular column:

BsmtQual: Evaluates the height of the basement

       Ex   Excellent (100+ inches)
       Gd   Good (90-99 inches)
       TA   Typical (80-89 inches)
       Fa   Fair (70-79 inches)
       Po   Poor (<70 inches
       NA   No Basement < Not to be confused with the nan value above

This is what I did for my other ranked columns but this one did not have NaN values:

train['ExterQual'] = train['ExterQual'].replace(['Ex', 'Gd', 'TA', 'Fa'], [4, 3, 2, 1])  # Exterior Quality

For numerical values, a common way is to fill all NaN values with the mean of the column.  But what is a good way of replacing the NaN values for columns such as these?

Here is the full dataset


texts: Do smaller neural nets always converge faster than larger ones?

texts:
In your experience, do smaller CNN models (fewer params) converge faster than larger models?
I would think yes, naturally, because there are fewer parameters to optimize. However, I am training a a custom MobileNetV2-based Unet (with 2.9k parameters) for image segmentation, which is taking longer to converge than a model with greater number of parameters (5k params). If this convergence behavior is unexpected, it probably indicates a bug in the architecture


texts:
I'm trying to build a cosine locality sensitive hash so I can find candidate similar pairs of items without having to compare every possible pair. I have it basically working, but most of the pairs in my data seem to have cosine similarity in the -0.2 to +0.2 range so I'm trying to dice it quite finely and pick things with cosine similarity 0.1 and above.

I've been reading Mining Massive Datasets chapter 3. This talks about increasing the accuracy of candidate pair selection by Amplifying a Locality-Sensitive Family. I think I just about understand the mathematical explanation, but I'm struggling to see how I implement this practically.

What I have so far is as follows

I have say 1000 movies each with ratings from some selection of 1M users. Each movie is represented by a sparse vector of user scores (row number = user ID, value = user's score)
I build N random vectors. The vector length matches the length of the movie vectors (i.e. the number of users). The vector values are +1 or -1. I actually encode these vectors as binary to save space, with +1 mapped to 1 and -1 mapped to 0
I build sketch vectors for each movie by taking the dot product of the movie and each of the N random vectors (or rather, if I create a matrix R by laying the N random vectors horizontally and layering them on top of each other then the sketch for movie m is R*m), then taking the sign of each element in the resulting vector, so I end with a sketch vector for each movie of +1s and -1s, which again I encode as binary. Each vector is length N bits.
Next I look for similar sketches by doing the following

I split the sketch vector into b bands of r bits
Each band of r bits is a number. I combine that number with the band number and add the movie to a hash bucket under that number. Each movie can be added to more than one bucket.
I then look in each bucket. Any movies that are in the same bucket are candidate pairs.

Comparing this to 3.6.3 of mmds, my AND step is when I look at bands of r bits - a pair of movies pass the AND step if the r bits have the same value. My OR step happens in the buckets: movies are candidate pairs if they are both in any of the buckets.

The book suggests I can "amplify" my results by adding more AND and OR steps, but I'm at a loss for how to do this practically as the explanation of the construction process for further layers is in terms of checking pairwise equality rather than coming up with bucket numbers.

Can anyone help me understand how to do this?


texts: "DeepTree" network with binary classifiers at each node

texts:
I need to do a project where I build a "DeepTree Network". Let's say I  have a 3-classification problem with classes A, B, and C.

My tree should then look like this:

first node: CNN binary classifier classifying between class C, and superclass A+B
second node: CNN binary classifier classifying between class A and B

Does anyone know if there are any existing papers on this subject? Or any implementation? The leaves don't need to be CNNs necessarily, any sort of tree implementation like this in TensorFlow would help me a lot.


texts: Clustering with restrictions - Silhouette and C index metrics

texts:
I am working on clustering with DBSCAN but with a certain constraint: the points inside a cluster have to be not only near in a Euclidean distance way but also near in a geographic distance way. It is feasible if you use the pseudocode and work on it.

My issue however is about the proper metric on evaluating the clustering results. Applying the silhouette and c index metrics I get (as expected) results lower than the ones of clustering without constraints. A visual check shows that my clustering is well implemented but it seems as though that the interval of values of the metrics (for example [-1,1] for silhouette and [0,1] for C index) has changed. How can I verify that? I don't want a biased metric.


texts: Multi-label classification of text with variable tag distribution in Keras

texts:
I'm currently working with a dataset where each row contains 2 things:

Text
Related tags

Both are dutch.

I'm trying to build a model that allows me to assign tags to new text based on what it learned from the dataset.

The problem I think I'm having is that the dataset itself is very small (around 2000 messages) but the tags are also very unevenly distributed.

For instance, the most used tag occurs 121 times, but there are around 40 tags that only occur 5 times in a different article.

I tried to solve the data quantity problem by using dutch word-vectors and use them in an Embedding layer. This allows to give more context to the final model.

The tags problem I haven't got a solution for yet. I think I should make a custom loss function that lets text with tags occurring more frequently weigh less when they are valid, and tags occurring less will score higher. But I have no idea how to get started on that. If you can help me, thanks!

My model looks like this:

MAX_SEQUENCE_LENGTH = 100
EMBEDDING_DIM = 300

embedding_layer = Embedding(num_words,
                            EMBEDDING_DIM,
                            weights=[embedding_matrix],
                            input_length=MAX_SEQUENCE_LENGTH,
                            trainable=False)

sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
embedded_sequences = embedding_layer(sequence_input)
x = Dropout(0.1)(embedded_sequences)
x = Conv1D(128, 5, activation='relu')(x)
x = MaxPooling1D(5)(x)
x = Conv1D(128, 5, activation='relu')(x)
x = MaxPooling1D(5)(x)
x = Flatten()(x)
x = Dense(128, activation='relu')(x)
preds = Dense(len(labels_index), activation='sigmoid')(x)

model = Model(sequence_input, preds)
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

Data input is a padded sequence as input and output is a binary list of tags that are related to the input (made using MultiLabelBinarizer).


texts: Encode the days of week as numeric variable

texts:
I would like to understand if there is the possibility to encode the days of the week as a single numerical column to preserve the ordinal relationship between the days.
My task is a classification task.
So, something like this:

Monday: 0
Thrusday: 1
Wednesday: 2
...
Sunday: 6

I wouldn't use a one-hot encoding for this scenario because I would like to preserve the ordinal relationship between the days.
However, I don't know if this is an allowed approach or not.


texts:
I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far:

import numpy as np
from sklearn.datasets import load_boston
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

dataset = load_boston()
X = boston['LSTAT']
y = boston['MEDV']

def linearRegression2(b0 = 0, b1 = 1, lr = 0.00001, n_times = 1000):
partial_B0 = partial_B1 = 0
errors = []
m = len(y)
i = 0
for i in range(n_times):
    error = y - (b0 + X * b1)
    #Updating b0 and b1
    partial_B0 += -lr * (error.sum() / m)
    partial_B1 += -lr * (X.T.dot(error) / m)
    b0 = b0 - (lr * partial_B0)
    b1 = b1 - (lr * partial_B1)
    errors.append(mean_squared_error(y, (b0 + X * b1)))
    i += 1

print(b0, b1)
print('Iteration i', i)
return errors

error = linearRegression2()
plt.plot(np.arange(1, 1000+1), error)
plt.ylabel('Error')

However, when I plot my cost function, it is decreasing but still too high. I tried to decrease the learning rate, increase the iteration but not changing much.


texts: any efficient way to find surrounding adjective/verbs with respect to the target phrase in python [updated]?

texts:
I am doing sentiment analysis on given documents. My goal is to find out the closest or surrounding adjective words with respect to the target phrase in my sentences. I do have an idea how to extract surrounding words with respect to target phrases. But how do I find out relatively close or closest adjective or NNP or VBN or other POS tag with respect to target phrase ?

Here is the sketch idea of how I may get surrounding words with respect to my target phrase.

sentence_List = {
    "Obviously one of the most important features of any computer is the human interface.",
    "Good for everyday computing and web browsing.",
    "My problem was with DELL Customer Service",
    "I play a lot of casual games online[comma] and the touchpad is very responsive"
}

target_phraseList = {
    "human interface",
    "everyday computing",
    "DELL Customer Service",
    "touchpad"
}

Note that my original dataset was given as DataFrame where the list of the sentences and respective target phrases were given. Here I just simulated data as follows:

import pandas as pd
df=pd.Series(sentence_List, target_phraseList)
df=pd.DataFrame(df)

Here, I tokenize the sentence as follow:

from nltk.tokenize import word_tokenize
tokenized_sents = [word_tokenize(i) for i in sentence_List]
tokenized=[i for i in tokenized_sents]

Then I try to find out surrounding words with respect to my target phrases by using this loot at here. However, I want to find out relatively closer or closest adjective, or verbs or VBN with respect to my target phrase.

How can I make this happen? Any idea to get this done? Thanks


texts:
I'm confused about the notation keras uses to describe tensor shapes in its API documentation. Specifically, I don't understand the use of the '+' sign in the notation. For instance, what kind of shape is a tensor whose shape is defined as:
batch_shape + (new_rows, new_cols, filters)


texts: In sklearn Pipeline, why are all parameters fit_transform(), but the last one can be just fit()?

texts:
I'm reading through Hands-On Machine Learning with Scikit-Learn & TensorFlow. We're going over sklearn Transformation Pipelines for preparing data for the ML algorithms.

Here's the code (housing_num is a dataframe of the numerical attributes of the main dataframe housing):

num_pipeline = Pipeline([
    ('imputer', Imputer(strategy='median')),
    ('attribs_adder', CombinedAttributesAdder()),
    ('std_scaler', StandardScaler())
])

housing_num_tr = num_pipeline.fit_transform(housing_num)

The text explains, "The Pipeline constructor takes a list of name/estimator pairs defining a sequence of steps. All but the last estimator must be transformers (i.e., they must have a fit_transform() method)."

1) Why do we fit_transform() some, but only fit() without transforming others? I'm guessing this has to do with the difference between estimators and transformers, which I'm confused on in this context.

2) So when using Pipeline, you can only include one estimator that won't be transformed, and it has to be the last input? ("all but the last estimator must be transformers")

Thanks,

Greg


texts: Is LASSO regression implemented in Statsmodels?

texts: I would love to use a linear LASSO regression within statsmodels, so to be able to use the 'formula' notation for writing the model, that would save me quite some coding time when working with many categorical variables, and their interactions. However, it seems like it is not implemented yet in stats models?


texts:
having two dataframes each having common column which is country_code. while merging it by 'inner' join, it's returns empty dataframe. but i can see there is many values are common like IND, USA....
pd.merge(df1, df2, on='country_code', how='inner')

checked the column name and the string not having any spaces. please help me on it.


texts: torchvision Dataloader using "transforms"

texts:
I made a torchvision.datasets dataloader for torchvision.datasets.SBDataset :

sbd_data = torchvision.datasets.SBDataset('./',transforms = data_transforms['train'],download=False)
data_loader = torch.utils.data.DataLoader(sbd_data,
                                          batch_size=4,
                                          shuffle=True,
                                          num_workers=4)

Here the data_transforms['train'] is a transforms.Compose instance.
And to show a batch i used this function :

def imshow(inp, title=None):
    """Imshow for Tensor."""
    inp = inp.numpy().transpose((1, 2, 0))
    inp = np.clip(inp, 0, 1)
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  # pause a bit so that plots are updated

Now when i try to call this function as

inputs,_ = next(iter(data_loader))
out = torchvision.utils.make_grid(inputs)
imshow(out)

I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-21-6753e8965891> in <module>
     13
     14 # Get a batch of training data
---> 15 inputs,_ = next(iter(data_loader))
     16 # Make a grid from batch

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    817             else:
    818                 del self.task_info[idx]
--> 819                 return self._process_data(data)
    820
    821     next = __next__  # Python 2 compatibility

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _process_data(self, data)
    844         self._try_put_index()
    845         if isinstance(data, ExceptionWrapper):
--> 846             data.reraise()
    847         return data
    848

/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)
    367             # (https://bugs.python.org/issue2651), so we work around it.
    368             msg = KeyErrorMessage(msg)
--> 369         raise self.exc_type(msg)

TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torchvision/datasets/sbd.py", line 115, in __getitem__
    img, target = self.transforms(img, target)
TypeError: __call__() takes 2 positional arguments but 3 were given

I read the sbd.py for SBDataset class which inherits from VisionDataset which clearly implements a StandardTransform class, which is only used when inputs transform and target_transform are both provided, does it assume the transforms to be an instance of StandardTransform? Is this intentional? Is there a neat way to solve this?
because the StandardTransform is not present in the docs.
Note the transform and transforms difference.
The transform.Compose class clearly doesn't support two inputs ie inputs , targets.

I could have just worked with transform but this class(SBDataset) only has transforms as an argument.

pytorch version : 1.2.0.dev20190805
Torchvision Version: 0.4.0a0+d31eafa
Python version : 3.6


texts: In Keras library, what is the meaning of "nb_steps_warmup" in the DQNAgent Object initialization?

texts:
I can't understand the meaning of "nb_steps_warmup", a parameter of the __init__ function of DQNAgent class of the Keras_RL module.

I just know that when I set small value for "nb_steps_warmup" the command line prints: UserWarning: Not enough entries to sample without replacement. Consider increasing your warm-up phase to avoid oversampling!

This is my code:

import numpy as np
import gym
import gym_briscola
import argparse
import os

from keras.models import Sequential
from keras.layers import Dense, Activation, Flatten
from keras.optimizers import Adam

from rl.agents.dqn import DQNAgent
from rl.policy import BoltzmannQPolicy
from rl.memory import SequentialMemory
import logging

def getModel(input_shape, nb_actions):

    # Next, we build a very simple model.
    model = Sequential()
    model.add(Flatten(input_shape = input_shape))
    model.add(Dense(nb_actions, activation = 'relu'))
    for i in range(2):
        model.add(Dense(2, activation = 'relu'))

    model.add(Dense(nb_actions, activation = 'relu'))
    # print(model.summary())
    return model

def init():

    ENV_NAME = 'Briscola-v0'

    # Get the environment and extract the number of actions.
    env = gym.make(ENV_NAME)
    env.setName("Inteligence")
    env.cicle = True
    nb_actions = env.action_space.n
    window_length = 10

    input_shape = (window_length, 5)

    # Finally, we configure and compile our agent. You can use every built-in Keras optimizer and
    # even the metrics!
    memory = SequentialMemory(limit=50000, window_length=window_length)

    #Maxwell-boltzmann distribution
    policy = BoltzmannQPolicy()
    dqn = DQNAgent(model=getModel(input_shape, nb_actions), nb_actions=nb_actions, memory=memory, nb_steps_warmup=11,
                   target_model_update=1e-5, policy=policy, dueling_type='avg', enable_dueling_network=True)

    print("Compila")
    dqn.compile(Adam(lr=1e-5), metrics=['mae'])

    try:
        dqn.load_weights('dqn_{}_weights.h5f'.format(ENV_NAME))
    except OSError:
        print("File non trovato")

    return dqn, env, ENV_NAME

def startTraining():

    dqn, env, ENV_NAME = init()

    print("Fit")
    dqn.fit(env, nb_steps=5E6, visualize=False, verbose=1, log_interval=1000)

    # After training is done, we save the final weights.
    dqn.save_weights('dqn_{}_weights.h5f'.format(ENV_NAME), overwrite=True)

def startTest():

    dqn, env, _ = init()

    print("Test")
    # Finally, evaluate our algorithm for 5 episodes.
    dqn.test(env, nb_episodes=10, visualize=False)

#Log config
def setLogging(show = True):
    logging.getLogger("Briscola").propagate = show
    logging.getLogger("IA").propagate = True
    logging.getLogger("Client").propagate = show
    logging.getLogger("Vincitore").propagate = show

    logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":

    #Parameter settings
    parser = argparse.ArgumentParser()
    parser.add_argument("-m", "--modality", help="The modality of the program", const="train", nargs='?')
    parser.add_argument("-l", "--logging", help="Enable logging", type=bool, const=False, nargs='?')
    args = parser.parse_args()

    setLogging(True)
    print("Avvio modalita' ", args.modality)

    if args.modality == "test":
        startTest()
    else:
        startTraining()

    print("Fine")

This is the doc of the module: https://keras-rl.readthedocs.io/en/latest/agents/dqn/

I hope I was clear with my English.


texts: I have a bunch of  articles about science from a certain website. When a new article is published, I want to determine if that article is really talking about science (and not politics for example). How can I do that? What machine learning technique shroud I use? I'm thinking at using something similar to spam detection. Is that ok? Thank you!


texts: Looking for an algorithm to deal with purely categorical data.  It was suggested to me to look into the K-medoids algorithm.  Anyone know if there is a K-medoids algorithm R library(package)?


texts: I'm going through this chatbot example, which uses the Cornell movie dialog corpus. Expanding this example, is it possible to build a "knowledge bot" (ie) a bot that can chat and be knowledgeable in a specific domain? So for example, I use the Bible to train the system. Then it should be able to answer questions on Bible. Or I use the wikia content of Star Wars and be able to answer all the questions related to Star Wars. Is such a system even possible, or is it still in the realm of scifi?


texts: Machine learning algorithm to classify blog posts

texts:
So I have a large collection of blog posts containing title, content, category, tags and geo-location fields and I'm looking to achieve three things:

Assign a category (or multiple categories) to all the posts and any new ones. I have a strict vocabulary of categories.
Add new tags to the posts that might be relevant to the post.
Mark the post if it contains information about a place. For example: Lorem ipsum dolor sit amet San Francisco, consectetur adipiscing elit.

I've been looking into different machine learning algorithms, most recently decision trees, but I don't feel that is the best algorithm to work out the problems above (or that I haven't understood them enough).

Many of these posts already contain categories, tags and geo-location data. Some do not contain any information and some have only a few details.

What would be the best machine learning algorithm to look into to solve each of the three areas?


texts: Closed form solution of linear regression via least squares using matrix derivatives

texts:
How is the closed form solution to linear regression derived using matrix derivatives as opposed to using the trace method as Andrew Ng does in his Machine learning lectures.  Specifically, I am trying to understand how Nando de Frietas does it here.

We want to find the value of $ \theta $ that minimizes $ J(\theta)=(X\theta-Y)^{T}(X\theta-Y) $, where $\theta \in \mathbb{R}^{N \times 1},  X \in \mathbb{R}^{M \times N}$, and $Y \in \mathbb{R}^{M \times 1}$

$\nabla_{\theta}J(\theta) = \nabla_{\theta} (X\theta-Y)^{T}(X\theta-Y)$

$ = \nabla_{\theta} (\theta^{T} X^{T}-Y^{T})(X\theta-Y)$

$ = \nabla_{\theta} (\theta^{T} X^{T}X\theta-\theta^{T} X^{T}Y - Y^{T}X\theta + Y^{T}Y) $

Note that $\theta^{T} X^{T}Y$ is a scalar, so $\theta^{T} X^{T}Y = (\theta^{T} X^{T}Y)^{T} = Y^{T} X \theta$

$\nabla_{\theta}J(\theta) = \nabla_{\theta}(\theta^{T} X^{T}X\theta-Y^{T} X \theta - Y^{T}X\theta + Y^{T}Y)$

$ = \nabla_{\theta}(\theta^{T} X^{T}X\theta- 2 Y^{T} X \theta + Y^{T}Y)$

$ = \nabla_{\theta} \theta^{T} X^{T}X\theta - \nabla_{\theta} 2 Y^{T} X \theta + \nabla_{\theta} Y^{T}Y$

$ = \nabla_{\theta} \theta^{T} X^{T}X\theta - \nabla_{\theta} 2 Y^{T} X \theta $

How do I apply the matrix derivatives described in that video to solve this?  He skip steps.

Edit:
Below is the suggested strategy of removing theta by differentiating, then taking the inverse of both sides.  So looking at one term at a time, we have

$ \nabla_{\theta} \theta^{T} X^{T}X\theta = ? $
How do I differntiate this?  This is like differntiating $x\alpha_{1} \alpha_{2} x$ w.r.t. x in the scalar case. I need to combine those $\theta$ terms to hit them with the derivative. Transposing seems to result in the same expression:
$$ (\nabla_{\theta} \theta^{T} X^{T}X\theta)^{T} = \nabla_{\theta} \theta^{T} X^{T}X\theta$$

Looking at the second term, we have

$ \nabla_{\theta} 2 Y^{T} X \theta = 2 X^{T} Y$.

Putting together this, we have:
$$\nabla_{\theta} \theta^{T} X^{T}X\theta = 2 X^{T} Y$$

Knowing the solution is $\theta = (X^{T}X)^{-1}X^{T}Y$ we can reverse engineer the problem, but I am just not seeing it.  And how do we get rid of that 2 factor?


texts: Spark creating dat file - header/trailer issue

texts:
Although this is a DS community but I saw spark tag so asking query here.

0

I've a specific scenario of creating a file extract (dat/delimited) in scala/spark; just need some suggestions on an alternate approach.

Header & Trailer creation:

val header = Seq(filename,system_time)
Seq(header).toDS.write.text(s"/path/to/header/creation/dir")
val trailer = Seq(rowscount,filename)
Seq(header).toDS.write.text(s"/path/to/trailer/creation/dir")

I've extract from hive table in a dataframe:

val df = sql("select * from hive")

The dataframe has different schema than header/trailer. At present I just do a merge of header-df-trailer and create a final file.

My query is, don't we have any option of creating the final file altogether without having to store them separately & merging?


texts: Interpolation in nlp - definition of O term

texts:
Reading definition of interpolation below how are the O terms defined? Is this a value that is set manually?

Example

P( Sam | I am ) = count( Sam I am ) / count(I am) = 1 / 2

Interpolation using N-grams
We can combine knowledge from each of our n-grams by using interpolation.
E.g. assuming we have calculated unigram, bigram, and trigram probabilities, we can do:
P ( Sam | I am ) = Θ1 x P( Sam ) + Θ2 x P( Sam | am ) + Θ3 x P( Sam | I am )

(original problem statement image)


texts: Feature selection in R too large dataset

texts: I'm doing credit risk modelling and the data have large number of features.I am using boruta package for feature selection. The package is too computationally expensive, I cannot run it on the complete training dataset. What i'm trying to do is take a subset of the training data(let's say about 20-30%) and run the boruta package on that subsetted data and get the important features. But when i use random forest to train the data I have too use the full dataset. My question is, Is it right to select features only on a part of train data but then build the model on whole of training data?


texts: LSTM - Incorporate word embedding in layer with multiple records in same date

texts:
I have a time series data having more than one record in a single date. Number of records in a single date is not consistent.

I have 3 input features namely phrase, cost and weight. My goal is to predict 'Cost'.
I have used Keras texts_to_sequences to deal with the text input 'phrase' column.

I have following concerns

how to incorporate Phrase column in the LSTM model
how to deal with multiple records in a single date
what will be the shape of input data
How do I structure the model like 1st layer to be Embedding layer, 2nd to Dense and so on.

Here is the sample of the data:

Here is the processed data:

Kindly assist me.


texts: In layman terms, what is the meaning of fitting a model into a data set?

texts:
When we fit any model into a data set for prediction, what exactly happens behind the scenes?

I am learning Regression and I am a little confused about how exactly it fits the classifier or regression into a dataset.


texts: How to select 500 most pertinents tags among 10000?

texts:
Say we have 100,000 documents tagged with 10,000 different tags (Max 5 tag per document). We wish to limit allowed tags to a list of 500 tags.

How to select 500 tags in order to cover the largest set of documents ?

Firstly, I chose the 500 most frequent tags. If we keep only these 500 tags, 70% of documents keep at least 1 tag.

I'm looking for a better method to pick the 500 most pertinents tags.

example :

my best set of 5 tags to cover all documents is [italian, german, french, chineese, japan].

it is not [italian, spanish, indian, finnish, english] although these are the most frequent tags.

I tried a Single Value Decomposition on the matrix : [documentID, set of tags]. but after that ? is it a good idea ? how to get 500 tags from the SVD results ?


texts: what is the loss function in char recognition using Tensorflow?

texts:
I have code in Tensorflow using convolution neural network to recognize the characters in street view Text (SVT) data.

Since the label type is string, what should I use instead of tf.nn.sparse_softmax_cross_entropy_with_logits() in the loss function?

I cannot use tf.nn.sparse_softmax_cross_entropy_with_logits() because the labels her must be an int dtype??


texts:
I have 2 datasets from 2013-2017 for each day.
a) Revenue generated by Locations and date. b) Holiday name and date

I would like to know how each holiday is impacting the revenue by location. I am confused about what method to use! Should 3-Sigma rule or Bollinger plot be enough or should I try spline regression or something else? FYI I am using python for this analysis.


texts:
I came across to find the correlation between the ratings assigned by two coaches to a same group of 40 players.

I have tabulated the results as below:

The Somers' D is 50%.

However, for the case below,

The Somers' D is 94.7%.

My question is, why both scenarios are having 2 deviations but the first scenario has so much lower Somers' D compared to the second scenario?


texts: Formatting multi-level variable width data for model

texts:
I would like to calculate a value based on the number of lines on outage at a given time.

An outage consists of the two endpoints and the voltage of the line:

outage = ["node1", "node2", "120"]

The historical values I have are based on per hour time stamps, so
the format I believe I will need to feed the model is like so:

[timestamp, [outage1, outage2, etc...], value]

Do most machine learning libraries accept data in a variable width format like this? Alternatively, the only other thing I can think of is to have a super-wide data set and do a one hot encoding where each column is a node and its value is the voltage (scaled perhaps).

Example:

  Columns:
    [timestamp, node1, node2, node3, etc..., value]
  Values:
    ["1/1/2019 14:00", 120, 120, 0, etc..., 5]

There are thousands of nodes on this grid. What is the best approach for formatting this data in a way that most models will accept?


texts: Classification problem in pytorch with loss function CrossEntropyLoss returns negative output in prediction

texts: I am trying to train and predict SVHN dataset (VGG architecture). I get very high validate/test accuracy by just getting the largest output class. However, the output weights are of large positive and negative numbers. Are they supposed to parsed as exp(output)/sum(exp(output)) to be converted to probability? Thank you!


texts: How to measure word similarity using wordnet for the information theoretic definition as detailed in Resnik 1995?

texts:
Resnik 1995 equation 3 uses count(n) to define P(c). What is count(n)?

Any solved example would be appreciated.


texts:
I've no experience in data science so this will be one of those questions...

I have data from >100k purchases made via a webshop regarding a catalogue of around >100 items. The history of purchases flattened out looks like

Item1 Item2 ... ItemN Sex State
    5     0         0   M    NY
   25    15         0   F    IL
    0     1         1   ?    NY

By playing around with the data, I can deduce simple facts like "90% of all purchases include at least 3 Item1", "If there are at least 4 of Item2, it is likely that Item3 is 0" or "60% of all customers from NY are male, but only 40% of those from IL are". Given the amount of combinations and data there is, the most obvious question: How can I approach wringing out more information from a data set like the above? I'm mostly interested in how one item does or does not entails inclusion of another...


texts: Stanford NER Training - Assign weight to each word

texts:
I am using the Stanford NER to recognize each entity in a search text. Once I identify entities, I need to pass that entities to an algorithm which calculates score for each entity type (e.g. country, customer) using weights of each word. Currently my training data has word and answer like below:
country_training.tsv
    brazil country
    japan country

customer_training.tsv
    hyundai customer
    apple customer

How can I associate weight to each above training data like below so when I can get weight of each word too?
country_training.tsv
    brazil country 1.5
    japan country 4.0

customer_training.tsv
    hyundai customer 4.0
    apple customer 2.3

Please advise.
Weights are input to NER to annotate each entity wiith their weights.


texts: Validation showing huge fluctuations. What could be the cause?

texts:
I'm training a CNN for a 3-class image classification problem. My training loss decreased smoothly, which is the expected behaviour. However, my validation loss shows a lot of fluctuation.

Is this something that I should be worried about, or should I just pick the model which scores the best on my performance measure (accuracy)?

Additional info: I'm fine-tuning the last layer of a Resnet-18 that was pre-trained on ImageNet data in PyTorch. I must note that I'm using a weighted loss function for the training phase since my data is highly unbalanced. To plot the losses, however, I use the unweighted loss as to be able to compare the validation and training loss. I would use the unweighted loss, were it not that the distributions of the training dataset and validation dataset are somewhat different (they are both highly unbalanced however).


texts: Why a GAN trained on same data and same parameters may produce different results?

texts: I am trying to train a Generative Adversarial Network and ran the training a few times with same dataset and same parameters but it seems tp produce different results. Why this may happen?


texts:
I have an input array of shape (1000,20, 4) and output(labels) of shape (1000,25,1). But don't know how to use Keras LSTM library to build a sequential model for this!

Can someone help me design a simple LSTM for doing that? (I tried to use RepeatVector() and TimeDistributed(Dense()) in several ways but I get errors like the following:

model = Sequential()
model.add(LSTM(units = un , input_shape = (20, 4), return_sequences = False))
model.add(RepeatVector(25))
model.add(LSTM(un , return_sequences=True))
model.add(TimeDistributed(Dense(20)))

  ValueError: Error when checking target: expected lstm_419 to have
  shape (25, 20) but got array with shape (25, 1)


texts: Follow Up Question - Why use fit when already have fit_transform?

texts:
This is a follow up question to: What's the difference between fit and fit_transform in scikit-learn models?
I want to know why should we use fit at all when we have fit_transform which is much faster than using fit and transform separately? After all we will always transform the training data after fitting it. Do we have any use of fit all by itself?


texts: Improving Accuracy of the Deep Learning Model


texts:
I have recently started learning Deep Neural Networks and was going through the tutorials online. Everywhere I saw that the topics post Image classification using CNN is a little hazy. No one seems to follow a guide as to what topics should one learn after learning image classification.
I would really appreciate if someone could guide me so as to learn advanced topics and become good to get started with research in this field.
The topics i have covered so far are:

Basic Machine Learning

Linear Regression
Logistic Regression
SVM
Decision Trees
Optimization Algorithms
Regularization Methods
Different Loss Functions
Different Metrics during Training
Feature Extraction
HyperParameter Tuning

Basic Neural Network

ANN
CNN

Image Classification
Image Augmentation before training
Visualizing Filters

I have started learning Object detection using CNN but I have this feeling that I am missing some topics which can strengthen my foundation.
Thanks in advance.
Any guidance is appreciated.


texts:
I am trying to be precise in definitions.

We can solve regression, classification, clusterisation, dimensionality reduction, visualization, feature extraction tasks.

But also there are supervised, unsupervised, ..., tasks of ML.

I see that regression is a part of the supervised learning.

Are there any other particular tasks?
How to make the word "tasks" more clear?


texts: How to detect medicine name from the medicine wrapper

texts:
I have got medicine wrapper ( Packaging ) of different medicines.

I want to detect medicine name out of it.

I'm using Google Cloud Vision to extract all the text from the medicine wrapper.

Text contains medicine mfg info, address, pin code, expiry date, ingredients and most importantly medicine name.

I want to extract only medicine name out of it.

What are the possible ways to get medicine name ?

Thanks!


texts: Combining different features as input to Neural Network

texts:
I use two different sources of information as input to my neural model. The model takes a word as input and produces binary [1/0] output. I have represented each word by using its word embedding (1024 dimensional vector) and its Valence, Arousal, and Dominance Lexicon (3 dimensional vector). I concatenate them and obtain 1027 dimensional vector for each word.

My questions are:

Should I normalize/preprocess those vector before concatenating them? My lexicon values are between [0,1] and my embeddings are the ELMO embeddings produced depending on a context. Therefore I am not sure about the range of the embeddings.
If yes, should I preprocess one of them before merging, what should I do ?
Is there any good resource that might have an answer for my question?


texts: Apply two different Sklearn classifiers to two different subsets of the same data

texts:
I have a dataset that I need to run through a classification Pipeline. The dataset has 2 types of rows:

described: description column POPULATED
non-described: description column EMPTY

I want to apply one classifier targetting ONLY the described data, and another one for the non-described data.
I am currently doing so by separating the dataset, and then preprocessing and feeding the dataset to their corresponding classifier separately. What I want to accomplish is fitting this process into a Sklearn pipeline. It should be something like this:
classifierPipe = Pipeline([('preproc_described', DescPreprocessor),
                           ('preproc_non_described', NonDescPreprocessor),
                           ('clf_described', CLF1),
                           ('clf_non_described', CLF2)
                          ])

classifierPipe.fit(X_train,y_train)

I was reviewing StackingClassifier, but according to the documentation, initial estimators are applied to all the rows in the dataset.
How can I create such a pipeline where each classifier targets a specific subset of the whole dataset?


texts: Constant validation loss & accuracy, training accuracy fluctuates

texts:
I am training a Squeeze-net model for binary classification of images. I have 79968 images for training (50:50 for and against) and 8892 images in the validation set. After 35000 iterations my training accuracy fluctuates between 1 and 0.96875. The validation accuracy is more or less constant between [0.985, 0.986]. The base learning rate is 0.01 and decreased upto 0.00001. As far as I can tell visually training loss doesn't exactly fluctuate between two numbers as such but is in the range [0.02, 0.09] for most part except a few occasional spike.

My question is what can i infer from this?
(1) Overfit?
(2) Model has converged? If not should I reduce the learning rate?
(3) Is the model stuck at a local minima? I am using softmax-with-loss as my loss layer.


texts: How do I handle class imbalance for text data when using pretrained models like BERT?

texts:
I have a skewed dataset consisting of samples of the form:
Category 1 10000
Category 2  2000
Category 3   400
Category 4   300
Category 5   100

The dataset consists of text with data labeled into one of the five categories. I am trying to use the pretrained models like BERT for the classification task but the model fails to identify the
categories 3-5 .I have tried to apply class weights in the loss criterion however it doesn't help much although it gives better performance as compared to simple fine tuning of the pretrained models. I have came to know about SMOTE and other methods in order to handle the class imbalance issues . But since most of the transformer models expect the inputs as text which are later tokenized by their respective tokenizers I am not able to do any kind of oversampling . If there is a workaround for this thing I would be interested to know about it.


texts: How feed a numpy array in batches in Keras

texts:
I have the data in the following format:

1: DATA NUMPY ARRAY (trainX)

A numpy array of a set of numpy array of 3d np arrays.
To be more articulate the format is: [[3d data], [3d data], [3d data], [3d data], ...]

2: TARGET NUMPY ARRAY (trainY)

This consists of a numpy array of the corresponding target values for the above array.

The format is [target1, target2, target3]

The numpy array gets quite large, and considering that I'll be using a deep neural network, there will be many parameters that would need fitting into the memory as well.

How can I push the numpy arrays in batches for trainX and trainY


texts: Real-time classification of audio - thousands of classes

texts:
There is a need to classify an audio stream in real-time - using one from thousands of the defined classes. Up to 5000 classes potentially.

What's the best machine learning algorithm for this task?

DTW? HMM? SVM? Some specific DSP algorithms?


texts: Confusion regarding prediction results of SVM and ANN on feature vectors

texts:
I am making a custom image classifier using Transfer Learning on Inception V3. I have 3 classes of images with ~6K images each. The input dimension of the network is 500X500 and the output of the network is 14X14x2048. I used global average pooling and finally got a vector of size 2048.

To get just a baseline model, I initially trained a linear SVM classifier on this feature vector, (PS: The network has not been trained for this dataset, right now I am just using the forward pass on the Imagenet weights).

The Linear SVM Classifier was giving an accuracy of 81%. (Which was a little surprising, but I guess the pre trained weights were able to capture a lot of distinctions in those images).

Finally, to at least replicate the results using neural networks, I made all the layers up till the 2048 feature vector untrainable, and straight away mapped it to a 3 output softmax activated output layer (which was the only trainable layer).

After putting the training on for ~400 epochs, the validation accuracy remains fluctuating between 33%-38% which is absolutely surprising. It goes on to implicate that SVM classifier on the same feature vectors is able to learn the features which neural networks aren't able to.

Is there any suitable explanation for this phenomenon?


texts:
I am looking for a confidence of model to predict well in a given situation.

So I have a model $f$ (generic, let's exemplify with a regression model of explicit form for brevity).
It well fits on the train-set (by looking at say $r^2$), it well performs on the test set (by looking on RMSE).
The fit is fine enough to predict with this model.
Yet I want to know how confident am I to predict in this particular case, so while typically I look at the confidence $C$ of the model:

$C(f)$

while now I want to know what is the confidence of predicting with this model (subject to $x$):

$C(y=f(x))$

and I believe this shall strongly be related to presence of similar observations in the train/test sets and performance on similar observations.

Any hints on this welcomed.


texts: Define a Model for predicting missing values in Data Set

texts:
I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset.
For example: I have a dataset which contains information about a product. This can be category, brand_name etc. Let's assume there are information missing of brand name for a specific product(4261 rows / 10 000). How can this information be predicted based on the information I have in my dataset?


texts: Feature Selection, Machine learning and Time Series analysis, for large financial timeseries

texts:
I have m( around O(millions) ) of rows of type

timestamp | val | ind1 | ind2 | ind3 | .... k entries

My task is to predict the value of "val" for any future given time. How should I model this as a machine learning problem. Note that we want the value of "val" given the past data without knowing the current value of

ind1 | ind2 | ind3 | .... etc.

of the same row. This is different in that sense.

Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j>i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.

Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?

About data I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe "val" is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.

I have m( around O(millions) ) of rows of type

timestamp | val | ind1 | ind2 | ind3 | .... k entries

My task is to predict the value of "val" for any future given time. How should I model this as a machine learning problem. Note that we want the value of "val" given the past data without knowing the current value of

ind1 | ind2 | ind3 | .... etc.

of the same row. This is different in that sense.

Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j>i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.

Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?

About data I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe "val" is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.

And please give any idea, How will system learn iteratively as new data rows are added. I do not want to do the whole training again and again.


texts: How to use Scikit-Learn Label Propagation on graph structured data?

texts:
As part of my research, I am interested in performing label propagation on a graph. I am especially interested in those two methods:

Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002 http://pages.cs.wisc.edu/~jerryzhu/pub/CMU-CALD-02-107.pdf
Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, Bernhard Schoelkopf. Learning with local and global consistency (2004) http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.115.3219

I saw that scikit-learn offer a model to do that. However, this model is supposed to be applied on vector structured data (i.e. data points).

The model builds an affinity matrix from the data points using a kernel, and then run the algorithm on the constructed matrix. I would like to be able to directly input the adjacency matrix of my graph in place of the similarity matrix.

Any idea on how to achieve that? Or do you know any Python library that will allow to run label propagation directly on graph structured data for the two aforementioned methods?

Thanks in advance for your help!


texts: Lower loss always better for Probabilistic loss functions?

texts: I am working on an neural net int Tensorflow that predicts percentages for win, draw, loss for given data of a game. The labels I provide are always {1, 0, 0}, {0, 1, 0} or {0, 0, 1}. After some epochs my accuracy doesn't increase any further, but the loss still decreases for a many epochs (also on the validation set, though very slowly). I am using a softmax activation in the last layer and the categorical crossentropy loss function provided by Keras. I was wondering if in this case, lower loss always corresponds to better probabilities (because I obviously wouldn't want the net to output only values like 1 or 0 for probabilities), or in other words, does this net output the "true" probabilites and if so, why does it do that?


texts: When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?

texts:
I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building a model.

I understand the difference between OHE, LabelEncoder and DictVectorizor in terms of what they are doing to the data, but what is not clear to me is when you might choose to employ one technique over another.

Are there certain algorithms or situations in which one has advantages/disadvantages with respect to the others?


texts: Machine learning dataframe dimension concept vs NumPy dimension

texts:
From Machine Learning for Absolute Beginners: A Plain English Introduction:

  Contained in each column is a feature. A feature is also known as variable, a dimension or an attribute - but they all mean the same thing.

From here (the supplement file for this book):

In NumPy, each dimension is called an axis.
The number of axes is called the rank.

For example, the above 3x4 matrix is an array of rank 2 (it is 2-dimensional).
The first axis has length 3, the second has length 4.

An array's list of axis lengths is called the shape of the array.

For example, the above matrix's shape is (3, 4).
The rank is equal to the shape's length.

The size of an array is the total number of elements, which is the product of all axis lengths (eg. 3*4=12)

Question: Is the dataframe dimension completely different not related to the NumPy dimension (just same word but describing different concept)?

I am learning Python and Machine learning but familial with R and R dataframe from statistical perspective


texts: How to train a multi inputs deep learning model using every combination of inputs?

texts:
I am beginner in deep learning. I want to create a multi inputs CNN model in Keras. The model takes two inputs of images to give the two images class. The
two images from differnt datasets that have the same classes exactly.

During training, I want to try every combination of the two images if the images are from the same class.

I created used multi input one output model, but I could not create a custom fit_generator function.

How can I make a generator that try every combination of the two images if the images are from the same class ??

Please help me. Please help me.


texts:
I have a list of ages and then a binary value associated with it (e.g. is the person overweight)

I would like to calculate from this the p-value.

I have plotted the values and then calculated the probability that at a certain age the person is overweight, but I don't think this is the p-value.


texts: Auto encoders to convert black and white image?

texts:
I am trying to get an intuitive understanding of auto encoders and how it works colorizing B/W images my questions are,
What kind of input data and test data is needed? What will be the class labels? How will the colouring be achieved?


texts:
I have two dataframes (both of different lengths, etc) but they have exact same columns and I would like to explore the distribution of one of these columns by plotting a boxplot.

However, how do I get it on the same axis? And how can I set the y-axis for this? The below code puts the boxplots side by side in separate graphs but I would like them to have the same axis so I can compare them easier.

f, axes = plt.subplots(1, 2)
sns.boxplot( x="status",y="assets" ,data=df1, palette="Set3",ax=axes[0])
sns.boxplot( x="status",y="assets" ,data=df2, palette="Set3",ax=axes[1])
fig.tight_layout()
plt.show()


texts: How does LightGBM deal with value scale?

texts:
I understand that the loss metric can be used as linear, or log, or other things. This is documented at http://lightgbm.readthedocs.io/en/latest/Parameters.html?highlight=logloss#metric-parameters

I would like to understand how LightGBM works on variables with different scale. In other words, is it necessary for me to harmonize scale when running LightGBM? (I am used to linear regression where you need to get into linear scale.)

If I had inputs x1, x2, x3, output y and some noise N then here are a few examples of different scales.

$y = x1 + x2 + x3 + N $
$y = exp(x1 + x2 + x3 + N) $
$y = log(x1 + x2 + x3 + N) $
$y = sqrt(x1 + x2 + x3 + N) $
$y = log(x1 * x2 * x3 * N) $


texts: Decode SQLite database blobs: how to start?

texts:
Updated: see bottom of this question.

Not sure if I am at the right channel here but I have good hopes someone here might be able to help me.

I am trying to process and analyse data from a Footscan system that is exported from the Footscan 9 Gait Essentials software to a .rsdb database. This database is a standard SQLite database with a different file extension.

I have no problems accessing the data via Python but the most interesting data (the raw data from the sensor) is stored as a 'blob' (see sqlite documentation for more details).

Exporting this data from SQLitePro to .csv shows that the data in the blob looks like this (although much longer):

<9c4e79cb cbd09de5 c380fc78 a74e819f ... ab2172c7 d9cf311a 357aebac>

My question is: what would be my first steps to try to discover how this data is encoded? And does anyone maybe recognize this type of encoding?

Extra hints: The data represents all forces of a force plate with 64x64 force sensors, sampled at 300Hz, so it's most probably a multidimensional array.

Update

I am making some progress. As commented below, the values are most probably hexadecimal values, but I'm not sure if the values are base 16: most values seem to be >10^9...

Furthermore: I found a table in the database called 'Contact' which linkes to entries in the blob table which contain the most hexadecimal values (>40000). My guess is that these are the entries which are recognized as actual steps. I extracted the data from one of these steps here (Contact id 2; blob id 24). This blob contains 50792 hexadecimal values.

This is the entry for this step in 'Contacts':

id: {35d9a73e-608d-4ba3-9823-de0540f11c93}
timestamp: 2
deleted: NULL
FrameWidth: 64
Frames: 24
FrameOffset: 12204
FrameCount: 214
FrameHeight: 64
FrameStoreWithContacts_Contacs: {5d10546e-5ef3-4469-b898-d7b9bd58458f}
OriginalId: {32ae0f73-8022-4151-92e7-510c0e86d09d}
orphan: false

Full export of the blob table can be downloaded here.
The full .rsdb can be found here (you might have to rename to .sqlite).
The tables 'Foot' and 'Region' in the full database might contain useful data too, but I'm not yet sure how...


texts: Searching for ressource recommendations (books,papers) to validate the results of a Topic Model (LDA)?

texts:
Hi I am building a Topic Model Process with Python.
To do this I am using the LDA.

However I am having trouble to determine the ideal amount of topics.

Currently I am using the CoherenceModel Module of the gensim libary to evaluate my results. Are there better/ more efficient ways to this?

Are there any good books or papers that tackle this topic?

If anyone here has experience with Topic Modelling, how do you evaluate, validate and interpret the results of the LDA?

Any help is welcome.


texts: Which are the latest Optimization techniques in artificial intelligence?

texts: My project work is optimization in power system using artificial intelligence (like fault location and classification,load forecasting and context awareness and IoT etc ) and I have used PSO (particle swarm optimization, supervised fine tuning strategy etc). Please can anyone suggest a new technique which will give best result in comparison to other techniques?


texts: Does using unimportant features hurt accuracy?

texts:
I'm using the scikit-learn gradient boosting classifier found here.

If I run the classifier on the same data without seeding the random number generator, I get different feature importances, and notably, different ones are 0 each time. If I just look at the features where the feature importance is < .005, most of the same ones appear, but there is a little variation there too.

My question is, does this have an impact other than computation time? If it were to hurt accuracy, I imagine that would be due to over-fitting. However, if the classifier already identifies the relatively unimportant features, then doesn't that take care of the problem?

If it does hurt accuracy, how should I select which features to get rid of?


texts: Modern reference on general Deep Learning theory

texts: In the present, Deep Learning is experiencing lightspeed growth, with plethora of new architectures and radiant ideas emerging each month. Since the last few years several influential ideas, applied in different papers have been developed - among which most important are Transformer, EfficientNet, MobileNets. Is there some modern reference, in the spirit of the famous https://www.deeplearningbook.org/, in some way, generalizing the most prominent advances in Deep Learning?


texts: Pattern Recognition (Bishop) - Maximum Likelihood

texts:
This refers to Chapter 2, section 2.2 (page 75) - Pattern Recognition and Machine Learning, Christopher Bishop.

The question is related to calculation of Maximum Likelihood variable:

Maximum likelihood is given as:
$$p(D|\mu) = \prod_{n=1}^{N}\prod_{k=1}^{K}\mu_{k}^{x_{nk}} = \prod_{k=1}^{K}\mu_{k}^{m_{k}} \tag{2.29}$$
where $m_{k} = \sum_{n}x_{nk}$ $\implies$ #occurrences of event $k$, $x_k$, over $n$ trials where, k={1,$\ldots$,K} and, n={1,$\ldots$,N}. Also, $\mu_{k}$ being probability,  and $D$, the observed data.

  Author applies Lagrangian multiplier $\lambda$ and take log-likehood
  of above equation to get: $$\sum_{k=1}^{K}m_{k}ln{\mu_{k}} + \lambda(\sum_{k=1}^{K}\mu_{k} - 1) \tag{2.31}$$  given the
  (probability) constraint): $$\sum_{k=1}^{K}\mu_{k} = 1$$

  Maximization of eqn. 2.31, w.r.t $\mu_{k}$ and equating it to 0,
  yields: $$ \mu_{k} = -m_{k}/\lambda \tag{2.32}$$
  Author substitutes the constraint in eq 2.32 and finally gets:
  $$
\mu_k^{ML} = \frac{m_k}{N} \tag{2.33}
$$

I am losing it in the last step - my (partial/ wrong) derivation of 2.31 w.r.t. $\mu_{k}$ yields:
$$ \sum_{k=1}^{K}\frac{m_{k}}{\mu_{k}} + \lambda\cdot K = 0$$

I am not able to understand:

Where am I going wrong
How to move ahead to get the result (only, if my derivation is correct)


texts: What options are out there to extract text from a group of PDFs where each PDF is formatted differently but contains the general same content

texts:
Think insurance/medical forms that come from different companies.  There is no standard on formatting.  I am trying to extract the text based on each section of a given form.  A form might have a section of texted divided by whitespace, border, lines etc.  I need to keep the text in context of that section.

Textract from Amazon is something I am waiting for, but was wondering if there was some other alternatives.


texts:
I am trying to use deviance in order to optimize my network. Note that my $y_{true}$ values are equal to 0 more than in 90% of cases (can also be 1 or 2).

The deviance is calculated as : $2 \times (log(y_{true}) - log(y_{pred}))$

The problem in my case is that $log(y_{true})$ when $y_{true}= 0$ gives me $-\infty$.

I derivated the deviance formula in my case (poisson unscaled deviance) and got this result :

If $y_{true} = 0$, then deviance is : $2D \times y_{pred}$ where $D$ is a feature of my data.
If $y_{true} \ne 0$, then deviance is : $2D \times (y_{true} \times ln(y_{true}) - y_{true} \times ln(y_{pred}) - y_{true} + y_{pred})$

This is the loss i came up with :

def custom_loss(data, y_pred):

    y_true = data[:, 0]
    d = data[:, 1:]
    # condition
    mask2 = keras.backend.not_equal(y_true, 0) #i.e. y_true != 0
    mask2 = KB.cast(mask2, KB.floatx())
    # returns 0 when y_true =0, 1 otherwise
    #calculate loss using d...
    loss_value = 2 * d * y_pred + mask2 * (2 * d * y_true * KB.log(y_true) + 2 * d * y_true * KB.log(y_pred) - 2 * d * y_true)
    return loss_value

Using a mask in order to only compute logs when  $y_{true}$ is different from 0. I don't think the formula is wrong but it still gives me NaN as loss result...

If anyone knows what is the matter here, or knows about a way to get around this problem i'd be very thankful.