texts:
I have a queuing model written in Scala where different categories of people end up a different queues. We have a dataset providing a map of features to the numbers of people ending up at each queue, ie multiple inputs to multiple outputs (continuous values)

I have some experience using mllib for single value predictions in Scala but I can't see that multiple outputs are supported. It doesn't even look to me that mllib has continuous value output support as I can't see how to get a layer without an activation function.

Does anyone know of an ml library that can both support multidimensional regression (please correct me if I have the terminology wrong) and integrate with Scala?

Going forward I'd also love to try an RNN as our data has a time sequence element to it, but I believe this is even rarer in ml libraries.


texts:
I'm trying to figure out a good (and fast) solution to the following problem:

I have two models I'm working with, let's call them players and teams. A player can be on multiple teams and a team can have multiple players). I'm working on creating a UI element on a form that allows a user to select multiple teams (checkboxes). As the user is selecting (or deselecting) teams, I'd like to display the teams grouped by the players.

So for examples:

If the selected teams have no players that intersect, each team would have its own section.
If the user selects two teams and they have the same players, there would be one section containing the names of the two teams and all the players.
If TEAM_A has players [1, 2, 4, 5] and TEAM_B has players [1, 3, 5, 6]. There would be the following sections: SECTION_X = [TEAM_A, TEAM_B, 1, 5], SECTION_Y = [TEAM_A, 2, 3], SECTION _Z = [TEAM_B, 3, 5]

I hope that's clear. Essentially, I want to find the teams that players have in common and group by that. I was thinking maybe there is a way to do this by navigating a bipartite graph? Not exactly sure how though and I might be overthinking it. I was hoping to do this by creating some type of data structure on the server and using it on the client. I would love to hear your suggestions and I appreciate any help you can give!


texts: Find 10 most similar data points from 100?

texts:
I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the top 10 most similar cells from 100.
Is there any way i can solve this?


texts: Training stateful LSTM with different number of sequences

texts:
I'm using a stateful LSTM for stock market analysis, and I have varying amounts of data for each stock, ranging from 20 years to just a few weeks (i.e. for newly listed stocks).

I use 3 years of data as a minimum for training as I want to create some state within the network. I set a year as my sequence length, so if I have 12 years of data then I will submit 4 batches with 3 sequences in each. Only after I've submitted all batches do I then reset the network state ready for the next stock.

But is there any issue training with differing number of sequences? E.g. if I train with a company that has 20 years of data then the network will build up much more state than a company that I only have 3 years of data.


texts: k- means clustering on Markov chain trasition probability

texts: I have data set of 50 students. I want to cluster them on their sequential data ( While doing a job they followed multiple sequences A, B, c total 7 stages). I am planning to apply k-means clustering on their first order Markov chain transition probability matrix. So that means, I have 50, 7x7 transition probability matrix. Each 7x7 matrix has 49 data point. So, I can make a 50x49 matrix. If I apply k- means on this matrix, is it a proper approach for clustering sequential data?


texts: Reason for capping Learning Rate (alpha) up to 1 for Gradient Descent

texts:
I am learning to implement Gradient Descent algorithm in Python and came across the problem of selecting the right learning rate.

I have learned that learning rates are usually selected up to 1 (Andrew Ng's Machine Learning course). But for curiosity reasons, I have tried alpha = 1.1 and alpha = 1.2.

I can see in the case of alpha = 1.2, we reach the lower cost faster than the other learning rates (simply because the curve touches the bottom first). Is it safe to say that alpha = 1.2 is the best rate?

I plugged in the theta values, where alpha = 1.2, to predict the price of an item, my implemented function provided the same answer as Sklearn's LinearRegression() in lesser iterations than it did with alpha = 1.0. Using lower alpha rates would increase the number of iterations.

So, why is the learning rate capped at 1? Is it mandatory or suggested?

Should I forget about selecting learning rates and let functions like LinearRegression() take care of it automatically in the future?

I am new to machine learning and I want to understand the reasoning behind the algorithms rather than calling the functions blindly and playing around with parameters using high-level libraries.
Feel free to correct me if I have understood the concepts wrong.


texts: accuracy and loss NAN for keras multi-label Neural network learning

texts:
When I ran a Neural Network modeling for multi-class labeling using Keras, the accuracy, loss, val_accuracy, and val_loss all seems to have nan at some point or other during the training process...
Running environment and other info

Keras version: '2.4.0'
TensorFlow version: '2.3.0'
OS: Ubuntu 20.04.1    LTS
GPU: Radeon RX 580 Series
Python running in Jupyter Notebook    (there is no library version issue)

Thank you for you kind help.
Code I used:
# data import n prep
data_irr = pd.read_csv( 'https://www.dropbox.com/s/97awk5f9rgn5k87/irrigation_machine.csv?dl=1')
df_TRAIN = data_irr.sample( frac=  78./100 )
df_TEST  = data_irr.drop( df_TRAIN.index )

# TRAIN data
X_train = df_TRAIN.drop( df_TRAIN.columns[ [0,21,22,23] ], axis= 1 ); X_train
Y_train = df_TRAIN[['parcel_0','parcel_1','parcel_2']]; Y_train

X_train = X_train.to_numpy(); Y_train = Y_train.to_numpy()

# TEST data
X_test = df_TEST.drop( df_TEST.columns[[0,21,22,23]], axis= 1 ); X_test
Y_test = df_TEST[['parcel_0','parcel_1','parcel_2']]; Y_test.head()

X_test = X_test.to_numpy();  Y_test = Y_test.to_numpy()

## model architecture

model1 = keras.Sequential()
model1.add(
            keras.layers.Dense( 64, activation= 'relu', input_shape= (X_train.shape[1],) )
)
model1.add(
            keras.layers.Dense( 3, activation= 'sigmoid' )
)

model1.compile(
                optimizer= 'adam', loss= 'categorical_crossentropy', metrics= ['accuracy']
)
model1.summary()

model1.fit(
            x= X_train, y= Y_train,
            epochs= 15, validation_split= 20./100,
            verbose= 2
)

The result of the training is:
Epoch 1/15
39/39 - 0s - accuracy: nan - loss: nan - val_accuracy: 0.5481 - val_loss: -5.4487e-02
Epoch 2/15
39/39 - 0s - accuracy: nan - loss: nan - val_accuracy: nan - val_loss: nan
Epoch 3/15
39/39 - 0s - accuracy: 0.5433 - loss: nan - val_accuracy: 0.7981 - val_loss: 1.8837e-07
Epoch 4/15
39/39 - 0s - accuracy: nan - loss: nan - val_accuracy: 1.1190 - val_loss: 0.3816
Epoch 5/15
39/39 - 0s - accuracy: nan - loss: nan - val_accuracy: 0.8221 - val_loss: nan
Epoch 6/15
39/39 - 0s - accuracy: 0.5529 - loss: nan - val_accuracy: 0.9802 - val_loss: 0.2807
Epoch 7/15
39/39 - 0s - accuracy: 0.5473 - loss: nan - val_accuracy: 0.8893 - val_loss: 0.0128
Epoch 8/15
39/39 - 0s - accuracy: 0.5537 - loss: nan - val_accuracy: 0.8287 - val_loss: inf
Epoch 9/15
39/39 - 0s - accuracy: 0.6186 - loss: nan - val_accuracy: 0.7756 - val_loss: 1522.9906
Epoch 10/15
39/39 - 0s - accuracy: 0.5905 - loss: nan - val_accuracy: 0.8085 - val_loss: 0.1824
Epoch 11/15
39/39 - 0s - accuracy: 0.6250 - loss: nan - val_accuracy: 0.8007 - val_loss: 0.0545
Epoch 12/15
39/39 - 0s - accuracy: 0.6514 - loss: nan - val_accuracy: 0.8980 - val_loss: 13932.9346
Epoch 13/15
39/39 - 0s - accuracy: 0.5457 - loss: nan - val_accuracy: 0.8000 - val_loss: nan
Epoch 14/15
39/39 - 0s - accuracy: 0.5545 - loss: nan - val_accuracy: 1.9837e-07 - val_loss: -1.8476e+10
Epoch 15/15
39/39 - 0s - accuracy: 0.5409 - loss: nan - val_accuracy: 2.0768e-07 - val_loss: 0.5513
<tensorflow.python.keras.callbacks.History at 0x7f92602e0f98>


texts:
Currently I read a paper about symmetric skip connections for autoencoder (link). One experiment of them changes the the 'training patch size'.

In my understanding patches are sub-boxes of an image that is used at one time of an convolutional layer. So if you have a 3x3 filter the patch is a part of the image with the size 3x3.

Do they mean by 'training patch size' the size of the input image? (The network is a symmetric autoencoder, so the input size is arbitrary)

Thanks in advance


texts: R: Revalue multiple special characters in a data.frame

texts:
R noob here..

I have the following data frame

>data
 Value Multiplier
1  15    H
2   0    h
3   2    +
4   2    ?
5   2    k

where the multiplier is of class factor. The values of K & k is 3, + is 5 and ? is 2.

I have used

> data$Multiplier <-  revalue(data$Multiplier, c("+"="5"))
> data$Multiplier <-  revalue(data$Multiplier, c("?"="2"))
> data$Multiplier <-  revalue(data$Multiplier, c("K"="3"))
> data$Multiplier <-  revalue(data$Multiplier, c("k"="3"))

Is there a better way of doing it?


texts: What are good public datasets for time series analysis with “certified” (by papers in the literature) good predictors of the target variable?

texts: I have to test different models for time series forecasting and predictors (exogenous covariates) goodness evaluation and I would like to use datasets used in relevant scientific publications that have "certified" the goodness of some predictor variables (of the target variable), so I can compare the results and try to refine my models.


texts:
Machine learning can be divided into several areas: supervised learning, unsupervised learning, semi-supervised learning, learning to rank, recommendation systems, etc, etc.

One such area is PU Learning, where only Positive and Unlabeled instances are available.

There are many publications about this, usually involving a lot of mathematics...

When looking at the literature, I was expecting to see methods similar to self-training (from semi-supervised learning), where labels are adjusted gradually according to the classifier margins.

I don't think these is what practitioners from the area do, and I was unable to navigate the mathematics or to find a survey on PU learning.

Could someone from the area perhaps clarify what said practitioners do? Why can they not just use a binary classifier where the negative class=unlabeled? Can negative labels exist among the unlabeled data? What is the goal and what metrics exist to evaluate said goal?


texts: Term for an identifier that has been superseded

texts:
Is there a 'proper' term for an ID (or IDs) that have been superseded by (or merged into) another ID?

My use case:

rsIDs are used by geneticists to refer to a SNP. These take the form of a string 'rs#####'
Over time, some of these rsIDs are merged into others when it becomes evident that they refer to the same SNP with another rsID. eg rs879161658
it is useful to keep track of these old identifier(s) that were used, even when using the latest rsID as the canonical identifier. rs879161658 will still be found in some legacy resources
it is not possible to control the creation or naming of these rsIDs

I've considered AKA, alias, and even Née (born as) but is there something traditionally used? I could ask this on a language-focused stackexchange but here seems like it would be more useful


texts: Adding a custom constraint to weighted least squares regression model

texts:
I am trying to run a weighted least squares model that looks something like this (but could be different):

$y = \beta_0 + \beta_1 x + \beta_2 log(x) + \epsilon$

with weights $w_1, w_2, ..$

However, I know, from external knowledge, that whatever the model the outcome must asymptotically converge to a constant for large values of $x$. How can I get an OLS estimate with this constraint.

As an example, let's say if I knew the asymptote $c$, then I can add two fake data points to my model, with very high values of $x$ and very high weights $w$ and $y=c$, and run the normal WLS model and it would give me what I need - except I don't know the value of $c$. Is there a way to impose this constraint - maybe through adding a custom error term to the model?


texts: Can we use k fold Cross Validation without any extra (excluded) Test Set?

texts:
I have seen this in two Papers:

The authors use 10 fold cross validation, and then present the results from this validation or even odder the results from the best Fold as their modelling Result.

there has been no testing data put aside to validate the final model. These are peer reviewed papers i am talking about. Is this right to do ? Can one even cite such sources?


texts: I am trying to classify gestures. I am using Python's scikit learn library classification algorithms for that. I have collected depth images for this purpose. 200 samples are collected for each gesture. Each gesture is made up of 25 frames and each frame is of size 240x420. I tried PCA for dimensionality framewise for reducing the size of each gesture (200 samples each) to make it easy to run on the machine. Still the large size of the data make it difficult to run in my machine when the number of gestures to classify are larger than 4. I am looking for methods to make it run on my machine.


texts: What does it exactly mean when we say that PCA and LDA are linear methods of learning data representation?

texts: I have been reading on representation learning and I have come across this idea that PCA and LDA are linear methods of data representation, however, auto-encoders provide a non-linear way. Does this mean that the embedding learned by PCA can be transformed only linearly to reproduce data points ?


texts: Correlation with Multi-Dimensional Clustering

texts:
I have a dataframe with multiple features, where I'm selecting 3 features to cluster on. Ex.
Col1 | Col2 | Col3
-------------------
A    |  1  |   i
B    |  1  |   j
C    |  2  |   k
D    |  3  |   j

I'm trying to cluster such that If there is a matching value in any two columns, it will add that row to the same cluster, for all matches within the values in a cluster.
So from this table, my intended output for a cluster would be:
A    |  1  |   i
B    |  1  |   j
D    |  3  |   j

Is there a clustering algorithm that I can use to achieve this type of clustering?


texts:
How to read factor vs. factor plots?

Particularly, what do negative values mean? Also, why is it possible for features to have negative effect on the factors?


texts:
I've successfully used Spike Slab in the past, but with this data it seems like something is going wrong. My code is:

>require(spikeslab)
>set.seed(2)
>model1_ss<-with(data_use,spikeslab(gdeaths_per_100_thousand~gdealers_per_100_thousand+Physically.Unhealthy.Days+Mentally.Unhealthy.Days, na.rm=TRUE))
>summary(model1_ss)

the first 10 lines of data_use is

 Physically.Unhealthy.Days  Mentally.Unhealthy.Days gdeaths_per_100_thousand    gdealers_per_100_thousand
 5.2        4.1     19.70210418  30.11828855
 4.9        4.1     21.6723146   32.83851989
 5.1        4.3     23.52217417  39.4076506
 5.1        3.6     NA           38.03625359
 3.4        4.1     17.78983533  27.28191219
 3.4        4.2     19.4576324   32.2906761
 3.3        3.8     13.9248167   40.75855495
 3.3        3.8     12.05513916  36.2194827
 5.9        3.9     NA           31.52340694

And it returns this, which is totally unlike what I expect from from Spike Slab:

    Length Class      Mode
summary           4  data.frame list
verbose          10  -none-     list
terms             3  terms      call
sigma.hat         1  -none-     numeric
y              2132  -none-     numeric
xnew          12792  -none-     numeric
x             12792  -none-     numeric
y.center          1  -none-     numeric
x.center          6  -none-     numeric
x.scale           6  -none-     numeric
names             6  -none-     character
bma               6  -none-     numeric
bma.scale         6  -none-     numeric
gnet              6  -none-     numeric
gnet.scale        6  -none-     numeric
gnet.path         2  -none-     list
gnet.obj         16  lars       list
gnet.obj.vars     6  -none-     numeric
gnet.parms        6  -none-     numeric
phat              1  -none-     numeric
complexity      500  -none-     numeric
ridge           500  -none-     list
model           500  -none-     list

Is there a way to interpret this that I don't know about? Or is there something wrong with how I'm running it? Thanks so much for any help, I have no idea what to try.


texts: Deep Learning approach for Signature extraction from emails

texts:
Deep Learning (DL) is now days are used in various fields. I may like to know any idea/approach on usin DL for Signature extraction from emails.

There are some papers which uses ML like http://keg.cs.tsinghua.edu.cn/jietang/publications/f142-tang.pdf
http://www.cs.cmu.edu/~vitor/papers/sigFilePaper_finalversion.pdf     there are few more.

There are some open source libraries in Python like talon which uses NLP (Regular expressions and other techniques) and uses ML algorithms ( SVM ) for the same.

I have done a little research but I am not getting any resource or any idea on how to proceed with DL for this task.
I thought of using seq2seq models but I am not sure it can be of much help for this task. I am not sure whether summarization techniques can be useful.
Any idea or resources for the same can be helpful.


texts:
I have three dataframes. Their shapes are (2656, 246), (2656, 2412) and (2656, 7025). I want to merge dataframes as above:

So It will result a (2656, 9683) Dataframe. Thanks for any help.

Typo on image: on Dataframe 3, it will 7025, not 5668.


texts: Constructing function - f(x,y) for the given minimums (Python)

texts:
Problem Statement:
I need to construct a function f(x,y) in which there're 3 minimums. 2 local and 1 global which are written below.

Locals are: z = f(0.2,0.3) = 0.7 | z = f(0.6,0.8) = 0.8
Global is: z = f(0.85,0.5) = 0.6

What I've tried so far
I've tried Polynomial regression to construct features that estimate the given minimum points. However, an estimated function z = f(x,y) just going through all the given minimum points without considering them as minimums. I need an approach by means of which I can construct a function that has 3 minimums that are mentioned above. I'd be more than happy if you could provide a code snippet of your approach. As a programming language, I use Python.
Code
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D

degree = 6

def func(x1, x2):
    arr = []
    for i in range(degree, 0, -1):
        arr.append(x1 ** i)

    for i in range(degree, 0, -1):
        arr.append(x2 ** i)

    arr.append(1)

    return np.array(arr)

def der_func(x1, x2):
    arr1 = []
    arr2 = []
    for i in range(degree - 1, -1, -1):
        arr1.append((i + 1) * x1 ** i)

    for i in range(degree + 1):
        arr1.append(0)

    for i in range(degree):
        arr2.append(0)

    for i in range(degree - 1, -1, -1):
        arr2.append((i + 1) * x2 ** i)

    arr2.append(0)

    return np.array([arr1, arr2])

given_x = np.array([[0.2, 0.3],
                    [0.6, 0.8],
                    [0.85, 0.5]])

given_y = np.array([0.7, 0.8, 0.6])

data_x = []
data_y = []

for i in range(len(given_x)):
    x1x2 = given_x[i]

    data_x.append(func(x1x2[0], x1x2[1]))
    data_y.append(given_y[i])

    if i < 3: # in case there's more than 3 data point in given_x
        data_x.append(der_func(x1x2[0], x1x2[1])[0])
        data_y.append(0)

        data_x.append(der_func(x1x2[0], x1x2[1])[1])
        data_y.append(0)

data_x = np.array(data_x)
data_y = np.array(data_y)

w = np.linalg.inv(data_x.T @ data_x) @ data_x.T @ data_y
pred_y = data_x @ w

cost = np.mean((data_y - pred_y) ** 2)
print("Cost | ", cost)

#######################################

X = np.linspace(0, 1, 100)
Y = np.linspace(0, 1, 100)

X, Y = np.meshgrid(X, Y)
Z = np.zeros(np.shape(X))
for i in range(len(X)):
    for j in range(len(X[0])):
        Z[i, j] = np.array([func(X[i, j], Y[i, j])]) @ w

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.contour(X, Y, Z, 150)
ax.scatter(given_x[:3, 0], given_x[:3, 1], given_y[:3])
plt.show()


texts: Metric (other than RMSE, MSE, etc.) to select the best model to better detect peaks in weather forecasting

texts:
I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know.

When I looked at the results, second model (RMSE = 0.15) was better able to detect some of the peaks than the first one (RMSE = 0.1). For example look at the two results:

Best model (RMSE = 0.1):

Here is the second model's result(RMSE = 0.15):

The model which is able to detect all the peaks, as much as possible (although its RMSE is not the least among all of the models), is more preferable than a model which has less RMSE but is not able to detect peaks.
I searched through the web but didn't find what I was looking for.

Can anyone suggest me a code to evaluate the results of the models, based on their ability to detect peaks better?

Suppose the results are simply 2 arrays:

import numpy as np
np.random.seed(10)
predicted_1 = np.random.rand(10, 1)
predicted_2 = np.random.rand(10, 1)
original = np.random.rand(10, 1)


texts:
I want to use a CNN to extract features from a dataset. My questions are:

What is the stopping criteria when training a CNN to extract features?
What is the difference between extracting features after training for 50 epochs or 100 epochs?
How should I choose the best layer for feature extraction?
Does it depend on the accuracy of the model?


texts: What is the meaning of the bubbles / spikes in the shap values plot?

texts:
Here are an example of shap values plot from here.
How to interpret the 'bubble' or 'spikes' on this shap values plot I highlighted in yellow color?


texts: How do I use matrix math in irregular neural networks such as those generated from neuroevolution (NEAT)?

texts:
I understand how to structure the matrix when every node in a layer is fully connected to every node in adjacent layers and I understand that in "irregular" neural networks I can just process each node individually. However, there are no explanations or examples online of how to structure a matrix for an "irregular" neural network. How would I handle recurrent connections? Would I just fill in the "gaps" in the matrix with zeroes? Take the irregular neural network in this diagram:

Could I somehow combine (or get the dot-product of):

[i0  i1  i2] and

[[w0 w1 0  w9 0 ]
 [0  w2 w3 0  0 ]
 [0  0  0  w4 0 ]
 [0  0  0  w5 w7]
 [0  w8 0  w6 0 ]]

to find [o0 o1 o2]? Would I need to give the input vector an additional two values of 0?


texts: How to create a for-like loop in Orange3

texts:
This question has been asked before here but never received an answer.

I'm planning to write a bunch of components that do some processing on single images. However, I have many such images and would like to find a way to iteratively run over a list of such images and execute that single-image component.

Is this at all possible in Orange3?

If not, do you have any ideas about how to build a component for that? I was thinking, perhaps it's possible to create a component "Repeat" that takes a single input (e.g., a list of image file paths) and repeatedly generates an output, or refreshes its single output repeatedly with new data. Or something along those lines.

Alternatively, I have to create a higher-level component that just takes a list of image file paths, processes them and outputs a new list of file paths pointing to the processed images. But from a OO-perspective, this component does too much and will be getting too complex. I'd rather keep the components simple and "atomic".

Does this make sense?


texts: PCA on MNIST dataset results in very poor performance

texts:
I am trying to build a model for classifying MNIST dataset using SVM. With raw features I am getting accuracy of around 94% (using linear kernel).

When I tried it with PCA, with different number of components (35,50,250,500) I am getting accuracy around 11%.

What can possibly the reason for this?


texts: Why AutoML open source tools only available for Structured Data only?

texts:
The open-source tools and libraries for AutoML such as AutoSklearn, AutoWeka, TPOT, etc.
they all work on structured data.
are there any open-source tools/libraries (non-commercial) that are available for unstructured data such as images/text/audio/video?
if not, why?!


texts:
I'm looking to parse human-generated cyber-physical sensor tags into a series of standardised tags and identifiers.
For example "Rm3.ZnT-SP" refers to ROOM ID:3 ZONE TEMPERATURE SETPOINT
The "state of the art" involves using CRF to tag each character using BIO and active learning to provide the labels. Note it's not always this easy - the same output can be encoded many different ways (T, temp, tmp etc.)
It seems there should be a way of avoiding the CRF step - I'd like to just map from the input character sequence to an output sequence of embedding without having to provide BIO labels. This in itself seems to be a fairly standard Seq2Seq problem. What I'm unsure about is the fact that the output sequence if a combination of (known in advance) embeddings (tags/tokens) and (unknown) identifiers from the original sequence.
This seems somewhat similar to NLP - for example, if I was translating from one language to another presumably you translate the words but there will be some literals which are just passed through verbatim  (i.e. a street number or a drivers license number). It also has some characteristics of Named Entity Recognition but from what I've seen of NER it only identifies the named entities.
Any pointers to similar problems are appreciated.


texts: Cannot underfit/overfit on the IRIS dataset

texts:
I am playing with the IRIS dataset and want to see underfitting and overfitting in action. I am using a multilayer perceptron (2 layers).

The problem is that I cannot underfit or overfit the data (see the plot below). I understand why I cannot underfit: it might happen if data is easily separable, but why I cannot overfit? The dataset capacity is 600 (# of samples (150) times # of features (4)), so I should be able to overfit using a network with a capacity bigger than that. I am trying to use a multilayer perceptron with a total # of parameters ranging from 15 to ~32000, but neither under-, nor over-fitting happens. What is going on? Maybe overfitting does not happen for the same reason, because the data is easily separable? Thank you!


texts:
In value-function approximation, in particular, in deep Q-learning, I understand that we first predict the Q values for each action. However, when there are many actions, this task is not easy.

But in policy iteration we also have to output a softmax vector related to each action. So I don't understand how this can be used to work with continuous action space.

Why are policy gradient methods preferred over value function approximation in continuous action domains?


texts:
Short question

I want to learn how to construct data science packages on top of core packages. Is there a list of excellent data science packages I can learn from?

Long question

I recently came across an excellent video where Joel Grus live codes a neural network library in Python. As an inexperienced data scientist without a software engineering background, this was the first time I saw the construction of a "complete" data science package from scratch.

My data analysis code up to this point has been very straightforward, only splitting into functions here and there, and never splitting into modules even where it may have been beneficial. I have since tried to find more resources to teach myself this skill. The Hitchhiker's Guide to Python has been useful.

I gather that it is important to learn by reading excellent packages. The Hitchhiker's guide does suggest some Python packages, but none that I could find were within data science (like Joel Grus's neural network library).

I am not asking for a list of the core data science packages of a language, e.g., numpy, scipy, sklearn, pandas, etc. Rather, a list of packages constructed on top of those.

Does such a list already exist? If not, can (should) we create it here?

EDIT: In hindsight, sklearn is possibly the perfect example of what I'm looking for, since its built on top of numpy and scipy.


texts: Summing three lexicon based approach methods for sentiment analysis?

texts:
I'm doing sentiment analysis using a lexicon based approach and I have a bunch of news headlines that needs to be categorized as negative, positive and neutral or within a scale ranging from -1 (very negative sentiment) to +1 (very positive sentiment).
I'm considering using three different methods: TextBlob, Vader (Valence Aware Dictionary and Sentiment Reasoner) and LIWC2015 as I have access to it. This would lead to three predicted sentiment of each headline.
I hove two question:

Which are the pros and cons of this technique?
And, once runned the 3 analisys, how can I melt all them into a single metrics? Is standardize/normalize the 3 metrics and averaging them a good solution?


texts: Correlation between nominal categorical variables

texts:
I have two arrays, whose values are nominal categorical variables. Each element represents a zone of a city: in the first vector we have the class each zone belongs to (so these might also be seen as ordinal, since values span from 0 to 3, with 3 being the upper class -let's say richest- and 0 the poorest, but I am not sure about this). The second vector is made of names: each item is the name of the candidate who won the Presidential elections in that particular zone.

I would like to calculate the correlation between the two vectors, to find whether there is some kind of relationship between the class of the zone and the winning candidate (i.e. candidate X systematically won in the poorest zones), but I am not sure on how to calculate correlation between nominal variables. I found this question somewhat helpful, but the example provided in the answer does not match with my case.


texts: One hot encoding of target variable containing classes 1 to 9 not including zero

texts:
While predicting a solution for a sudoku puzzle using CNN, the target variable should predict values from 1 to 9 for all the 81(9*9) values in the puzzle. Hence the target value shape is (81,9).
Using keras.to_categorical to convert target variable from (81,1) to (81,9) shows error.
to_categorical takes argument y which must contain integers from 0 to num_classes whereas 0 is not included in the target variable considered here. It works fine when num_classes is taken as 10 but results in (81*10) shaped variable. Is there any other way to get the target variable as (81,9) without altering the target variable values?


texts: Getting a balanced sample across many variables

texts:
Let’s say each element in my population has several attributes.
Let’s call then A, B, C, D, E, F.
Let’s say, for simplicity, each attribute has 10 values (but could be any number between 2 and 30).
Now I want to get a sample such that the distribution is the same across all features.
So for example if the whole population has about 15% of people in feature A with value 1, my sample should be the same.
What should be the way for me to select a size for the sample and choose a sample that has the desired properties?


texts: Why does regression model predict the same output

texts:
I have built a CNN for regression, but it is giving identical predictions (up to 8 sig. fig.) for almost 1/3 of the test data set. (The other outputs are different.) Is there a reason why this might be happening? Could this be a sign of over-fitting? I find this strange, as the nature of the prediction should be continuous. The input images are generated using another program, but I'm suspecting that the problem is broader than just a few problematic images.
Below is the code for the model (I'm using Keras).
model = Sequential([
    Conv2D(16, (7,7), activation='relu', input_shape=(1024,1024,1)),
    BatchNormalization(),
    MaxPooling2D((3,3), strides=(2,2)),

    Conv2D(32, (5,5), activation='relu'),
    BatchNormalization(),
    MaxPooling2D((3,3), strides=(2,2)),

    Conv2D(64, (3,3), activation='relu'),
    BatchNormalization(),
    MaxPooling2D((3,3), strides=(2,2)),

    Conv2D(128, (3,3), activation='relu'),
    BatchNormalization(),
    MaxPooling2D((2,2), strides=(2,2)),

    Conv2D(256, (3,3), activation='relu'),
    BatchNormalization(),
    MaxPooling2D((2,2), strides=(2,2)),

    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.5),
    Dense(128, activation='relu'),
    Dropout(0.5),
    Dense(128, activation='relu'),
    Dropout(0.5),
    Dense(1, activation='linear')
])

For training, I'm doing something like this
for i in range(6):
    model.fit_generator(train_batches, steps_per_epoch=TRAIN_STEP, validation_data=valid_batches, validation_steps=VALID_STEP, epochs=5, verbose=2)
    # some functions here to check up on the model
    model.save('model_epoch' + str((i+1)*5) + '.h5')

I have 267 images for training, 46 for validation, and 35 for testing.  train_batch contains the training images in batches, with image augmentation of horizontal_flip=True and vertical_flip=True. I'm using a batch size of 4 for all 3 sets of images. The steps value is calculated by dividing the number of images by the batch size (4 in this case)
Please let me know if I'm doing anything wrong. Thanks!


texts: How can I extract numerical information conditional on other information?

texts:
Given text data, I'd like to extract numerical information based on the presence of other information.

For instance, "There are 119 people in the gym right now. 100 people in the weight room, 19 on treadmills". If I was interested in "weight room" then I would extract "100" or if I was interested in "treadmills" then I would extract "19".

I can create dependency trees with Python's spaCy package. Is there a way to extract these dependencies?


texts: Looking for animals dataset for Deep Learning classification

texts:
Do you know any datasets that contain animals and their accurate classifications? I am looking for any dataset that categorizes animals.
For example:

a dataset with insects, with an image of an insect and its scientific name as label
a dataset with dogs of different breeds, their scientific names and images.
a dataset with Myriapodes (like centipedes or millipedes), images and their names as labels
mammals
birds
reptiles
...

Generally, I am looking for datasets of all animals worldwide, classified as good as possible.
I would very much appreciate it!
Here a list of possible classes (superclasses, groups, etc.).


texts: Machine learning methods for panel (longitudinal) data

texts:
I have a panel data set, for example:

obj  time        Y       x1      x2
1    1         0.1     1.28    0.02
2    1        0.11     1.27    0.01
1    2        -0.4     1.05    -0.06
2    2        -0.3     1.11    -0.02
1    3        -0.5     1.22    -0.06
2    3        1.2      1.06    0.11

I`m new at ML and until recently I did not know that this is a special (panel) data type. I predicted the value of a variable $Y(t+1)$ by values $x_1(t)$ and $x_2(t)$ (time lag) using linear regression model and MLP. But now I read some information about panel data analysis and realized that the methods I used were not suitable. At the moment I found that fixed/random effects models are suitable for panel data analysis. So, I have several questions:

What other methods are correctly used to analyze panel data (I`m interested in neural network models)? I read that these methods must take into account the dependency between particular object values and previous occurring values of this object (
what is in models with fixed and random effects).
I also tried to use MLP by feeding 2D data to it. I divided the panel data into $k=time$ $quants$ $count$  2D blocks and passed this data to the MLP input. For example above, $k=3$, ($input$ $layer$ $size= 4 =
number$ $of$ $predictors* block$ $objects$ $count$). In this case $batch$ $size=1$.
If I make the $batch$ $size=2$ and feed the neural network with 1D data ($input$ $layer$ $size=2$ for example above, will there be any difference? In both cases, the weights of the neural network will be rebuilt after the observations on all objects are transmitted in one quantum of time.


texts:
Is it ok to split a GridsearchCV?
At first, I would try estimators from 100-300 (100 steps) for a random forest regressor and some other parameters and after that, I would start the GridsearchCV with the same parameter and just change the estimators from 400 - 600.
Is there any aspect that would disagree with that logic?


texts: Training Keras model with multiple CSV files

texts:
I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem.
I could merge all of the data into a single CSV file and then read the CSV by the chunks of the data. This would simplify the process, however, I would not be able to shuffle the data each epoch.
Other approach that I can think of is to create a custom fit_generator. However I'm not sure how to actually implement this. Should I create a special generator for each dataset, and then cycle them? This would allow me to shuffle each dataset every epoch and even their order.
I believe that it would be more elegant to implement just a single generator, and manage all of the files inside. However to make this work, I would need to know the number of batches prior to training. That would require me to first get number of samples in each file, which would take a while. Also, I would need to sort out the issue of some lines being incorrect after reading the file with pandas. That limits me from using some fast os-level functions to get the number of lines.
Or is it possible to actually create a fit_generator without setting the number of batches prior to training? What approach would you recommend?


texts: Using machine learning to find the most similar image that contains another image

texts:
As the title states I want to use ml (maybe some kind of CNN autoencoder?) to find the most similar image (I have a list of 10k+ images) within another image. I am currently just using opencv with KAZE to get my current results here.
If that isn't clear then here's an example

Here's an example of images I will want to be searching to find something 'similar' (ex1, ex2, and ex3):

Here's the matching I do (I use KAZE)
from matplotlib import pyplot as plt
import numpy as np
import cv2
from typing import List
import os
import imutils

def calculate_matches(des1: List[cv2.KeyPoint], des2: List[cv2.KeyPoint]):
    """
    does a matching algorithm to match if keypoints 1 and 2 are similar
    @param des1: a numpy array of floats that are the descriptors of the keypoints
    @param des2: a numpy array of floats that are the descriptors of the keypoints
    @return:
    """
    # bf matcher with default params
    bf = cv2.BFMatcher(cv2.NORM_L2)
    matches = bf.knnMatch(des1, des2, k=2)
    topResults = []
    for m, n in matches:
        if m.distance < 0.7 * n.distance:
            topResults.append([m])

    return topResults

def compare_images_kaze():
    cwd = os.getcwd()
    target = os.path.join(cwd, 'opencv_target', 'target.png')
    images_list = os.listdir('opencv_images')
    for image in images_list:
        # get my 2 images
        img2 = cv2.imread(target)
        img1 = cv2.imread(os.path.join(cwd, 'opencv_images', image))
        for i in range(0, 360, int(360 / 5)):
            # rotate my image by i
            img_target_rotation = imutils.rotate_bound(img2, i)

            # Initiate KAZE object with default values
            kaze = cv2.KAZE_create()
            kp1, des1 = kaze.detectAndCompute(img1, None)
            kp2, des2 = kaze.detectAndCompute(img2, None)
            matches = calculate_matches(des1, des2)

            try:
                score = 100 * (len(matches) / min(len(kp1), len(kp2)))
            except ZeroDivisionError:
                score = 0
            print(image, score)
            img3 = cv2.drawMatchesKnn(img1, kp1, img_target_rotation, kp2, matches,
                                      None, flags=2)
            img3 = cv2.cvtColor(img3, cv2.COLOR_BGR2RGB)
            plt.imshow(img3)
            plt.show()
            plt.clf()

if __name__ == '__main__':
    compare_images_kaze()

Here's the result of my code:
ex1.png 21.052631578947366
ex2.png 0.0
ex3.png 42.10526315789473

It does alright! It was able to tell that ex1 is similar and ex2 is not similar, however it states that ex3 is similar (even more similar than ex1). I am hoping ml will result in better result instead of a local feature extraction algorithm. Any ml things that can be done to keep only ex1 as similar and not ex3?


texts: sklearn gridsearch lasso regression: find specific number of coefficients

texts:
I am using GridSearchCV and Lasso regression in order to fit a dataset composed out of Gaussians. I keep this example similar to this tutorial.
My goal is to find the best solution with a restricted number of non-zero coefficients, e.g. when I know beforehand, the data contains two Gaussians. So far, I used the grid search over the parameter space of number of features (or their spacing) and the width of the features, as well as the alpha parameter. Unfortunately, GridSearchCV does not return the coefficients for each fit, but only for the best one. What is the best way to find the fit which uses exactly two features?
Here my working example
import numpy as np
from matplotlib import pyplot as plt
from sklearn.linear_model import ElasticNet
from sklearn.pipeline import make_pipeline
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.model_selection import GridSearchCV

class GaussianFeatures(BaseEstimator, TransformerMixin):

    def __init__(self, N, width=2.0):
        self.N = N
        self.width = width

    @staticmethod
    def _gaussian_basis(x, y, width, axis=None):
        arg = (x - y) / width
        return np.exp(-0.5 * np.sum(arg ** 2, axis))

    def fit(self, X, y=None):
        self.centers_ = np.linspace(X.min(), X.max(), self.N)
        self.width_ = self.width
        return self

    def transform(self, X):
        return self._gaussian_basis(X[:, :, np.newaxis], self.centers_,
                                 self.width_, axis=1)

# create data
rng = np.random.RandomState(10)
x = 10 * rng.rand(50)
y = np.exp(-np.power((x - 2) / 1.5, 2.) / 2.) + \
    np.exp(-np.power((x - 8) / 1.5, 2.) / 2.) + \
    0.01 * rng.randn(50)

# initialize pipeline
widths = 2.0
number = 20
gauss_model = make_pipeline(GaussianFeatures(number, widths),
                            ElasticNet(alpha=0.005, l1_ratio=1, positive=True))

# initialize grid search
grid_width = np.arange(0.5, 5, 0.5)
grid_number = np.arange(1, 50, 1)
grid_alpha = [0.005, 0.004, 0.003]
grid_params = {
    'gaussianfeatures__N': grid_number,
    'gaussianfeatures__width': grid_width,
    'elasticnet__alpha': grid_alpha
}
gs = GridSearchCV(gauss_model, grid_params, scoring='neg_mean_absolute_error')
gs.fit(x[:, np.newaxis], y)

best_widths = gs.best_params_['gaussianfeatures__width']
best_number = gs.best_params_['gaussianfeatures__N']
best_alpha = gs.best_params_['elasticnet__alpha']
best_score = gs.best_score_
print('GridSearch best params: ', gs.best_params_)
print('best score = ', best_score)

# plot best prediction
gauss_model = make_pipeline(GaussianFeatures(best_number, best_widths),
                                  ElasticNet(alpha=best_alpha, l1_ratio=1, positive=True))
gauss_model.fit(x[:, np.newaxis], y)

xfit = np.linspace(0, 10, 1000)
yfit = gauss_model.predict(xfit[:, np.newaxis])

plt.figure()
plt.scatter(x, y)
plt.plot(xfit, yfit)
plt.show(block=False)

plt.figure()
plt.plot(gauss_model.steps[0][1].centers_, gauss_model.steps[1][1].coef_, '.', color='black')
plt.ylabel('theta')
plt.show(block=False)

output:
GridSearch best params:  {'elasticnet__alpha': 0.004, 'gaussianfeatures__N': 9, 'gaussianfeatures__width': 1.0}
best score =  -0.03473800683807379


texts: Does REPEATED K-fold cross validation make sense with Random Forest?

texts:
When using random forest, would using normal cross-validation and just taking the average results from multiple models with different random states give me the same results as using Repeated K-fold cross validation?
Repeated K-fold cross-validation basically repeats cross-validation with multiple different splits of the data and reports the average results.


texts:
I have the next code that I am trying to run in parallel:
inputs = range(10)

def processInput(i,x):
   return model(x)

results = Parallel(n_jobs=num_cores)(delayed(processInput)(i,x) for i in inputs)

Where model is a convolutional neural net, and x a batch of images.
It raises a gigantic error which ends with "PicklingError: Could not pickle the task to send it to the workers."
I just want to run the model in parallel N times with dropout to do some variance calculation later. Am I doing it wrong?


texts: How to quantitatively compare two or more complex data sets

texts:
I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response variable to the 100 or so variables.

My problem is that it is often difficult to compare two or more datasets and attribute the response variable to the 100 or so variables, due the large number of variables.

What do people do when faced with such a problem? What are some common analyses for doing this kind of work? It's preferable if the solution can be explained to people with a non-technical background.


texts: Situations where advanced theoretical knowledge of ML helped solve a real world problem?

texts:
I've invested lot of time trying to understand the theoretical aspects of Deep Learning and Neural Networks - but I'm now questioning whether it is worth it or not, given that I am someone who works mainly on the applied/business side of things.

By advanced theoretical knowledge, I mean things: Like understanding the details of how the Kernel trick works in SVM, the "no free lunch" theorem and its relationship to machine learning, the details of how neural networks are universal approximators, the VC dimension of a classifier, etc...

By applied Data Scientist, I mean someone who solves business problems using existing algorithms as opposed to someone who develops new ones.

So my question: Are there situations in an applied Data Scientist's life where such theoretical knowledge is useful? Or is this type of knowledge useful only for people who work on developing new algorithms?


texts: Dimension of the manifold on which my data sits

texts:
Suppose that I have data points, in the form of vectors with binary entries. We create a metric space, or Vietoris-Rips complex, using the Hamming distance between the data points.

I would like to imagine that my data points naturally sit on some manifold, which I want to somehow reconstruct (at least, its dimension).

What are the tools/techniques/references/keywords for such data analysis?

Thank you!


texts: Determine document novelty/similarity with the aid of Latent Dirichlet allocation (LDA) or Named Entities

texts:
Given an index or database with a lot of (short) documents (~ 1 million), I am trying to do some kind of novelty detection for each newly incoming document.

I know that I have to compute the similarity of the new document with each document in the index. If the similarity is below a certain threshold, one can consider this document as novel. One common approach - which I want to do - is to use a Vector Space Model and compute the cosine similarity (e.g. by using Apache Lucene).

But this approach has two shortcomings: 1) it is computationally expensive and 2) it does not incorporate the semantics of documents and words respectively.

In order to overcome these shortcomings, my idea was to either use an LDA topic distribution or named entities to augment the Lucene index and the query (i.e. the document collection and each new document) with semantics.

Now, I am completely lost regarding the concrete implementation. I have already trained an LDA topic model using Mallet and I am also able to do Named Entity Recognition on the corpus. But I do not know how to use these topics and named entities in order to realise novelty detection. More specifically, I do not know how to use these features for index and query creation.

For example, is it already sufficient to store all named entities of one document as a separate field in the index, add certain weights (i.e. boost them) and use a MultiFieldQuery? I do not think that this already adds some kind of semantics to the similarity detection. The same applies to LDA topics: is it sufficient to add the topic probability of each term as a Payload and implement a new similarity score?

I would be very happy if you could provide some hints or even code snippets on how to incorporate LDA topics or named entities in Lucene for some kind of novelty detection or semantic similarity measure.

Thank you in advance.


texts: Best practices to modelize top layers over CNN

texts:
I'm fine-tuning a InceptionResnetv2 network to get a features extractor, so I'm training a classical classifier with my data (one label/data, i'm using a softmax).

I would like to know how to choose architecture for top layers (fully connected), I read that usually Flatten -> Dense -> Dropout -> Softmax were used.

How to choose between

Flatten/MaxPool/AvgPool
Dense(256)/Dense(512)/Dense(1024)

is it purely empyric ?


texts: Multiprocessing or Parallel Computing Python Code

texts:
I am working on bioinformatics big datasets; training set and optimisation taking huge time to execute. I check and found that training and optimisation compiling on one core of cpu and because of that reason its taking huge time.

I  tried to get look for multiprocessing or parallel computing code but I didn't get any good relevant sample to code to execute on my script. I am new in multiprocessing, can anyone share the multiprocessing or parallel computing code in Python please.


texts: Learning character embeddings with GenSim

texts:
I am learning deep learning, and as a first exercise to myself I am trying to build a system that learns a very simple task - capitalize the first letter of each word.  As a first step, I am tried to create "character embeddings" - a vector for each character. I am using the following code:

import gensim
model = gensim.models.Word2Vec(sentences)

where sentences is a list of lists of chars which I took from this long Wikipedia page. For example, sentences[101] is:

[' ', ' ', ' ', ' ', 'S', 'p', 'e', 'a', 'k', 'i', 'n', 'g', ' ', 'a', 't', ' ', 't', 'h', 'e', ' ', 'c', 'o', 'n', 'c', 'l', 'u', 's', 'i', 'o', 'n', ' ', 'o', 'f', ' ', 'a', ' ', 'm', 'i', 's', 's', 'i', 'l', 'e', ' ', 'e', 'x', 'e', 'r', 'c', 'i', 's', 'e', ... ]

To test the model, I did:

model.most_similar(positive=['A', 'b'], negative=['a'], topn=3)

I hoped to get 'B' at the top, since 'A'-'a'+'b'='B', but I got:

[('D', 0.5388374328613281),
 ('N', 0.5219535827636719),
 ('V', 0.5081528425216675)]

(also, my capitalization application did not work so well, but this is probably because of the embeddings).

What should I do to get embeddings that identify capitalization?


texts: The notation of $splits(label)$ under Random Forest

texts:
On the "Fair Forests: Regularized Tree Induction to Minimize Model Bias", it is written that

We propose a simple regularization approach to constructing a fair decision tree induction algorithm. This is done by altering the way we measure the information gain $G(T, a)$, where $T$ is a set of training examples, and $a$ is the attribute to split on. We will denote the set of points in each of the $k$ branchs of the tree as $T_{i \ldots k}$. This normally is combined with an impurity measure $I(T)$, to give us
$$
G(T, a)=I(T)-\sum_{\forall T_{i} \in \text { splits }(a)} \frac{\left|T_{i}\right|}{|T|} \cdot I\left(T_{i}\right)
$$
The information gain scores the quality of a splitting attribute $a$ by how much it reduces impurity compared to the current impurity. The larger the gain, the more pure the class labels have become, and thus, should improve classification performance. In the CART algorithm, the Gini impurity is normally used for categorical targets.
$$
I_{\mathrm{Gini}}(T)=1-\sum_{\forall T_{i} \in \text { splits }(\text { label })}\left(\frac{\left|T_{i}\right|}{|T|}\right)^{2}
$$

Especially, those two things make me confused very much.

The definiton of $\mathrm{splits(a)}$ is not written in the paper.
It is written that $T_i$ is the branch, however, for me, it seems a node.

I have already learned about Random Forest on the ESL, so I know the normal definition of impurity as:
$$
I_G(p) = 1 - \sum_{i=1}^{k} p_i^2
$$
However, I can't figure out each definition in this paper...


texts: Multiple search/replace on pandas series

texts:
I have a pandas dataframe with school names as one of the columns. However, there is quite a bit of misspelling in the school names, for example:

'Abernethy Elem School', 'Abernethy Elementary School', 'BOISE/ELIOT ELEM SCHOOL', 'Boise/Eliot Elementary School', 'Boise-Eliot Elementary School' ...

I am trying to clean up the names by doing this:

school_perf_report["SCHLNM"] = school_perf_report["SCHLNM"].str.lower().str.replace(r"elementary","elem").str.replace(r"/"," ").str.replace("-"," ").str.replace(r"\s+"," ")

Is there some cleaner way to perform the same operation? Basically

Change "elementary" to "elem"
Remove / or - and replace them with space.
Remove multiple spaces
Lower case everything

Thanks


texts: How to get rid off differences between prediction and actuals while predicting or forecasting time series data

texts:
I have performed Boxcox transformation on my time series data and processed it through ARIMA modeling. Converted prediction values to the actual. I see significant differences between actual and predictions(In sample), would like to get optimal prediction values which are close to actual values.

Have tried Machine learning models such as Linear Regression, tree-based(Random Forest, XGboost) as well. Before processing data through these models, it(sales) was aggregated by day wise and extracted, sale day, month, sale dow..etc.

For the same data, lagged features and rolling statistics were also generated, trained and tested. However, did not get optimal sale predictions and found differences between actual sale values and predictions.

I have ensure no white noise in my data and made stationary before processing data thru models.

Can someone suggest techniques such as feature engineering or any other modeling to overcome the problem of differences between actual and predictions and future forecasting?

Sample data

Actual   Predcition
72      310.255
2148    1246.419
208     497.629
531     257.541
80      125.156
480     464.411
229     365.513
60      296.102
192     279.578
90      278.131
72      180.8
70      48.544
16      341.464
32      225.55
1560    253.01
960     708.296
1528    498.59


texts: CV Pipeline for Time Series with Differing shape for X and y

texts:
I am playing around with ML models to forecast a time series. I'd like to generate a sklearn pipeline like

from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler

pipe = Pipeline(
    steps=[
        ("gen", FunctionTransformer(generate_features)),
        ("imp_nan", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
        ("reg", keras.wrappers.scikit_learn.KerasRegressor(build_model, epochs=35))
    ]
)

Where generate_features takes in my data X and produces a collection of features for a given timestamp. For example a 5 day rolling standard deviation. However, my target y could have a different shape.

For example, I may want to forecast 5 hours in the future (and so my target will have a date time index with period 5 hours), but my input data could have a frequency of 1 min (and so initially an index with period 1 min that gets transformed into a period of 5 hours). This leads to issues when trying to cross validate. For example, if I use

splitter = TimeSeriesSplit(n_splits=3)
grid = GridSearchCV(pipe, {"gen__kw_args": [{"windows": (2,4,6)},]}, cv=splitter)
grid.fit(X_train, y_train)

I immediately get an error telling me the shape of X_train and y_train are not the same. This makes sense because the splitter needs to be used on both X and y.

I want to do this because I'd like to perform a grid cv on the parameters of my generate_features function (as well as the parameters of my model...)

My Question

Is there a nice way to create a pipeline and perform cross validation on time series feature generation when our input shape of X could be different to our input shape of y?

Edit: Here is the traceback

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-f0c6e65ecbb8> in <module>
      1 splitter = TimeSeriesSplit(n_splits=3)
      2 grid = GridSearchCV(pipe, {"gen__kw_args": [{"windows": (2,4,6)},]}, cv=splitter)
----> 3 grid.fit(X_train, y_train)

~/.local/lib/python3.8/site-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
    648             refit_metric = 'score'
    649
--> 650         X, y, groups = indexable(X, y, groups)
    651         fit_params = _check_fit_params(X, fit_params)
    652

~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py in indexable(*iterables)
    246     """
    247     result = [_make_indexable(X) for X in iterables]
--> 248     check_consistent_length(*result)
    249     return result
    250

~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py in check_consistent_length(*arrays)
    209     uniques = np.unique(lengths)
    210     if len(uniques) > 1:
--> 211         raise ValueError("Found input variables with inconsistent numbers of"
    212                          " samples: %r" % [int(l) for l in lengths])
    213

ValueError: Found input variables with inconsistent numbers of samples: [522323, 52417]


texts: Advice on what Machine Learning Algorithms to study for a Job to candidate matching algorithm

texts:
I have asked in a few places and this seems to get downvoted for some reason. If this is not the place to ask this then some advice on how and where to ask it would be appreciated.

I'm creating a web platform 2 sided market that matches available consultants to short term contracts.

We currently have a rules-based algorithm that matches based on a small number of features like industry, job title, rate, and availability. For that list of consultants, It then calculates a percentage match based on other features like skill set, qualifications, certifications, years of service, etc.

We have data on candidates that were accepted and candidates that were rejected.

I've spent a year revising maths, statistics, and studying basic machine learning with python. But I am no closer to understanding how machine learning could be used to match consultants to assignments.

I assume what I need to look into is feature engineering and some sort of classification algorithm. It would seem that a contract assignment would have an ideal candidate and so we are matching all other candidates to the features of the ideal. Is it some sort of unsupervised learning to classify the candidates into groups and putting forward those candidates that fall in the same group?

Does anyone have an idea on how best this would best be done?

I am looking at what to study to get closer to a solution so any advice to so that I can get to a point of understanding how this could best be done would be appreciated.


texts: Neural Network for detecting/checking for requirements in diagrams

texts:
My question is more about what approach is a good/the best approach for my problem:

THE PROBLEM -

I'm an (mechanical/software) engineer and we take extensive amount of time to review technical drawings prior to them being complete/ready/meeting requirements. This quality check process consists of a loop between our CAD drafter, drafting checker, engineer, and final engineering check.

We are always needing to check drawings for consistent format and information that is contained within it regardless of type of drawing (mechanical, electrical, piping, property sketches, etc).

I have spent quite a bit of the last few years dealing with relatively straight forward machine learning techniques (decision trees/supervised learning methods), and this seemed like a good opportunity to have a go at something more complicated. I also wrote a (un-optimised) neural network in Go(lang) a few years ago, so I'm not totally new to the area.

A Solution:

What approach would you suggest for writing an algorithm that could solve this type of issue?

My current plan is to:

Collect as many drawings in various stages of completeness of lots of different parts/diagrams/networks etc.
Split them into groups classifying them as "complete" and "not complete"
I was going to start with a decision tree/random forest because it seems like a classification problem with probably and finite number of classificaitons
However first I would need to convert the diagram (pdf) into a pixel array, as standard image recognition wouldn't work

This is the reason I think this is actually better for a neural network, as parsing the raw data is easiest to input to a neural network (if all diagrams are the same dimensions)

So I could feed the diagram into a neural network and give each the output of what is wrong with it, or if its correct.
What's important is two diagrams of completely different objects could both fail for the same reason (e.g missing a dimension label). I would a) like to know what the problem is, and b) if possible, be able to output exactly why it failed.

Could anyone advise?
Thanks


texts: how to draw a negative exponential plot of this situation with python

texts:
I have a dataset with binary output ($Y$) and I have a column (Duration) contains the duration of each task that is stored by "days" and varied from 1day to 350 days.

when I think logically in our situation, I can deduce that the probability of getting a positive output value ($Y = 1$) require to have small duration task.

But I need to justify my opinion with some plots

I have tried the following source code but It doesn't represent correctly my assumption.

#LoadData

min_duration = plot_data['Duration'].min()
max_duration = plot_data['Duration'].max()
xr_ = list(range(min_duration,  max_duration,  5))
y_ = []

for i in range(0,(len(xr_)-1)):
    a_ = np.logical_and(plot_data['Duration'].values >= xr_[i], plot_data['Duration'].values < xr_[i+1])
    b_ = np.logical_and(np.logical_and(plot_data['Duration'].values >= xr_[i], plot_data['Duration'].values < xr_[i+1]), plot_data['output'].values==1)
    y_.append(sum(b_)/sum(a_))

import matplotlib
matplotlib.pyplot.plot(xr_[1:len(xr_)], y_, 'o')

Based on my previous assumption I must get a plot which contains an exponential form like :

But I have got contrary the following plot:

I want to know where I have a mistake and If there is any other method to justify my assumption


texts: What are the motion planning benchmarks?

texts:
Suppose I wanted to try and improve on existing motion planning algorithms. What benchmarks should I be trying to beat?
Papers with code site has no motion planning benchmarks. I searched online and found the Planner Arena
Is this it or did I miss something? Are there widely recognized benchmarks in the field?


texts: Which is the best way to install Moses for SMT?

texts:
I have tried many ways but it always ended up in error. My Configuration: Ubuntu 16.04, Python 3.6.10 The error that I am unable to solve is: undefined reference to

boost::unit_test::unit_test_main(bool (*)(), int, char**)

I was initially following this guide here but had to tweak the commands a lot. Still, unable to settle things off.


texts: Are word embeddings further updated during training for document classification?

texts:
I am relatively new to the area of using word embeddings in NLP tasks. From a large corpus of documents, I train word2vec word embedding vectors and afterwards I am going to use these for document classification, combined with RNN based classifiers (LSTM, GRU), which is a pretty standart pipeline nowadays.

There is one issue; that should we update the word embeddings during the document classification training as well.

I am used to tasks like image classification/object detection in the past. You get an image input and the convolution features extracted from that image are updated during the numerical optimization of a CNN. But the image itself is never updated, naturally, since it is the original data.

How do we treat the embedding vectors in the world of text documents? They are not "natural" exactly like the images, we learn them from an unsupervised method first (word2vec, GloVe or any other tool) so I think they can be further fine-tuned during the supervised training. Is it the common practice to update the embedding vectors as well as the RNN parameters during the training of the sequence classifier or should we leave them as constant (in order to avoid overfitting) ?


texts: How to interpret predicted data from a keras model


texts: How can we predict Geo Tagged Pollution Parameters using a Machine Learning Model with an Android Device?

texts:
Newly installed Pollution Control Sensory Platforms are dumping Continous/Discrete Environment Pollution Parametric Data from various locations.

Whats the best format structured or Unstructured of data storing efficient from day one and compatible to ML ?
What should be the strategy : Supervised or Reinforced or UnSupervised form of ML  ?

End objective is to predict the pollution levels in a specific route and recommend/alert the time for Bicycling , Walking and Running for sports enthusiasts .


texts: Converting a string to a recommendation type string

texts:
I am trying to build a recommendation system and some of the labels are
'printer has been replaced',
'scanner replaced',
'updated software to windows 10',
'password has been changed',
'monitor replaced',
'...'

What I want to do is convert these strings into more action item/ recommendation strings. Something like this
'I would recommend replacing the printer',
'I would recommend replacing the scanner',
'A software update to windows 10 is needed',
'I would recommend changing the password',
'I would recommend replacing the monitor'
'...'

Is there a library out there that does this? I have over a 1000 different resolution strings so the solution would need to be scalable.


texts: plateaus and local minima in deep learning

texts:
Why training deep learning models is more likely to suffer from plateaus than local minima?

Could anyone illustrate it.


texts:
May the training set and validation set overlap?

Similarly, may the testing set and validation set overlap?


texts: hypeparameters tuning neural network according to loss vs according to scoring function

texts:
During hyperparameters tuning we select a metric to measure performance of the model. Example of metrics : f1 score, precision, recall, AUC ...

In general, for the training of neural networks, back-propagation tries to optimize the weights of the model according to the value of the loss function.

Here comes the question:
Why don't people use the loss function as a main performance metric for neural networks optimization?


texts: How to use cluster analysis with grouped data so one cluster may only have not more than one item from each group?

texts:
I need to group items by their approximity to each other in multi (<5) dimensional space. The items also have a categorical feature. I need to form groups (clusters) such that none of the records from the same category would appear in thr same cluster.  Is there a class of clustering algorithms that can do that?

My way of thinking is to use custom distances that would measure two records in the same category further then ones in different categories. That works to some extent, but it does not guarantee satisfaction of the given requirement.

A simple one dimension (x) + category (c) example:

    x       c
0   0.80    0
1   0.90    1
2   0.10    0
3   0.30    1
4   0.20    0

The goal is to group records into two clusters [0, 1];and  [2 or 4, 3]; then record 4 or 2 respectively should remain outside of the second cluster because a record with c=0 is already present in the cluster.

Any suggestions?


texts:
I'm new to data science and stats, so this might seems like a beginner question.

I'm working on a dataset where I've user's Twitter followers gain per day. I want to measure the average growth he had over a period of time, which I did by finding the mean of growth. But someone is suggesting me to use median for this.

Can anyone explains, in which use-case we should use mean and when to use median?


texts: Error using image: Error when checking input: expected dense_56_input to have 2 dimensions, but got array with shape

texts:
I am trying to train a Sequential model using simple flow_from_directory() but i am getting this error , I have tried using lesser layers but the error dose not go away.

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
train_directory,
target_size = (28,28),
class_mode = 'categorical')

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Test'
validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
validation_directory,
target_size = (28,28),
class_mode = 'categorical'
)

model = Sequential()

model.add(Dense(128, input_shape = (784,)))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',metrics = ['accuracy'])

model.fit_generator(training_generator,epochs=10)

Here is the error:

File "C:\Users\Ankit\.spyder-py3\temp.py", line 31, in <module>
    model.fit_generator(training_generator,epochs=10)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training.py", line 1732, in fit_generator
    initial_epoch=initial_epoch)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training_generator.py", line 220, in fit_generator
    reset_metrics=False)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training.py", line 1508, in train_on_batch
    class_weight=class_weight)

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training.py", line 579, in _standardize_user_data
    exception_prefix='input')

  File "C:\Users\Ankit\anaconda3\lib\site-packages\keras\engine\training_utils.py", line 135, in standardize_input_data
    'with shape ' + str(data_shape))

  ValueError: Error when checking input: expected dense_56_input to have 2 dimensions, but got array with shape (32, 28, 28, 3)


texts: Machine learning algorithm for Low dimension input to high dimension output

texts: I am plaining on training a network for body generation, i.e. given some specific measurement,(5 features)  the output will be the a set of vertices representing the obj of the bodies. I am wondering what kind of machine learning algorithm is best suited for this scenario.


texts:
I am working with data from the government regarding COVID immunization numbers and would like to convert object columns into numeric. I am using Python in Jupyter Notebook. I tried using to_numeric, but the dtype is still an object. Any suggestions?
Thanks in advance :)


texts: Predict status of upcoming project milestones with intermediate activities

texts:
I have data of 100+ project data. Each project has about 175 sequential activities from start to end. There are approximately 7 key milestones between those 175 activities that we want to predict. Data is completely categorical (means every activity status is R, A, G, B, GR.) So we want to predict the status of those 7 milestones (R,A,G), say after every 25 activities.
Projects are civil work projects where sequential activities are reqt gathering, review, approvals, high level design, review, low level design, risk identified, build, deliver etc. Milestones are End of Reqts, End of Design, End of Build and End of Deploy etc. Have past data of 500 such projects to train for 24 months. Since we are new to Machine learning, we tried random forest for each milestone. But, that requires all activities data ready. To predict milestone in advance, this seems to be incorrect model.
Proj Start--activity1,2,3....24,Milestone1 (Predict this), 26,27,28...49,MS2 (Predict),51,52,.....74,MS3(Predict)....MS4...MS6, 151...174 MS7(Predict)--ProjEnd
Also, we should be in position to predict at least one next milestone (e.g. MS3) status based on current milestone status (e.g. MS2) and take action accordingly for activities (e.g. 51 to 74) in between. Please suggest how should this problem be approached?


texts: Clustering a very large number of very small clusters with most data unrelated

texts:
I'm trying to detect duplicates in a data set of about 34k distinct items. When I say "duplicate," I don't mean identical items, just very similar. I have an algorithm that will Cartesian join the items and return a sparse matrix of similar items with a similarity from 0.0 to 1.0. This matrix has about 5k non-zero entries in it, so it's very sparse.

I already know what the similar items are by pairs. What I need is a good way to cluster them together. I expect there to be many clusters with only 2 items in them, and a couple thousand with more than 2 items, with the vast majority of items being unclustered.

Is there a clustering algorithm that fits this scenario well? I've tried several, but they either give poor results, or are designed with few large clusters in mind instead of many small clusters, and so tend to crash.

Is clustering the wrong approach?

I'm using Spark, but the sparse matrix is small enough that I don't mind exporting to something else if necessary.


texts:
I have following problem: I would like to feed LSTM with

  train_datagen.flow_from_directory

The input is basically a spectrogram images converted from time-series into time-frequency-domain in PNG format that has a dimension of: timestep x frequency spectrum. 1 sample = 1 PNG image in uint8. In my example: 3601 timesteps with 217 frequency spectrum (=features) / timestep.

The spectrogram itself is just 1D, but I think "flow from directory" function was hard-coded to only prepare 3D image matrix and thus the input shape was becoming , which is totally pity because there are some people who are only working with purely greyscale uint8 image, and some who are working with multispectral and hyperspectral images.

My codes are following:

import keras
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.layers import LSTM
from keras import optimizers
from keras import backend as K
import tensorflow as tf

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 10
nb_train_samples = num_classes*70
nb_validation_samples = num_classes*20
epochs = 20
batch_size = 10
input_shape = (img_width, img_height)

model.add(LSTM(units=256, input_shape= input_shape, return_sequences=True))
model.add(LSTM(units=128, return_sequences=True))
model.add(LSTM(units=64))
model.add(Dense(128))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size)

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size)

model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    callbacks=[plot_losses],
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

And then as soon as I run the program, of course it gives an error message::

**ValueError: Error when checking input: expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)**

The message:

  expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)

clearly suggests it does not agree with my definition of input shape of: (3601, 217)

Any idea to easy fix the problem?
Thanks in advance.


texts: Why do I not get 100% Accuracy with KNN with $K=1$

texts:
I am playing with KNN on the Iris Dataset

I expected to get 100% accuracy with $K=1$ since every point should predict itself based on the Voronoi volume around it created by the KNN algorithm.

However using Scikit Learn I do not find this result. Here is my code.

import pandas as pd
import numpy as np

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import confusion_matrix
from sklearn.datasets import load_iris

iris = load_iris()

X = pd.DataFrame(iris['data'])

X.columns = ['sepalLength','sepalWidth','petalLength','petalWidth']

featureX = 'sepalLength'
featureY = 'sepalWidth'

X_2 = X[[featureX,featureY]]

y = iris['target']

y = (np.array(y)==2).astype(np.int)

knn = KNeighborsClassifier(n_neighbors=1, p=2)
knn.fit(X_2, y)
y_pred = knn.predict(X_2)
cm = confusion_matrix(y, y_pred)

print(cm)

I get the following output for the confusion matrix. I see 11 out of 150 samples are incorrectly classified.

[[95  5]

 [ 6 44]]

Why is it not perfect ?


texts: How to sort numbers using Convolutional Neural Network?

texts:
Recently, in an interview I got this question:

Design a convnet that sorts numbers. Operators are ReLU, Conv, and Pooling. E.g. input: 5, 3, 6, 2; output: 2, 3, 5, 6

I am not sure how can you sort a list of numbers using CNN. I know it is possible using RNN. Is it even possible?


texts: inconsistency occurred when using cross_val_score in python

texts:
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.svm import SVC

#import data
iris = load_iris()
X = pd.DataFrame(iris.data, columns=iris.feature_names)
y = iris.target

#split data
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = 0.2, shuffle=False)

#modeling using SVM
model = SVC(kernel='linear', C = 1)
model.fit(X_train, y_train)
model.score(X_test, y_test)

# here I get score: 0.8666666

# Now I use cross_validation with cv = 5,

from sklearn.model_selection import cross_val_score
model = SVC(kernel='linear', C = 1)
scores = cross_val_score(model, X, y, cv = 5, scoring = "accuracy")
scores

#here I got array([0.96666667, 1.        , 0.96666667, 0.96666667, 1.        ])

None of the numbers in the above arrays equal to 0.866666, I wonder why the inconsistency happens(since "cv = 5" matches the condition "test_size = 0.2").


texts: Dealing with low-information centroids using Nearest Centroid Classifier and bag of words method

texts:
I am currently working on a problem where we have projects and e-mails that belong to a single project each.

My goal is to create a recommendation system for incoming e-mails which presents the projects the e-mail might belong to.

The number of projects is constantly growing, just like the number of e-mails. This is why I decided to use the Nearest Centroid Classifier because the "training" of new classes is easy (after all, just calculating a mean over the e-mails belonging to a centroid) and it seemed to be promising to me.

I use NCC in combination with the bag of words method and for that I am calculating the scores of the words via TF-IDF.

The data pool is not the greatest actually, which is another reason I tried using a less complex model like NCC. I only have 5000 useful e-mails and around 300 projects.

The problem is though that when I calculate the distances to every project centroids some centroids win in every case. For nearly every e-mail the first 10 centroids of the recommendation are the same, all the time. When I had a look at them I noticed that the "best" centroids are simply centroids which do not hold much information, they seem to have very few text data. And if the e-mail does not have much text obviously the error is low and thus the distance is low.

Is there any way to deal with that problem? Or is TF-IDF and NCC not a good combination?


texts: Fisher Scoring v/s Coordinate Descent for MLE in R

texts:
R base function glm() uses Fishers Scoring for MLE, while the glmnet appears to use the coordinate descent method to solve the same equation. Coordinate descent is more time-efficient than Fisher Scoring, as Fisher Scoring calculates the second order derivative matrix, in addition to some other matrix operations. which makes expensive to perform, while coordinate descent can do the same task in O(np) time.

Why would R base function use Fisher Scoring? Does this method have an advantage over other optimization methods? How does coordinate descent and Fisher Scoring compare? I am relatively new to do this field so any help or resource will be helpful.


texts: Which process step in KDD or CRISP-DM includes labeling of the data?

texts:
KDD and CRISP-DM are both processes to structure your Data Mining procedure. Is data labeling not also a important part of Data Mining?

Data labeling is for example in unsupervised learning the target of the Data Mining process. So if I want to classify a data set that was labelled by me before, do I just do the process twice? In my opinion sometimes the labeling is quite trivial, so that doing the process twice would be quite unnecessary?

Is it possible to include the labeling into the data exploring or preprocessing phase? E.g. in CRISP-DM Preprocessing there is something like generating a new parameter. Can this Parameter be also a new target/label?

I know this question is quite process orientated and in Data Mining you are quite free but just assume that in this case you have to follow the process.


texts: Feature Normalization/Scaling: Prediction Step

texts:
I'm just doing a simple linear regression with gradient descent in the multivariate case. Feature normalization/scaling is a standard pre-processing step in this situation, so I take my original feature matrix $X$, organized with features in columns and samples in rows, and transform to $\tilde{X}$, where, on a column-by-column basis,
$$\tilde{X}=\frac{X-\bar{X}}{s_{X}}.$$
Here, $\bar{X}$ is the mean of a column, and $s_{X}$ is the sample standard deviation of a column. Once I've done this, I prepend a column of $1$'s to allow for a constant offset in the $\theta$ vector. So far, so good.

If I did not do feature normalization, then my prediction, once I found my $\theta$ vector, would simply be $x\cdot\theta$, where $x$ is the location at which I want to predict the outcome. But now, if I am doing feature normalization, what does the prediction look like? I suppose I could take my location $x$ and transform it according to the above equation on an element-by-element basis. But then what? The outcome of $\tilde{x}\cdot\theta$ would not be in my desired engineering units. Moreover, how do I know that the $\theta$ vector I've generated via gradient descent is correct for the un-transformed locations? I realize all of this is a moot point if I'm using the normal equation, since feature scaling is unnecessary in that case. However, as gradient descent typically works better for very large feature sets ($> 10k$ features), this would seem to be an important step. Thank you for your time!


texts: What could be a right datascience approach of doing fuzzy string matching in large amount of short text data?

texts:
I have two lists to compare. List 1 contains strings which represent the strings in List2. Sometimes, they are direct matches, some times, one word match, sometimes there are acronyms etc. An example is as below:

   List1           |  List2
  ------------------------
   great hall town | Great Hall
   grt_town        | Great Hall
   gHall           | Great Hall
   great town      | Great Hall
   mind great town | Great Hall

There are two main problems:

List2 is very large ( millions of records, all short in length)
List1 records are very unrelated sometimes. There are longer prefixes, suffixes which are not in List2. Some words in List2 are reduced to a letter.

I vaguely have some thoughts around clustering List2 and trying to find distance between centroids and the elements from List1. Do you think this as a starting point?
Thanks


texts: What Kernel is suitable for the following data for SVM classification?

texts:
I have the following 2 class data, as shown below. .

Its a hand crafted example using two ellipse equations. I want to know what might be a recommended kernel to be used with this problem if I want to use an SVM based binary classifier.


texts: What algorithgm Android's ActivityRecognitionAPI uses?

texts: I am curious to find out what is the algorithm behind Google's ActivityRecognitionAPI on the Android platform to find out what is the current state of the user (e.g., i the user us walking, running or laying) and how it works in a nutshell? Thank you in advance!


texts: How to calculate probability of non independence using bayes theorem?

texts:
i looked into one of the post about  naive bayes calulation of naive part

Predit the class label for instance (A=1,B=2,C=2) using naive Bayes classifcation.

Let C1 be class 1 and C2 be class 2.

For C1, by the assumption of Naive Bayesian Classifier, we have
P(A=1,B=2,C=2∣C1)=P(A=1∣C1)⋅P(B=2∣C1)⋅P(C=2∣C1)

Take P(A=1∣C1) as an example. There are 4 training records of C1, among which there are 2 records with A=1. Therefore, P(A=1∣C1)=24. Similarly, you can calculate P(B=2∣C1) and P(C=2∣C1).

It is similar to calculate P(A=1,B=2,C=2∣C2).

My query is how can we calculate the NON Naive part of this ?

P(A=1,B=2,C=2∣C1)= P(A=1∣C1)⋅P(B=2∣C1)⋅P(C=2∣C1)  -> Here the events are considered to be independent of each other

What if the events  are dependent ? How should we calcualte in that case ?


texts: Implementing Gaussian Mixture Model from scratch on an Image dataset

texts:
I recently learned about GMM. From what I understand it basically gives a probability that a given data point belongs to a specific cluster. I need to implement GMM on the STL10 training dataset with 5000 images with 96x96x3. From what I understand, my dataset will have 5000 samples with 27648 features. Since the STL10 has 10 classes, I set the GMM to have 10 clusters. Using PCA I have reduced the number of features to 1167 retaining 95% of the variance and then applied GMM.
Using sci-kit I could do it like the following:
from sklearn.mixutre import GaussianMixture
from sklearn.decomposition import PCA

# PCA
var_to_retain = 0.95
pca = PCA(var_to_retain)
principalComponents = pca.fit_transform(train_data)

# Reshape
X = principalComponents.reshape((-1,96,96,3))

# GMM define the model and test using 10 clusters
n = 10
gmm_model = GMM(n_components=n, covariance_type='full')
gmm = gmm_model.fit(X)

And further using gmm.predict(X) gives me the cluster labels (which is not necessarily the labels for each data).
However I need to implement the EM algorithm manually and from researching, most sources I found directly use a 2D gaussian generated dataset with a set of mean centroids and covariance matrix. My initial attempt at this was simply getting the mean average for all the samples in each class. However I have no idea on getting the other parameters. My goal is to show a visualization of the data similar to this or this.  From my understanding I need to somehow initalize the means, covariances, weights using my dataset. Since those are what is needed in the estimation step. How do I format my dataset to be used as input in the EM algorithm?


texts: How do you determine the architecture of a model(keras)?

texts:
I have solved quite a few kaggle playground problems lately, but I can't understand how to come up with good enough model architecture which gets 0.9+ validation accuracy and without overfitting.

Is there some formula, or is it some hit and trial method for determining filters and neurons.
And also I'm always using relu activation in hidden layers(read it somewhere) when will other activations be used in hidden ones.

Can you lay down some guidelines that I should follow?. Considering I mostly work with image classification problems.

And I you have some other tips as well you can share in your answer.


texts: Learn (common) grammar / pattern from set of sample strings?

texts:
So I currently have a text pattern detection challenge to solve at work. I am trying to make an outlier detection algorithm for a database, for string columns.
For example let's say I have the following list of strings:
["abc123", "jkj577", "lkj123", "uio324", "123123"]

I want to develop an algorithm that would detect common patterns in the list of strings, and the indicate which strings are not in this format. For example, in the example above, I would like this algorithm to detect the following regular expression:
r"[a-z]{3}\d{3}"

given that the majority of the entries in the list obey this pattern, except the last one, which should be marked as an outlier.
The first idea that come to my mind was to use a genetic algorithm to find the regular expression pattern, where the fitness function is the number of entries on the list that match the pattern. I haven't worked out the details (crossvers function, etc..), and there is already the difficulty in the sense that the pattern ".*" will match everything, hence will always maximize the fitness function.
Anybody already worked on a similar problem? What are my options here? Thank you!


texts:
At http://www.speech.zone/exercises/dtw-in-python/ it says

  Although it's not really used anymore, Dynamic Time Warping (DTW) is a
  nice introduction to the key concept of Dynamic Programming.

I am using DTW for signal processing and are a little surprised: What is used instead?


texts: How to structure time series for binary classification in Keras?

texts:
I'm trying to classify churn (1 or 0) for a user based on day-to-day time series and activity levels, something like this:

User     Date      Activity Churn
1    09-01-2018       35      0
1    09-02-2018       17      0
2    09-01-2018        0      1
2    09-02-2018       13      1
etc...

Could someone give a few pointers on how this would be setup in Keras to predict the churn value?


texts: Policy Gradient Methods - ScoreFunction & Log(policy)

texts:
In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy

Question:

If we have a Neural Network that holds parameters defining our policy,
we will have to perform backpropagation to each weight (each parameter). Why would we want to avoid dealing with:

$\nabla_{\theta}\pi_{\theta}(s,a)$

and instead would wish to deal with its re-arranged version:

$\pi_{\theta}(s,a)\nabla_{\theta} log \big( \pi_{\theta}(s,a) \big) $

Don't we still have a $\pi_{\theta}$ that sits inside the log? By the Chain Rule, we will still have to compute $\frac{\partial \pi_{\theta}}{\partial \theta}$ once we arrive to it. Aren't we just postponing inevitable?

My Understanding:

I know that in Neural Networks that perform classification, using a Cross-Entropy function (that deals with logs) right after a Softmax Activation would allow for a really neat derivative during backprop: (result - target) Perhaps that's the intent here as well?

Thus, since we are dealing with the Log(policy), am I implied to always use a Softmax function for the last layer of my Neural Net, when working with Policy Gradients? - at 39:37 in the video, David also describes Gaussian Policy. Are these two selected because they work really well with Log, during backprop?

As I understand, in general, the benefits of wrapping a product in a $log()$  is being able to split it into 2 summands:  $log(ab) = log(a) + log(b)$ but I can't see where it would be useful in Policy Gradients.

My intuition is this might be useful to prevent vanishing gradients.

Maybe it's related to the fact that it's called "Score Function", therefore we are dealing with a Log for some other reason?  In supervised learning we have a Squared Error cost function that provides us a "beginning value", during the backpropagation.  Is Score function = cost function?


texts: Determine what data, does certain column have in common

texts:
I have an Excel sheet with n columns, these columns contain info about the students. For admissions we have the score of test scores in multiple subject areas, scores from an interview, and scores of a written test and comprehension test. There is a column which contains student's academic level (High, M.High, Middle, M.Low, Low). I want to compare the last column with the others variables and see whether there are common features that passing students have in common.

Is there software for this? If this can be done with excel, how can I do it? Does SPSS provide this kind of analysis?


texts: Flipping the labels in a classification problem

texts:
Let us say A- we have a binary classifier with labels 1 as healthy and 0 as sick. The precision we got is 100% and the recall is 70%.
Now let us say B-we flip the labels where 0 is healthy and 1 as sick.
Are Precision and recall  get flipped in their values if you flip the labels? So in the new case, recall is 100% and precision is 70%?
Or the (70,100) values in the first case belonged to class 1 aka, healthy people and I should calculate the precision and recall for class 0?
My understanding was that  we were able to recall 70% of sick people and out of every people who we detected as sick, no person was healthy (precision as 100%).  But I am confused now


texts: How can I use idh and random when I use hupossion in mcmcglmm?

texts:
Here is my problem: I need to use hupossion in MCMCglmm package. Here is my prior:

> prior = list(R = list(V = diag(2), nu = 0.002),
 +                G = list(G1 = list(V = diag(10), nu = 0.002),
 +                           G2 = list(V = 1, nu = 0.002)))

and here is my training:

> myTrain <- MCMCglmm(pipenumyearnosameday ~ trait - 1 +
+                         at.level(trait,1):lastyearnumnosameday +
+                         at.level(trait,1):pipeInsideSizemm,
+                     random = ~idh(pipematerialgroup1):installyeargroup1,
+                     data = train,
+                     prior = prior,
+                     verbose = F,
+                     pr=T,
+                     family = "hupoisson",
+                     rcov = ~idh(trait):units)

There are 5 levels in the attribute pipematerialroup1. I know sth must be wrong with the prior. I got the following information:

Error in MCMCglmm(pipenumyearnosameday ~ trait - 1 + at.level(trait, 1):lastyearnumnosameday + : prior$G has the wrong number of structures

there is one more thing I need to say:

myPredict <- predict(myTrain,newdata = myTest,type = "response",marginal = ~idh(trait):pipematerialgroup1,interval = "confidence")

This is my prediction part when I just take pipematerialgroup1 as random effect. If combine pipematerialgroup1 and installyeargroup1, what this prediction would be?

Can someone help me get the prior right? Thanks a lot.


texts: Estimating destination according to previous data

texts:
I need an advice. I can resume my problem like that :

I have some travels in a database, for example :

Person1 travelled from CityA to CityB on Date1
Person1 travelled from CityB to CityC on Date2
Person2 travelled from CityB to CityD on Date3
...

We can consider that these cities are in the complete graph.

Now, according to all the travels in the database, I would like to know where a PersonX is likely to go. I can know when he come from (or not).

I don't know if I should use machine learning, data-mining or graph theory.


texts:
If I do online learning in a setting where I have a HUGE amount of data, is that faster than doing minibatch learning (even if I optimize my batch size for GPU use, that is, use a multiple of 32 examples per minibatch)?

Details:

I have 12600 time series examples, each with 24 time steps, and each time step has 972196 binary labels. This is a multilabel problem.

Assuming float32 numbers:

loading the entire dataset should take about 1095 GB (32*12600*24*972196) = 9.4x10^12 bits)
loading a minibatch of size 32 should take about 2.7 GB (32*32*24*972196 bits)
loading one training example should take about 89 MB (32*24*972196 bits)

I'm currently using an LSTM with online learning, and it's taking about 10 seconds per training example. I'm looking to speed this up.

Other related answers:

One answer to this question implies that minibatch learning will not be helpful when there are millions of features or millions of observations (I think they're just talking about the memory limitations).

Another question is related to what I'm asking, but on the implementation side of things.

I've read this question and the answers, and it's very good but doesn't quite answer my question.