texts:
I want to make simple predictions with Keras and I'm not really sure if I am doing it right. My data looks like this:

col1,col2
1.68,237537
1.69,240104
1.70,244885
1.71,246196
1.72,246527
1.73,254588
1.74,255112
1.75,259035
1.76,267229
1.77,267314
1.78,268931
1.79,273497
1.80,273900
1.81,277132
1.82,278066

Now, I want to predict col2 by col1 and this is how I'm doing it:

df = pandas.read_csv('data.csv', usecols=[0, 1], header=None)
X = df.iloc[:, :-1].values.astype(np.float64)
y = df.iloc[:, -1:].values.astype(np.float64)
scalarX, scalarY = MinMaxScaler(), MinMaxScaler()
scalarX.fit(X)
scalarY.fit(y.reshape(len(y),1))
X = scalarX.transform(X)
y = scalarY.transform(y.reshape(len(y),1))

model = Sequential()
model.add(Dense(4, input_dim=1, activation='relu'))
model.add(Dense(4, activation='relu'))
model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer='adam')
model.fit(x=X, y=y, epochs=3, verbose=1)
for num in range(1, 21):
    Xnew = np.array([[float(Decimal('2.{}'.format(num)))]])
    ynew = model.predict(Xnew)
    print("X=%s, Predicted=%s" % (Xnew[0], ynew[0]))


texts: Are they any technical SNA tutorials that focus on real-world business problems?

texts:
Most of the technical (e.g. NetworkX, iGraph, Gephi) SNA tutorials I see focus on "toy" examples. They show how to run various algorithms, but it's not clear what real problems these would solve---which makes it all a little dry.

Does anyone know of tutorials (or other SNA learning resources) that explain how to use an analysis library/tool in the context of a real business need?


texts:
I have the last couple of month worked with an regression problem, of turning a framed audio file into a set of mfcc features, for a  speech recognition application

I tried a lot different network structures, Cnn, different normalisation techniques, different optimizer, adding more layers and so on..

but finally i've got some decent result, but i don't understand why..
What i did was i added a linear layer as output, and somehow that minimised the error tremendeosly, and bit puzzled why a linear layer would have that much effect?...

I mean am still tried to fit the actual output to the desired output?..
Why would the activation function matter here?... I mean the weight are being adjusted based on the error, so why is the neural network better at adjusting for the error when the output is linear rather than non-linear (such as: tanh, Relu).. ?


texts: Is there an Orange server to deploy the models developed in Orange 3

texts: I am new to Orange 3 and while the tool looks very promising, I was wondering if there is an Orange server where the model can be deployed for a production instance.


texts: Does it make sense to flatten two different features into a single array

texts:
When I have N samples of 2 different features (different scale, different meaning etc.), but whose values are scalar numbers of the same format.

Does it make sense to flatten the data into a 1D array of length [N*2] and feed it in this 1D shape into the neural network. Or must I keep them separate (as columns or otherwise)? I will standardize each feature before combining them into the single array.

(I want to do this for convenience reasons i.e. not deal with multidimensional arrays).

The data is input for a classification problem.


texts: Is it possible to customize the activation function in scikit-learn's MLPRegressor?

texts:
Similar to this question about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway.

Is it possible to change the activation function of the output layer in an MLPRegressor neural network in scikit-learn?

I would like to use it for function approximation.  I.e.

y = f(x)

where x is a vector of no more than 10 variables and y is a single continuous variable.

So I would like to change the output activation to linear or tanh.  Right now it looks like sigmoid.

If not, I fail to see how you can use scikit-learn for anything other than classification which would be a shame.

Yes, I realise I could use tensorflow or PyTorch but my application is so basic I think scikit learn would be perfect fit (pardon the pun there).

Is it possible to build a more customized network with MultiLayerPerceptron or perhaps from individual layers (sknn.mlp)?

UPDATE:

In the documentation for MultiLayerPerceptron it does say:

  For output layers, you can use the following layer types: Linear or
  Softmax.

But then further down it says:

  When using the multi-layer perceptron, you should initialize a
  Regressor or a Classifier directly.

And there is no example of how to instantiate a MultiLayerPerceptron object.


texts: I can use CNN to classify MNIST images, but I don't know whether CNNs are applicable on iris data as well? If not, why?


texts:
I want to build a recommender system for shops, where I recommend items.

I've learned about these systems like with content-based, collaborative filtering and so on. But now I want to make one on a very huge data.

Here I talk about a recommender system for shops where I want to recommend some items to shops, based on their similarities with other shops. To have an idea, I speak about 3000 shops, and 1 000 000+ items (I can take categories (around 1000) to begin).

So, I know that I would have to compute the similarities between shops and builds a score to determine "how much shops likes items", and other steps.

So, my questions are about :

Where should I begin ?
Do I need to try to compute a score for shop on items with utility matrix?
How to store this huge utility matrix? Is there any database designed for this?
Do you recommend any libraries, technologies or databases ? (I have heard about Neo4j for relation between shop and item, or MongoDB for shops).

I will use Python for that task.

I write this to get opinions (there is maybe lot of way to do that) and answers (like with technologies).

I'm beginner and never did recommender on huge data (around 1TB of history data for 4 years).

Thanks


texts: Weighting influence of two neural networks on classification

texts:
I'm training a model that has two neural networks. One of them is a resnet18 CNN which has as it's input images. The other one is a small one hidden layer network that has as it's input four other variables.

At this moment I concatenate the outputs of these networks in the first (and only) fully connected layer, after which datapoints are classified into three classes in the classification layer.

I was wondering how I could design my network in such a way that the two different neural networks have a specific amount of influence on the classification. For instance, I'd like the CNN to have 60% 'say'/influence in the eventual classification, and have the other NN have 40% 'say'/influence.

I can't find an intuitive way to to this in the current structure. The only way I can imagine doing it is by separating the networks, and then weighting the losses of both the networks. Does anybody know another way to achieve this without separating the networks?


texts: How to process list type questions in Question Answering task

texts:
How to generate question-answer-context triplets for questions with multiple answer strings? How to measure performance for it?
For a question with one single answer, we generate one question-answer-context triplet, and calculate EM/F1 score. Then take average scores of the whole training set as the overall performance.
For a list type question, is it correct to generate multiple triplets for each candidate answer string as separate records in the training set? Even they would share the same context and question. When calculating performance, should we combine answers from separate triplets (which share the same question and context) first, then compare them with the 'true' answer list of the question. Or just take average scores as other records with different questions in the training set, like how we calculate performance for questions with single answer?


texts: How to create training dataset based on sampled or original data?

texts:
I am trying to used SMOTE and Feature Selection by following this paper http://jad.shahroodut.ac.ir/article_825_679b8f128dec2874a8fbc314fc922127.pdf
In this paper, the authors have mentioned about 4 scenarios. I took 2 scenarios as example.

Scenario 3 (S3): using CFS, we select features from the sampled
data, and create the training dataset based on the original data.
Scenario 4 (S4): using CFS, we select features from the sampled
data, and create the training dataset based on the sampled data.

The codes for the scenarios (data sampling and feature selection) are:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
sm = SMOTE(random_state=2)
X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel())
select = SelectFromModel(LogisticRegression(class_weight='balanced',penalty="l1",C=0.01,solver='liblinear'))
X_ = select.fit_transform(X_train_res, y_train_res)
model = LogisticRegression(class_weight='balanced',penalty='l2',C=0.01).fit(X_,y_train_res)
y_preds = model.predict(X_)

My question is how could I create training data based on the samples or the original data as mentioned in the scenarios?


texts: Predicting Wave Trends of Candelestick Charts in Tensorflow (JS)

texts:
I'm relatively new to ML but my goal it to use Tensorflow.js and build a ML model that can help me detect a certain wave formation for an automated trading system. Examples of the 3-leg pattern I am trying to identify:

I've been studying ML concepts but I still can't wrap my head around how to start writing actual code in tensorflow.js in relation to this problem, what would be the logical steps to create this model? Can someone point me in the right direction, perhaps even show some untested code that might help me understand how to go about this problem.


texts:
I've trained a model with Keras, saved it and when I'm trying to apply it on new data, I'm encountering an error :

ValueError: Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1)

Here's the code for training and saving the trained model :

# Import necessary modules
import numpy as np  # numpy is just used for reading the data
import keras
from keras.layers import Dense
from keras.models import Sequential
from keras.models import load_model # To save and load model

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# load the dataset
dataset = np.loadtxt("modiftrain.csv", delimiter=";")

# split into input (X) and output (Y) variables
X = dataset[:,0:5]
Y = dataset[:,5]

# create model
model = Sequential()

# Add the first layer
# input_dim=   has to be the number of input variables.
# It represent the number of inputs in the first layer,one per column
model.add(Dense(12, input_dim=5, activation='relu'))

# Add the second layer
model.add(Dense(8, activation='relu'))

# Add the output layer
model.add(Dense(1, activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model
model.fit(X, Y, epochs=150, batch_size=10)

# Evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

# Save the model
model.save('model_file.h5')

Here's the code for predicting :

import numpy as np
from keras.models import load_model # To save and load model

# Load the model
my_model = load_model('model_file.h5')

# Load the test data file and make predictions on it
predictions = my_model.predict(np.loadtxt("modiftest.csv", delimiter=";"))

print(predictions.shape)

my_predictions=my_model.predict(predictions)

print(my_predictions)

Here comes the error :

Traceback (most recent call last):
  File "predict01.py", line 14, in <module>
    my_predictions=my_model.predict(X_predictions)
  File "C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\models.py", line 913, in predict
    return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\training.py", line 1695, in predict
    check_batch_axis=False)
  File "C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\training.py", line 144, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1)

Thank you for your help.


texts: How to use ImageDataGenerator for uint16 images

texts:
Keras does not support png uint16 images and convert them into uint8. I found some changes made here, but not  adopted yet in the installed version in Google Colab. I wonder if there is  any way to adopt it.
Thanks.


texts:
I'm a newbie researcher working on model-based genetic algorithms, mainly linkage learning in both discrete and continuous spaces, using data modeling. I would like to ask you about Ising Spin Glass (ISG) problem in the context of optimization. I've seen a lot of papers running benchmarks on this particular problem. I've googled it, checked wiki but eventually understood nothing :/

What is an intuitive definition of this problem, making it well suited for optimization?
When referring to ISG problem in papers, which specific model researchers have in mind?
How to formulate this problem in the context of optimization? (encoding, fitness evaluation)

I don't ask nor expect a ready solution, some general direction would suffice as I feel kinda lost.

Update 1

I've found some python repo on github. It calculates energy in a following way:

import numpy as np

def totalEnergy(J, configuration):
    """
    calculate the energy for a given configuration.
    Parameters:
        >>> J:
        nSpin*nSpin matrix, J[i,j] is the interation between spin i and j.
        >>> configuration:
        1*nSpin vector, configuration[i] stores the spin at site i.
    Returns:
        the total energy of this configuration.
    """
    # 0.5 for double counting
    return 0.5*np.dot(configuration,np.dot(J,configuration))

Update 2

Found some papers:

Pál, K. F. (1996). The ground state energy of the Edwards-Anderson Ising spin glass with a hybrid genetic algorithm. Physica A: Statistical Mechanics and Its Applications, 223(3-4), 283–292. doi:10.1016/0378-4371(95)00348-7
Prügel-Bennett, A., & Shapiro, J. L. (1997). The dynamics of a Genetic Algorithm for simple random Ising systems. Physica D: Nonlinear Phenomena, 104(1), 75–114. doi:10.1016/s0167-2789(96)00163-7
Finding Ground States of Sherrington-Kirkpatrick Spin Glasses with Hierarchical BOAand Genetic Algorithms
The role of crossover operator in the genetic optimization ofmagnetic models
Searching Ground States of Ising Spin Glasses with a Tree Bond-Based Representation

Once I get through them, I'll code it into my benchmarks package. I'm going to release it on pypi in a few weeks, if anyone's interested.

Answer

There's a list of ISG instances at Parameterless Population Pyramid (P3) repository.

-140 1110100000000101010100001011011001010010111011010111000110101010111010101111100010101010100001101010
200
0 1 1
1 2 1
2 3 -1
3 4 -1

It works like this:

-140 - global optimum

111010... - best genome

200 - number of dependencies

From record 3 onwards you got specified dependencies.

Example

Let's take genome 1 0 1 1 1 0 1. Values 0 must be substituted to -1. As problem is binary that is just convention in repo associated with data types underneath.

After transformation we have following genome: 1 -1 1 1 1 -1 1.
Let's assume following dependencies:

0 1 1
1 2 1
2 3 -1

We iterate over each row. We take first row 0 1 1. First 2 values are indices. You take gene values at that positions. In genome 1 -1 1 1 1 -1 1 values at these indices (0, 1) are 1 and -1. We multiply those values. We get -1. Third value in this tuple (0 1 1) denotes multiply factor. You take your previous score -1 and multiply it by that factor 1. We get -1. You repeat that process for every dependency. To get overall fitness you just sum up all the subscores.

genome: 1 -1 1 1 1 -1 1

dependencies:

0 1 1 = 1 * -1 * 1 = -1
1 2 1 = -1 * 1 * 1 = -1
2 3 -1 = 1 * 1 * -1 = -1

fitness = sum(-1, -1, -1)


texts:
I've searched the web and there are hundreds of recommendations on what to read. The time moves on and new better quality techniques are published, so I would like to know what is relevant in 2018?

My background is 4 years of BSc in Maths & Stats (top uni) + 1 year of role in Data Science (building predictive models, no NLP).

If possible, please separate it into sections/readings, e.g.

Background (History, e.g. philosophical)
Theoretical (Mathematics)
Practical (Using Tensorflow and other NLP libraries to build algorithms)

I have a few side projects that I would like to do:

Build an algorithm which answers multiple choice questions

E.g. given a question:

Which is not a fruit?
1) Apple 2) Cucumber

I would like NLP to understand negation, and find that the topic of the question is fruit. Then I'd probably incorporate Google Search API or something.

Categorise a list of 'keyword' searches into categories.

Let's take google which probably has something like this, it categorises every keyword and gives recommendations. Given a list of 10,000 searches, I would like to categorise them into N categories, based on similarity (not just how similar the words are, but including synonyms).


texts: GAN training the average of my train data

texts:
I have been training a GAN with 1D convolutional layers on sinus functions. However if I start varying my sinus (random amplitude for example), the model generates only the average of the random range. So if I give the amplitude a uniform range of (0,1], after training my generator only produces sinus functions with amplitude 0.5.

When training on batch, I sample n=batch_size samples from function;

x = np.array(np.tile(np.linspace(-3 * np.pi, 3 * np.pi, num=ts_dim), n).reshape(n, ts_dim))
y = np.random.rand(n,1)*np.sin(x)

I tried using MC dropout (dropout not only in training, also in prediction) to get generated timeseries from a probabilistic distribution. This way I steer the generator into outputing also 'less plausible' timeseries. It seemed to help a little, but still the generated functions are around amplitude 0.5.

Are there any other ways to overcome this behaviour?


texts: How are batch gradients computed on embedding layers?

texts:
Consider the following model, which is more or less a 12-dimensional vector lookup table with 10 rows, initialized to all zeros.
model = keras.models.Sequential()
model.add(keras.layers.Embedding(input_dim=10, output_dim=12, embeddings_initializer=keras.initializers.zeros))
model.compile(optimizer=keras.optimizers.SGD(),loss=keras.losses.MeanSquaredError())

I simply want it to train to the following data:
x = numpy.append(numpy.zeros(10000), numpy.ones(10000))
y = numpy.append(numpy.random.multivariate_normal(numpy.zeros(12), numpy.diag(numpy.ones(12)), 10000),
                 numpy.random.multivariate_normal(numpy.ones(12)*2, numpy.diag(numpy.ones(12)), 10000), axis=0)
model.fit(x,y,epochs=1,batch_size=1)

When the batch size is 1, the model behaves predictably; using stochastic gradient descent, we train the weights of the embedding layer towards the means of the two conditional distribution - i.e. model(0) tends towards [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], and model(1) tends toward [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2].
However, for batch sizes larger than 1, the weights of the model tend toward the average value of y, without respect for the value of x - i.e. both model(0) and model(1) tend toward [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1].
Why does this happen? If we just want a lookup table, we can alternately implement it using a dense layer with no bias term; this simply requires that we one-hot encode the input.
model = keras.models.Sequential()
model.add(keras.layers.Dense(12, input_shape=(10,), use_bias=False, kernel_initializer=keras.initializers.zeros))
model.compile(optimizer=keras.optimizers.SGD(),loss=keras.losses.MeanSquaredError())

x = tensorflow.one_hot(numpy.append(numpy.zeros(10000), numpy.ones(10000)), 10)
y = numpy.append(numpy.random.multivariate_normal(numpy.zeros(12), numpy.diag(numpy.ones(12)), 10000),
                 numpy.random.multivariate_normal(numpy.ones(12)*2, numpy.diag(numpy.ones(12)), 10000), axis=0)
model.fit(x,y,epochs=16,batch_size=128)

This behaves as I would expect, even when using mini-batch for training. So why doesn't it work correctly using the embedding layer?


texts: How does $\chi^2$ feature selection work?

texts:
I can't find the information how $\chi^2$ are used to select numerical features for a model.

Fro instance, If I employ the sklearn library:

from sklearn.datasets import load_iris
from sklearn.feature_selection import chi2

iris = load_iris()
X, y = iris.data, iris.target # X contains 4 features, y does 3 classes

I can build the following table for the dataset

pivot_table = np.zeros((X.shape[1], len(np.unique(y))))
pivot_table[:, 0] = X[y==0].sum(axis=0)
pivot_table[:, 1] = X[y==1].sum(axis=0)
pivot_table[:, 2] = X[y==2].sum(axis=0)

print(pivot_table.T)
array([[250.3, 171.4,  73.1,  12.3],
       [296.8, 138.5, 213. ,  66.3],
       [329.4, 148.7, 277.6, 101.3]])

Then applying  the contingency table to it:

from scipy.stats import chi2_contingency
chi2_contingency(pivot_table, correction=False)

Output:

125.58397740261773
1.0924329599765022e-24
6
[[213.82265358 301.31664021 361.36070621]
 [111.8757204  157.6540915  189.0701881 ]
 [137.51492279 193.78458652 232.40049069]
 [ 43.88670323  61.84468177  74.168615  ]]

But this only provides the information about dependence of variables. And they are dependent because p-value is quite low.

Then I use the other function and I get what I want:

chi2(X,y)
(array([ 10.81782088,   3.7107283 , 116.31261309,  67.0483602 ]),
 array([4.47651499e-03, 1.56395980e-01, 5.53397228e-26, 2.75824965e-15]))

But how does this work? What math lies at the heart of it?


texts: Creating validation data for model comparison

texts:
I am working on building a scoring algorithm for student data,

say the attributes are :

name, location, age, class, school_name, skill1, skill2, skill3

based on these data I need to create a student score.

I need to assign weight-ages for age, class, school_name skills and come up with a score for student.

say I have 2 scoring models like :

score_1 = x1*location_weight + x2*age_weight + x2*class_weight + x3*school_name_weight + x4*skill1_weight + x5*skill2_weight + x6*skill3_weight

score_2 = y1*location_weight + y2*age_weight + y2*class_weight + y3*school_name_weight + y4*skill1_weight + y5*skill2_weight + y6*skill3_weight

now how can I compare these models and evaluate them?

The problem is I don't have a test or validation set to prove or compare how accurate each of these model is, so in this case what is the best approach to compare and validate different models? also what is the best ways to build a validation set from scratch?


texts:
sapply(c(2:5), function(y){
x[y,]<-x[y,]-(x[1,]*x[y, 1]/x[1, 1])})

for (i in c(2:5)){
  x[i,]<-x[i,]-(x[1,]*x[i, 1]/x[1,1])}

Can someone explain to me why I am getting different results, I am coming from Python, I am not used to these apply functions of R

Thanks!


texts: Can training label confidence be used to improve prediction accuracy?

texts:
I have training data that is labelled with binary values. I also have collected the confidence of each of these labels i.e. 0.8 confidence would mean that 80% of the human labellers agree on that label.

Is it possible to use this confidence data to improve the accuracy of my classifier?

Would the following work?

1a) If the label is 0 and the confidence data for that label is 0.8
then give the training data a new label of 0.2
1b) If the label is 1 and the confidence data for that label is 0.8
then give the training data a new label of 0.8
2) Calculate the new label using this method for every entry in the
training set
3) Treat the problem as a regression problem (where the labels range
between 0 and 1)
4) Classify unlabelled data based on whether the new labels are above
or below a specific value. i.e Give all predicted labels class 0 if below X and class 1 if above X.

I am currently using a SVM with an RBF kernel for my model.

Thanks in advance!


texts: Preparing training data for crop species classification from drone images

texts:
I have high resolution multispectral (R,G,B,NIR,RE bands) images of a field taken from MicaSense RedEdge mounted on a drone. There are various species of crops planted. I want to classify the crops or crop species somewhat like this.

Image ref: https://www.intechopen.com/books/processing-and-analysis-of-hyperspectral-data/hyperspectral-image-classification
I am new to this. This is a classification task, so I'll also have to label the data. How should I label the data so that it can be fed to a classification mechanism, say CNN or SVM? What would the final X -> y mapping of training data look like?
I think we can use ArcGIS for this. Still, I'd like to know the manual procedure and what is done under the hood just to gain some more understanding.
Do I have to create bounding boxes of some size around the crops, like what's done in object detection?


texts: How to improve results for clustering of words

texts:
I have a list of words (names actually) on which I would like to apply some entity resolution.
My first guess was to create clusters of similar names so I could extract a representative entity from multiple name shapes.

I need to specify that I have no labelled data, and I am not working on a document analysis (this is different from Improve results of a clustering for example), only a raw list.

To do so, and based on what I could read, I attempted two approaches :

apply n-gram transformation on my names and use k-means clustering
apply n-gram transformation, compute a similarity matrix (cosine distance) and use it for affinity propagation

Both approaches give me interesting results, yet I can't understand some of the results. For example, I get the following clusters :

Geronese, Varonese, Veronefe, Veronese, Veronesse, ...

Cameroni, Veronèse, Veronèse P., Veronése, Veronêse

Why do I get two different clusters for shapes that look so similar (except for Cameroni which I don't know why it is in that cluster) ? Is this a problem in the k-means algorithm tuning ?

Also, I tried using the silhouette metrics to find the optimum number of clusters but I get the exact same value no matter what is the number of clusters (0.315 for what it's worth).

As for the affinity propagation approach, I get a lower silhouette score for my clusters, and I get some similar effects, like having this kind of cluster :

Birttetti, Laruette, Laruelle, Larvette, Laurette, ...

Any ideas how I could improve my results (if this is possible) ?
Or maybe any idea for a better approach than mine ?


texts: R - Error in KNN - Test and training differ

texts:
I am trying to classify set of text files whether it belong to category A or category B using KNN algorithm

Here is the error that i get

> knn.pred <- knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand)

  Error in knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand) :      dims of
  'test' and 'train' differ

 > dim(tdm.stack.nl)
    [1]  184 1599
    > dim(tdm.stack.nl_test)
    [1]   1 992

As obvious the number of words used by training dataset is different from train. How can I avoid this?

I want to mention that this error i get when i divide the training and test dataset manually to (70:30) ratio. However i dont get this error if i divide the training and test dataset using samlpling method.

> # create hold-out
> set.seed(500)
> train.idx <- sample(nrow(tdm.stack), ceiling(nrow(tdm.stack) * 0.7))
> test.idx <- (1:nrow(tdm.stack))[-train.idx]
>
> # create model - knn clustering
> tdm.cand <- tdm.stack[, "targetcandidate"]
> tdm.stack.nl <- tdm.stack[, !colnames(tdm.stack) %in% "targetcandidate"]
>
> # set up model
> knn.pred <- knn(tdm.stack.nl[train.idx,], tdm.stack.nl[test.idx,],    tdm.cand[train.idx])
> dim(tdm.stack.nl[train.idx,])
[1]  129 1599
> dim(tdm.stack.nl[test.idx,])
[1]   55 1599

However this method will not work for me as I want to classify a document in a real time using the model that was built before.

Please help. Kindly let me know if you need any other info.
Arun


texts: Can someone clarify what the linear assumption of PCA is?

texts:
0
For the past few hours I've been trying to search what this linear assumption is. Some of the articles states that that your independent variables have to be linear in relationship and need some type of transformation if there is no linearity. Other articles state that your data has to be linearly separable. Which is it? Is it both?
Does it mean that that you first have to check if the independent variables are linear in relationship, then after applying PCA, check if the data is linearly separable?
OR
Check if the data, before applying PCA, is linearly separable with techniques like linear programming.
Then there is KERNEL PCA which after searching states that it is an extension of PCA where it is applied to nonlinear data. Does that mean nonlinear in relationship or linear inseparable?


texts: Doubt to use accuracy or macro f1 measure in an unbalanced classification task

texts:
I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure.

The provided data is unbalanced, and I don't have an idea about the test set (is it balanced or not), but I think it will be balanced since they use accuracy.

Anyway ..

My question: Is it a good idea to tune my system using F1-macro rather than Accuracy? since the training data is unbalanced.

or it's better to use the Accuracy?


texts: Should you perform feature scaling/mean normalization when predicting with a model?

texts: Should you also perform feature scaling/mean normalization when predicting with a model, that was trained and tested on with feature scaling/mean normalization?


texts: Basis Function Regression - Providing analytic expression - What to do with implicitly defined parameters?

texts:
within the probability distribution of p(y|x,w,c,d,a) c and d are implicitly defined within f(x). For a) I would have suggested:
p(y | x,w,c,d,a) = N(y | f(x),a)
My question: Is it okay to ignore c,d in the N-(Gaussian) expression?
And if so, concerning b): Is it correct that the expression p(t|x,w,c,d,a) = N(t | f(x),a)
so it's simply the same as we had before since y tries to approach t?


texts:
What kind of feature engineering techniques should one apply for longitudinal data comprising of individual binary time interval data about when an activity was done during the day(we have this data for every person in our database) with each interval being represented by the starting and end time ?

I have thought about pre classifying people into number of logical groups and then for each group dividing the entire day into slots of fixed interval and then in each slot finding:

number of people in that sub group that started that activity in that slot
number of people in that sub group that ended that activity in that slot

I have also thought about finding average number of time that each group spends in a day performing that activity, and repeat the above process for all days

Do we have some existing research that address this kind of data?


texts: Matplotlib pyplot: How to plot multiple trends in different areas of the plot?

texts:
I am trying plot some trends using matplotlib.pyplot. The task is straightforward, but there is one detail I can't get right. I wish to plot a numpy.array called trend, that is of a given length. I have two other trends representing the prediction intervals, called PI_high and PI_low. I wish to plot them as a continuation of trend.

trend is of length 60, therefore I wish to plot trend on the 0:60 x-axis. While PI_high and PI_low are of length 12 and shall be plotted on the 60:72 x-axis. However, when I do:

import matplotlib.pyplot as plt

plt.plot(trend)
plt.plot(PI_high)
plt.plot(PI_low)
plt.show()

The prediction interval is displayed at the very beginning of the plot (i.e. on 0:12 x-axis). How can I make it start more forward on the x-axis, on the segment that I want?


texts: How do you avoid 'analysis paralysis' when choosing a method to implement?

texts: When you have multiple methods to accomplish a task, how do you choose which one to implement?


texts: Classification model performance - metrics for getting number in each class correct?

texts:
I'm fairly new to predictive modelling, so apologies if this is a stupid question.
I am working on a classification problem (predicting if customers commit fraud or not), and have been comparing a few different algorithms, as well as tuning some of those algorithms.
As part of this I want to be able to compare how well each model (or version of each model) can predict based on some unseen data with known response values.
My response is quite unbalanced, with a ratio of about 99:1 of the binary outcomes. I haven't use up/down sampling, as I have a very large training dataset, so there are still plenty of examples of the less common outcome.
My question is about which metrics of model performance to use. I've read a fair bit about this, and have looked at various measures including confusion matrix, sensitivity, specificity, kappa etc.
My main aims are for a model that predicts the correct number of fraud cases. After that, it would obviously be better if the cases predicted as fraud are true positives. The latter I think can be measured with "Specificity". For the former I've tried just comparing the number of predicted fraud cases with true predicted fraud cases.
My question is whether there exists a metric that combines these two measures? In my reading I haven't found anything that quite does this, but wondered if others new of an existing measure?


texts:
I have several sentences that I transformed into vectors. With these vectors I would like to predict another vector (which represents a vector of a sentence (the answer)).

Can you tell me if this kind of neural network fits this problem? (In term of loss function, activation function (I didn't use an activation function for the output), and metrics.

model = Sequential()
model.add(Dense(1024, input_shape=(1200,)))
model.add(Activation('relu'))
model.add(Dense(100))
model.compile(loss='mse', optimizer='rmsprop', metrics=['mae'])

Input: Several sentences, converted into 100 dimensions array (so, 12 sentences)

Output: 100 dimensions array, that should correspond to the kind of answer we would have from the 12 sentences.

Thanks!


texts: does entity recognition comes under classification problem?

texts: I want to extract named entities from a text but I don't know whether that comes under classification or not.if it comes under classification then how to prepare classes for recognizing entities in a text.(does the collection of few org names and person names and using them as my training dataset makes the solution?)


texts: What common/simple problem would work well as a web app?

texts:
Context
I'm currently writing a simple tutorial to demonstrate a tool to data scientists and analysts that turns Jupyter Notebooks into web apps. Basically, it discusses setting up the web app as a front end, running some code in the notebook and then returning data to the web app.
Question
My question is, what is a small interesting problem in data science that I could solve in the notebook?
I'm looking for something more interesting than doubling an input but smaller/simpler than building a computer vision model.
Additional information
As you can probably tell, I am new to data science. Apologies if this is the wrong forum for this type of question.
Here is the version with a non-interesting problem being solved in the notebook, it may provide more context if needed.
Thanks in advance for any help.


texts: How to tell if a problem should use regression or classification model?

texts:
Once we know that the problem needs to be solved using supervised learning, how do we know if we have to use regression analysis or classification algorithm to solve it? Is there some thumb rules that can be used?

Most of the resources online give the standard example of prices of house for regression model and malignant/benign cancer plot for classification model. This isn't helping me much.

Is there come conceptual method to analyze the given problem?


texts: Which part should be frozen during transfer learning?

texts:
I want to use transfer learning and fine tuning and I need to decide which part of
the original model will be used and which part will be frozen. I'm thinking about four possilbe cases:
small/large new dataset and this set is similar/not similar to the original dataset. What should be done to achieve best results in each of the cases?


texts: What is the geometrical representation of the y value in a 2-variable perceptron plot?

texts:
This figure represents a perceptron model with a 2 dimensional feature vector input.

The hypothesis space of the perceptron is defined by this set:

$\{y | y = w\cdot x + b\}$

What is the geometrical representation of the y value in this perceptron plot? Is it the perpendicular distance from a point to the separating hyperplane?


texts: Splitting a neural network in 2 microservices

texts:
I have a neural network, that is already trained locally that can detect objects in the scene.

But I have to split the neural network into 2 parts, let's say it has 16 layers, and I want to have one microservice handle the first 8 layers and give the output of the 8th layer to the next microservice and it takes the data from there and proceeds with the 9th layer (1st layer in the 2nd microservice).

Sending an image to the 1st microservice will give the result from the 2nd microservice, is this feasible using TensorFlow?


texts:
I'm studying the following code, which cross_val_score_ was used as well as .mean() and .std(). I read many documentation of the meanings, but didn't get what each of the above does.
import pandas as pd
import numpy as np
from sklearn import tree
import graphviz
from sklearn.model_selection import cross_val_score

#importing the dataset
d = pd.read_csv('student-por.csv', sep= ';')

d['pass'] = d.apply(lambda row: 1 if (row['G1']+ row['G2']+ row ['G3']) >= 35 else 0 , axis=1)
d = d.drop(['G1', 'G2','G3'], axis=1 )

#Doing one-hot encoding
d=pd.get_dummies(d, columns =['sex','activities','school', 'address', 'famsize','Pstatus','Mjob','Fjob','reason','guardian','schoolsup','famsup','paid','nursery','higher','internet','romantic'])

#shuffle rows
d = d.sample(frac=1)

#split traning and test
d_train = d[:500]
d_test = d[500:]

d_train_att = d_train.drop(['pass'], axis=1)
d_train_pass= d_train['pass']

d_test_att = d_test.drop(['pass'], axis=1)
d_test_pass= d_test['pass']

d_att = d.drop(['pass'], axis=1)
d_pass = d['pass']

t = tree.DecisionTreeClassifier(criterion ='entropy', max_depth = 5)
t= t.fit (d_train_att, d_train_pass)

#to export the tree
dot_data = tree.export_graphviz(t,out_file = 'students-tree.png', label ='all', impurity=False, proportion= True, feature_names=list(d_train_att), class_names=['fail', 'pass'], filled = True, rounded=True)

t.score (d_test_att, d_test_pass)
scores = cross_val_score(t, d_att,d_pass, cv=5)

print ('Acuracy %0.2f (+/- %0.2f)' % (scores.mean(), scores.std() *2))

in short this is what I need to know:
scores = cross_val_score(t, d_att,d_pass, cv=5)

print ('Acuracy %0.2f (+/- %0.2f)' % (scores.mean(), scores.std() *2))

one more thing, am I suppose to get the same score as in the original code publisher? because I didn't.


texts: Modelling population changes with years on a network graph

texts:
This may be a silly question, apologies if it is. And further apologies if I've said something wrong.

Network graphs, or visualisations look somewhat like this, with each node representing a thing connected to another thing. So for my example, I'd like to model population changes for refugees from their origin country to their destination country.

I've been tasked to create a graph which shows a similar change over a period of years, but I am really not sure how years could be modelled? e.g. how would a graph like the example above, show a change in years?

Edit: preferably using the NodeXL plugin


texts: How do we solve this logistic regression question?

texts: Can an exact answer for this question be found? By intuition, I think the answer is 0. But can someone explain the steps on how to solve this question?


texts: Regression for binary classification and AUC metric

texts:
In the kaggle forums I found an example model where someone was using XGBRegressor for a binary (0/1) classification problem (sorry, cannot find the link any more).
This was for a competition where output is measured by AUC, so only ranks mattered.

I then tried this for a different problem with my data and also found a slight improvement using the Regressor instead of the Classifier.

Some of my predictions where negative, but it would be no problem to shift them to the [0,1] interval.

So is using regression for binary classification a common approach, and what are the pros and cons?


texts:
An undirected graph represents a database where nodes of the graph represent   tables, edges represent the joiner columns.  There are 100 databases( it means, 100 undirected graphs). We have to build a clustering model which can cluster these graphs into four groups based on attributes of nodes( that is the list of column names), attributes of edges ( that is the joiner column) and overall structure of the graph database.

How can we build the clustering models ?


texts: Unexpected shape of "training curves" in NN

texts:
I'm trying to find the best configuration for my NN (in terms of batch size, learning rate etc) and noticed the following unexpected behavior.

The AUC scores, computed on validation data, as well as the loss (also on validation data) display a "two-step progress". I'm attaching the progress of AUC and loss for 4 of my runs with a variety of learning rates:

As you can see, at about 400000 batches (that's more or less after 30 epochs),
it looks like the network gets a "second wind". I think, judging by the loss, that it starts overtraining at around 700000 batches (except in the run with the fastest degrading learning rate).

I've never seen such behavior mentioned before, is this my network escaping some local minimum? Is it normal or something I should be worried about?


texts: What are some nice algorithms/techniques for optimizing and predicting Click Through Rates (CTR)?

texts:
Definition of Click Through Rates:

  CTR is the number of clicks that your ad receives divided by the
  number of times your ad is shown expressed as a percentage (clicks ÷
  impressions = CTR).


texts: Difference between Granger causality and VAR(1)?

texts: For my VAR(1) I get that the causal variable in each equation is statistically significant at 10%. But for Granger causality at 10% I only get that 1 variable granger causes the other and not the other way round. How is that possible?


texts: How to convert this forumal to a proper code?

texts:
I have the following formula [1] that I attempted to transform in to a function via R. However, my initial outputs are disconcerting. Before I go through many blocks (images), I am hoping to get assistance with QC. The formula & my attempt to transform it are below:

  Where $f_k(i,j)$ is the intensity value of pixel$(i,j)$ in block $k$ & $\mu_k$ is the mean value of all pixels in $k$.

$$\sigma_k = \sqrt{\frac{\sum_{i=1}^H \sum_{j=1}^H(f_k(i,j)-\mu_k)^2} {H*H}}
$$

library(raster)
image_files<- list.files(".",pattern = "*.PNG$")
img_A<-raster(image_files[1])
matrix_A<-raster::as.matrix(img_A)
>>dim(matrix_A)
174,288
sigma_k<-sd(matrix_A)

  Where $c_k$ is the center pixel intensity in $k$ and $t=2.97$

$$ L_k = \frac{1 | f_k(i,j)-c_k|<t \sigma_k} {0  |f_k(i,j)-c_k|>t \sigma_k}
$$

center_A<-matrix_A[144,144]
t<-2.97

k<-apply(matrix_A,2,function(x) {x-center_A})
sigma_t<-sigma_k*t

Lk<-lapply(k, function(x) {
if (x < sigma_t){
x=1} else {
x=0
}
})

Lk<-unlist(Lk)

block_label<-data.frame(sort(table(Lk),decreasing=T)

  Finally, a comparison is conducted to identify majority "0" or "1". The majority is assigned as the mean intensity value of all pixels in the present block.

$\bar I_{\mathbf i \mathbf n} = [\hat I_{\mathbf i \mathbf n,1}, \hat I_{\mathbf i \mathbf n,2}, \hat I_{\mathbf i \mathbf n,3},...\hat I_{\mathbf i \mathbf n, n}]$

[1]https://www.hindawi.com/journals/mpe/2017/4728425/


texts: How to group categorical columns into similar types?

texts:
(Forgive me if the question is ill put. I am a novice in data science. Please comment or edit so that the question can be improved)

I have a dataset where we have to predict the future sale of a shop. "City' is one of the columns. The problem with 'City' is that data provided per city is very uneven in size(detail of only 1 shop in one city while giving detail of nearly 50 shops in another) Moreover many cities which are not present in train are present in test.

My plan is, based on other columns in train, I will find which cities are similar to each other so that I can classify those cities as a single group. (Logic is, say area, demand for an item are similar for city 1 and 3 so the sales in both of them will be similar.)

I can think of KmeansClustering or aggregated features. But I am not exactly sure how to achieve this. Please suggest some ideas, what are some good ways to deal with such cases.


texts:
I have a data set with more than 20 features, and I applied PCA:

M.fit_transform(all_data)
variance = M.explained_variance_ratio_
var = np.cumsum(np.round(M.explained_variance_ratio_, decimals=3)*100)
plt.ylabel('% Variance Explained')
plt.xlabel('# of Features')
plt.title('PCA Analysis')
plt.ylim(30,102.5)
plt.plot(var, marker="s")
plt.show()

Printing the var variable, I get

array([ 89., 100., 100., 100., 100., 100., 100., 100., 100., 100.])

I understand this tells us that the variance is explained by 2 features.

So I calculated it again, now the 2 components:

from sklearn.decomposition import PCA
M = PCA(n_components = 2)
X = M.fit_transform(all_data)
plt.scatter(X[:,0],X[:,1])

And this gives a "random looking plot". I understand that the data was changed during the PCA process.

What can I do with this information? How will this help me understand the data?

Is it useful per se? Is it useful as a preparation method for other methods? Which ones can I try?


texts: can anyone please offer suggestions on ways to programmatically generate time series data artificially. if possible, mimic the distribution of an existing dataset (say hourly humidity readings) and add some noise if required. Any suggestions will be greatly appreciated!


texts: How to replace short words into full words from tweets using python

texts:
I am doing sentiment analysis on tweets. Most of the tweets contains short words and i want to replace them as original/full word.

Suppose that tweet is:

I was wid Ali.

I want to convert:

wid -> with

Similarly

wud -> would

i have 6000 tweets in which there are lots of short words. How i can replace them ? is there any library available in python for this task? or any dictionary of shorts words available online?

i read answer of Replace appostrophe/short words in python Question but it provides dictionary of appostrophe only.

Currently i am using NLTK but this task is not possible with NLTK.


texts: Reduce size of a network graph for bipartite projection

texts:
I have a graph that I created from a pandas data frame. The length of the graph is ~450k edges. When I try to run the weighted_projected_graph function, it runs for a long time (I have not seen it finish), presumably because of the size of this data set. What is a good method for reducing the size of this data set before creating the bipartite graph?
I have tried narrowing it down by using the most connected components:
trim1 = len([c for c in net.connected_component_subgraphs(g) if len(c) > 10])
C = net.Graph()
gg = sorted(net.connected_component_subgraphs(g), key=len, reverse=True)[trim1]

but I don't think this is giving me the results I want and, further, I'm not confident this is an analytically sound strategy. Does anybody have any other recommendations for reducing the size of this set?
EDIT: full code, without reducing. What I would be trying to do with the above would swap out g below for gg from above in the call to bi.projected_graph
reviews = pd.read_csv(r"\BX-Book-Ratings.csv", sep=";", encoding = "ISO-8859-1")
users = reviews['User-ID'].values.tolist()
books = reviews['ISBN'].values.tolist()

g=net.from_pandas_edgelist(reviews,'User-ID','ISBN',['Book-Rating'])

print(len(g))
>>> 445839

p = bi.projected_graph(g, users)
```


texts: Automatically Remove highly correlated features using Variance Inflation Factor?

texts:
I want to be able to automatically remove highly correlated features. I am performing a classification problem using a set of 20-30 features and some may be correlated.

Multiple features can be correlated at once too and I fear it may pose a problem in my Logit model significances & coefficients of the features. After removing these features I plan to then also use this reduced feature set into an Xgboost model as well.

  Multicolinearity on the other hand is more troublesome to detect because it emerges when three or more variables, which are highly correlated, are included within a model. (Reference here)

Would it be correct to remove correlated variables this way?:

X=data[feature_set]
vif = pd.DataFrame()
vif['vif_factor'] = [variance_inflation_factor(X.values,i) for i in range(X.shape[1])]
vif['features'] = X.columns

vif.sort_values('vif_factor',axis=0,inplace=True, ascending=False)

features_to_remove = vif.loc[vif['vif_factor'] > 10,'features'].values
features_to_remove = list(features_to_remove)
print(features_to_remove)

Output:

vif_factor |  feature

21         |   age
9.7        |   income
7          |   gender ....and so on

So in this case age would be removed as a feature from the model.


texts: Classifier Performance - Binomial proportion confidence interval

texts:
I am solving a binary classification task.
As an output of Random Forest classifier, I get a probability of how sure RF is that class is 0 or a 1.
How can I calculate the needed threshold, to be 95% sure that calculated output is really a class my RF predicted.
(I do not want to split the data at 0.5, rather I would take class 0 if it is lower than 0.3 and class 1 if it is higher than let say 0.7. Although I would like to get the least spread and get 95% of right data choices for 0 and 1).


texts: Formulate a MDP for a problem given below

texts:
Can someone help me to formulate an MDP for the below problem ?.

Problem Definition

Bunny wakes up in a strange room with 2 doors; one on the left, and one on the
right.  In front of him is a map of the building, laying out clearly what lies behind each door.  Behind one of the doors is a room with the way outside, and behind the other one a room with a hungry tiger.  Bunny does not like tigers.  Especially hungry ones.

Consider  the  control  problem  where  the  current  state  is  specified  by  the current  room  Bunny  is  in,  and  the  actions  Bunny  can  take  are  to  move through  a  door  to  another  room.   Assume  that  there  are  2  doors  in  the starting room,  each leading to its own room,  and that one of these other rooms (specifically the one on the left) has a single door that leads to the exit, while the other one (specifically the one on the right) has a tiger in it.
Moreover, assume that Bunny has full knowledge of these details but that
there is a 10% chance that when trying to move through one of the doors,
he gets confused and goes through the other door instead.  Lastly, assume
that the episode does not end when Bunny enters the room with the tiger.

Questions

Formulate a Markov Decision Process (MDP) for the problem of con-
trolling Bunny’s actions in order to avoid the tiger and exit the building.
(Give the transition and reward functions in tabular format, or give the
transition graph with rewards).

I would like to know, is there any procedures or rules, that needs to be considered before formulating an MDP for a problem

  What i have done so far

State A - Starting room
State B - Room with an exit door
State C - Room that has a tiger in it

I don't know how to proceed further.


texts: Pretrained features return worse results for sklearn classifier/pipeline

texts:
So I have a following scenario:

Pipeline, that transform text/dict/numerical data and classifies the result with linear regresion. It looks something like this:

args = {
    'y': labels,
    'cv': 5,
    'scoring': 'accuracy',
    'n_jobs': -1
}

feats = ('features', FeatureUnion([
    fulltext_pipeline,
    numerical_pipeline,
    nela_desc_pipeline,
    v_tags_pipeline,
    os_pipe
]))

clf = ('clf', LogisticRegression(multi_class='auto',
    solver='liblinear',
    tol=0.02,
    C=500,
    penalty='l1',
    random_state=0))

os_pred = Pipeline([
    feats,
    clf
])

And for evaluation I am using cross_val_score. So far so good, everything works as suppoused, but if I try to extract "features" and run them separately, I get worse results. The example looks as follows:

features = feats[1].transform(data)
np.average(cross_val_score(clf[1], features, **args))
np.average(cross_val_score(os_pred, data, **args))

I expected to get an exact same results here (as in theory they should do the same - maybe I am missing something?)

I've tried with .transform(data), .fit_transform(data), .fit_transform(data, y=labels). The odd thing for me was that all three methods generate the same output (sparse matrix) which is far worse compared to results from pipeline.

Data is in form of sparse matrix

<421x862 sparse matrix of type '<class 'numpy.float64'>'
    with 361371 stored elements in Compressed Sparse Row format>

While labels are normal strings (3 classes).

Results are as follows:

0.672436 from np.average(cross_val_score(clf[1], features, **args))

0.772509 from np.average(cross_val_score(os_pred, data, **args))

I double checked for any random seed or something of this sort, but the results are consistend and pipeline/classifier return always the same result. And for me it looks like that this is quite a big difference (given the fact that the range is [0.00; 1.00]).

Question at the end is: Why classifier seems to be biased in a pipeline scenario compared to just prefeeling it with features directly?

I do aknowledge that pipeline does some kind of additional steps of transforming/fit-transforming/etc., but with my current knowledge of sklearn library I cannot reproduce them "outside" the pipeline.


texts: How to do target encoding when data has repeated rows?

texts:
How can I do encoding for a category when data has repeated rows? Can I do target encoding? Or Is there another encoding I can use?

I want to figure how to include a categorical variable in a model to predict a numerical variable Y.    Because I am working with some legislative data, my challenge is my category code is over 4000 unique values, those values that cannot be easily grouped(*), and they can have repeats.  In fact, anecdotally I know that both that code is a strong predictor of Y and repeats in a code per row will have a stronger impact on Y.

My attempt at a solution:
In fact, I tried to do this myself thinking that maybe something as simply as mean target times the count of each code (Table B * Table C). However, I am not sure if this would be valid type of encoding.

A = sample legislative data to predict ethics variable Y, the values have been anonymized

+-----+-----------+------------+
| row | Category  | Y variable |
+-----+-----------+------------+
|   2 | 15-11-2   |        370 |
|   2 | 15-11-106 |        370 |
|   2 | 15-5-44   |        370 |
|   2 | 15-11-41  |        370 |
|   3 | 15-7-23   |        363 |
|   3 | 15-7-23   |        363 |
|   3 | 15-11-37  |        363 |
|   3 | 15-11-37  |        363 |
|   3 | 15-5-21   |        363 |
|   6 | 15-11-106 |        287 |
|   6 | 15-5-21   |        287 |
|   7 | 16-5-70   |        931 |
|   7 | 15-5-23   |        931 |
|   7 | 15-5-41   |        931 |
|   9 | 15-11-7   |        336 |
|   9 | 15-2-30   |        336 |
|   9 | 40-11-76  |        336 |
|   9 | 15-2-2    |        336 |
|   9 | 15-11-131 |        336 |
|   9 | 15-11-131 |        336 |
|  10 | 15-2-30   |        350 |
|  10 | 15-11-131 |        350 |
+-----+-----------+------------+

B = The Mean Values for the unique rows

+-----------+-----------------------+
| Category  | Average of Y variable |
+-----------+-----------------------+
| 15-11-106 | 334.714               |
| 15-11-2   | 298.994               |
| 15-11-37  | 251.931               |
| 15-11-41  | 399.210               |
| 15-11-7   | 288.021               |
| 15-2-2    | 234.533               |
| 15-2-30   | 303.127               |
| 15-5-21   | 317.999               |
| 15-5-23   | 252.109               |
| 15-5-41   | 300.882               |
| 15-5-44   | 499.405               |
| 15-5-70   | 277.306               |
| 15-7-23   | 244.976               |
| 40-11-76  | 261.938               |
| 15-11-131 | 319.799               |
+-----------+-----------------------+

Table C The counts of each category

+-----+-----------+-------------------+
| row | Category  | Count of Category |
+-----+-----------+-------------------+
|   2 | 15-11-106 |                 1 |
|   2 | 15-11-2   |                 1 |
|   2 | 15-11-41  |                 1 |
|   2 | 15-5-44   |                 1 |
|   3 | 15-11-37  |                 2 |
|   3 | 15-5-21   |                 1 |
|   4 | 15-7-23   |                 2 |
|   4 | 16-8-18   |                 1 |
|   5 | 16-7-1    |                 1 |
|   6 | 16-8-2    |                 1 |
|   6 | 15-11-106 |                 1 |
|   7 | 15-5-21   |                 1 |
|   7 | 15-5-23   |                 1 |
|   8 | 15-5-41   |                 1 |
|   8 | 16-5-70   |                 1 |
|   8 | 16-11-37  |                 1 |
|   8 | 16-7-21   |                 1 |
|   9 | 15-11-131 |                 2 |
|   9 | 15-11-7   |                 1 |
|   9 | 15-2-2    |                 1 |
|   9 | 15-2-30   |                 1 |
|   9 | 40-11-76  |                 1 |
|  10 | 15-11-131 |                 1 |
|  10 | 15-2-30   |                 1 |
|  10 | 40-11-76  |                 1 |
|     |           |                   |
+-----+-----------+-------------------+

Footnote:
(*) What I mean by "the category code cannot be easily grouped" is that 15-11-2 and 15-11-1 cannot be group under 15-11 or 15.  I have also tried clustering.


texts:
I am trying to reproduce results from one paper, where authors minimized the following loss function
\begin{align}
\min_{w \in R^d} \frac{1}{n} \sum_{i \in [n]} log(1 + exp(-y_ix_i^Tw))+\frac{\lambda}{2}\|w\|^2,
\end{align}
where $w$ are weights are $\lambda$ is regularization parameter for  ijcnn1 dataset.

This dataset is specific by unbalanced data (90%-0, 10%-1).
As a preprocessing step, I applied MinMaxScaler and StandardScaler.

I use model written in keras, which looks simply as:

model = keras.Sequential([
    keras.layers.Dense(1, activation="sigmoid", kernel_initializer='uniform', kernel_regularizer=regularizers.l2(1e-4), use_bias=True)
])
sgd = optimizers.SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(optimizer=sgd, loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_dataset, epochs=1000)

However, I manage to obtain only 91% accuracy at the best.
Looking at the prediction, I observe that my model learns to predicts almost everything to be zero. I also tried using class_weight, but it did not seem to help.
Does anybody have any suggestions, how to obtain better results?


texts: Do pseudo r squared metrics make sense for classifiers that aren't logistic regression?

texts:
I'm working with some domain scientists that are used to using logistic regression to predict a binary value.  One of the ways they evaluate their logistic regression model is through the Nagelkerke $r^2$ value, which attempts to measure how much of the variance of the dataset is explained by the model.

They're now trying to compare the performance of their logistic regression model to some machine learning models (random forest, for example), and they'd like to calculate the $r^2$ value for the machine learning models.

Is this possible?  Is there a good reason this wouldn't make sense to do?

The Nagelkerke $r^2$ value is a rescaled version of the Cox and Snell $r^2$, which is calculated using the likelihood of a fitted logistic regression model vs. the likelihood of a null logistic regression model, according to wikipedia.

$r^2_{CS} = 1 - (\frac{L_0}{L_M})^{2/n}$

where $L_0$ is the likelihood of the null model and $L_M$ is the likelihood of the fitted model.

In my mind, I can calculate likelihood (or some proxy) for machine learning models like a random forest, so I should be able to compare it to a null logistic regression model, right?  I can at least get an estimation of the probability of each class out of a random forest model, which I should be able to use to calculate something like the likelihood.

Is that a bad idea, though?  I was surprised to not find much on this with a bit of googling.


texts:
I am trying to understand and use Bayesian Networks. I see that there are many references to Bayes in scikit-learn API, such as Naive Bayes, Bayesian regression, BayesianGaussianMixture etc. On searching for python packages for Bayesian network I find bayespy and pgmpy.

Is it possible to work on Bayesian networks in scikit-learn?


texts: Should a filter learned within a residual block be different form its vanilla CNN counterpart?

texts:
I have a very basic CNN using Conv2d with multiple layers and activations, each layer $\ell$ has parameters $w_\ell$ inducing a mapping $f_\ell(x,w_\ell)$. Now I decide to introduce skip connections to each layer separately, meaning the output of each layer is now $x+f_\ell(x,w_\ell)$. Should I expect any significant difference if I were to try to visualize each kernel?
My intuition is that in theory any change to the filter wouldn't stem from the fact we are learning the residual part of the function rather than the function itself: the identity part could be seen as a convolution with a kernel that has one pixel set to 1 and the rest are 0, so $w_\ell$ shouldn't differ too much before and after but one pixel. So if there is any difference, would it only stem from the learning stability and us being able to get better filters? I might be wrong, am I new to Res-Nets and trying to wrap my head around how they act.
Note by "visualizing filters" I don't mean anything specific, we can see the weights as numerical values or look at the images the kernels induce.


texts: Incident duration prediction but online with trend

texts:
I am currently looking into some papers discussing Incident duration prediction. The dataset consists of ca. 250 monitored events. For all of those incidents are multiple features available and the main goal is the prediction of the incident duration. These durations are right-skewed and log-normal distributed. ANOVA test (and KS) identified 13 binary features to be independent.

To predict the durations the researchers compared Multiple linear regression, DT, ANN, SVM / RVM and KNN.

Currently, I am looking into a way to transfer the knowledge from this case to another task. Predicting the duration of incidents in an online system (online in the sense of continuous flow of information) and also accounting for trends (e.g. increasing duration of incidents because it is getting darker or colder at the same time throughout different seasons).

Unfortunately, there is such a huge amount of methods besides the mentioned methods (non-parametric, hazard models, reliability models, etc) - it seems like a jungle


texts:
In a Random Forest context, do I need to setup dummies/OnehotEncoder in a dataset where features/varibles are numerical but refer to some kind of category?
Let's say I have the following variables:

Where Y is the variable I want to predict. X's are features.
I will focus on X1. Its numerical but refers to a specific category (i.e. 1 refers to math, 2 refers to literature and 3 for history).
Do I need to apply OnehotEncoder (or dummy approach) for a Random Forest algoritm?
I guess I don't need to do it, but I 'm not sure.


texts: How to model the probability of detecting an image, given it is seen multiple times

texts:
Are there any existing methods/models describing the probability of an object being detected by a computer vision algorithm given it is seen $n$ times at similar angles and orientations? I know that an autonomous car may, for example, have trouble recognizing a stop sign and as a result the image recognition system being used may have a bounded box around the stop sign continuously appearing and disappearing signaling that the object recognition algorithm is only detecting the stop sign a certain amount of the time it sees it. I would like to understand this phenomenon in a more general sense.
More precisely put, I would like to model that: given an object is detected with probability $p$ if it is seen 'once' (AKA during a moment in time), then what is the probability of it being detected if it is seen for a second time, a third time, etc... Each 'moment in time' may be characterized by a single video frame/image or some other unit of time. Each time the image is seen it may be in a similar but not identical orientation.


texts: Auto-Encoder to condense (pre-process) large one-hot input vectors?

texts:
In my 3D game there are 300 categories to which a creature can belong.
I would like to teach my RL agent to make decisions based on its 10 closest monsters

So far, my Neural Network input vector is a concatenation of ten 300-dimensional one-hot encoded vectors. It would be awesome if I had, say ten 40-dimensional vectors instead.

So, is it possible to pass a 300-dimensional one-hot encoded vector through an auto-encoder, to snatch its compressed version (embedding) from the middle of the auto-encoder?

This would allow me to concatenate several of such compressed embeddings as a "total" input (ten concatenated 40D vectors), without bloating this total input vector.

In my game, each one-hot is supposed to represents a distinct category. My categories are not correlating in any way.

1) Will it be valid, or will it introduce unwanted assumptions, like Label-Encoding does?

2) How to train this Auto-Encoder? Can I simply pass my one-hots through it, and fine tune until I get an exact one-hot on the other end of the Auto-encoder?

3) Is there a better way to describe several monsters?


texts: Why can't I choose my hyper-parameter in the training set?

texts: Say I've divided the data into 3 parts: training, validation and test. I know for example, that in Neural Networks, the number of hidden layers is a hyper parameter. Why can't I train numerous NN architectures in the training set and then test their accuracy in the test set; thereby allowing me to choose one final model? What is the purpose of the validation set in this instance?


texts: Why are RNN/LSTM preferred in time series analysis and not other NN?

texts:
I had recently a great discussion about the advantages of RNN/LSTM in time series analysis in comparison to other Neural Networks like MLP or CNN.

The other side said, that:

The NN just have to be deep enough to model the time connections
RNNs are just used, because they are faster
With RNN the older (e.g. t-20) timepoints are not that relevant in comparison to the newer timepoints (e.g. t-1), because the data from older timepoints have to go through all neurons until the weight is updated in this neuron.
It is easier to understand the weight matrix of normal NN in comparison to RNN to understand partly the behavior of the NN

My answer to that was:

Everyone suggests using the LSTMs for times series analysis
It can consider the time better in comparison to a sliding window approach especially if the length of the sequence is quite long
To 3. That happens just if you use vanilla RNN instead of LSTM, because of the gradient explosion

So I know that I did not answer him completely (and maybe not correctly) and I am now really interested in the real reasons why LSTMS are suggested the most of the times. Maybe he is right? What do you think?

Can a normal NN model the time connections the same way like a
RNN/LSTM does when it is just deep enough?
Does an RNN need more or less data in comparison to a NN to get the
same/ better results?
Are there time series where normal NN or RNN/LSTM perform better?
Is it time data depended which Model will perform the best or are there some guidelines?
Can the NN behavior be understood better than the RNN behavior?


texts: Useful metrics to compare network-output image to true image?

texts: I'm designing a supervised network that would require to output an image. I'm wondering what are the best metrics to find similarity between the output and actual target image. So far, my best assumption is to calculate the distance of RGB values, and give less weight to the background. Are there better ways?


texts: Problems with Linear Discriminant Analysis Classifier

texts:
I wrote two functions for determining the linear discriminant classifier of a EEG data-set. The data set consists preprocessed EEG data ∈5×62×5322 and stimulus labels ∈2×5322 during a copy-spelling paradigm with a P300 speller. The data matrix X contains 5 selected time windows of EEG activity at 62 electrodes after a visual stimulus was presented on the screen in front of the subject. If the first row of  is 1, the stimulus was a target stimulus, if the second row of  is 1, the stimulus was a non-target stimulus. The first function returns the weight vector and the bias term. The second function is a graph class to show the result

def lda_fit(X,Y):

# class means

unique_classes=np.unique(Y)
mu=np.zeros((len(unique_classes),X.shape[1]))
for i,name in enumerate(unique_classes):
    mu[i,:] = X[Y==name,:].mean(axis=0)

mupos=mu[1]
muneg=mu[0]
mupos=mupos.reshape(155,2)
muneg=muneg.reshape(155,2)
Xneu=X[0].reshape(155,2)

# D-by-D inter class covariance matrix (signal)
Sinter = np.dot((muneg-mupos),(muneg-mupos).T)

# D-by-D intra class covariance matrices (noise)
Sintra =np.dot((Xneu-mupos),(Xneu-mupos).T)+np.dot((Xneu-muneg),(Xneu-muneg).T)

# solve eigenproblem
eigvals, eigvecs = sp.linalg.eig(Sinter,Sintra)
w = eigvecs[:,eigvals.argmax()]
# bias term
b = (w.dot(mupos) + w.dot(muneg))/2.
# return the weight vector
return w,b

I get the following error: "ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 155 is different from 310)"

I know it has something to do with the shape of the matrices, but I really stuck.


texts: How sensitive are feedforward Neural Network to weight changes?

texts:
Let's consider the space of feedforward neural networks with a given structure: $L$ layers, $m$ neurons per layer, ReLu activation, input dimension $d$, output dimension $k$.

Which means I'm considering the map $F: \mathcal{W}_1 \times \mathcal{W}_2 \times \dots \times \mathcal{W}_L \times \mathbb{R}^d \to \mathbb{R}^k$, where $\mathcal{W}_i$ is the space of possible weights for layer $i$. We also assume, for simplicity, that every weight matrix has a norm upper bounded by a constant $M$.
Let's now assume that I have fixed parameters so that we obtain $v = F(W_1, \dots, W_L, x^*) \in \mathbb{R}^k$ (note that $x^*$ is fixed as well).

Now imagine that I inject some random noise $\eta \in \mathbb{R}^{m \times m} $ in a weight matrix $W_i$, where the norm of the noise is 10% of the norm of the matrix, e.g. $||\eta|| = ||W_i||/10$  . How does it affect my final output?

which means, what's the expected value of $||v - v_*||$, where $v_*$ is the output of the network obtained after the small changes in the weights described before?

Note that this has nothing to do with the learning process, it's just about the sensitivity/resistance of a neural network with respect to random noise injected in a weight matrix.


texts: How to discretize certain features with a feature set?

texts: I am working with typing data with timing features(unit: ms) and some of the features are based on the keyboard keyCodes(positive integers, range:[8, 222]). Currently, I use StandardScaler() by scikit-learn to scale all the features, so that my learning models do not overweigh the keyCode based features. I would like dicretize the keyCode based features and run the StandardScaler() for the timing features only. How can I go about this?


texts:
My question is on the number of units in an LSTM cell.

I've come across the following example which is a model for predicting a value in a series based on its 2 lag observations. I'm wondering why the number of units in the LSTM cell is 100 which is much higher than the number of features. In general, wouldn't be more logical to set the number of units to the number of input features? What is the advantage of having a number of units higher than the number of features? And what does it intuitively mean?

My code:

series = array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

n_features = 1

series = series.reshape((len(series), n_features))

n_input = 2

generator = TimeseriesGenerator(series, series, length=n_input, batch_size=8)

model = Sequential()

model.add(LSTM(100, activation='relu', input_shape=(n_input, n_features)))

model.add(Dense(1))

model.compile(optimizer='adam', loss='mse')


texts: How to use ML to forecast sales of a brand new product

texts:
I am working on a forecasting problem and came across this issue. How do I forecast sales of a brand new product? For example, a product has been introduced in the store and the store would like to know how the sales will look like.

I figured this would be similar to cold start problem in recommender systems. But still I would like to know if there are any standard approaches for such problems.


texts:
I am following this paper. You don't have to read it all as I'm about to explain the best I can. Consider the network that they have developed:

This network takes as an input a vector of 10 values. Input is fed to convolution layer. Conv layer has a total of 384 filters of three different types. By type here I mean filter size. 128 filter for each size. Sizes are 3,4 and 5. Note that these are all vectors and not matrices. Total of 384 feature vectors are produced as a result. These feature vectors are fed into Relu and then maxpool. Maxpool takes only one max value from the feature vector. After maxpool layer I have 384 vector and this is then fed into FC layer. FC layer has 384 neurons. And then output layer which has 3 neurons (for three classes) and a softmax applied to it.

Backpropogation:

To update weights of output layer, I have used following equation

Wnewij = Woldij - (learning_rate * (ActualValueOfThatNeuron - ExpectedValueOfThatNeuron)*Inputi

Where i is the input neuron in FC layer to j neuron in output layer.

To calculate gradient/error/loss to be propogated back (from FC layer neurons), I have used the following equation:

ErrorInputi = ( (ActualValueOfThatNeuron - ExpectedValueOfThatNeuron)*Wij)

(By input here I just mean a neuron in FC layer
Then to calculate the weights from input to FC layer, I use the same equation as stated above except I use ErrorInputij like this:

Wnewij = Woldij - (learning_rate * (ErrorInputij)*Inputi

And then I computed Loss/gradient/error to be propogated back the same way.
And now I am stuck at Maxpool (though Im not even sure about the above approach either). For the maxpool, gradient at max is one and for the rest zero. In this case, just considering one neuron, My vector might look something like this [0 loss 0 0 0 0]. Am I supposed to multiply it with input feature vector? How do I backpropogate error further down?

Complete newbie, please keep the mathematical equations to a minimum.


texts: Pandas - Sum of multiple specific columns

texts:
I created this script:
        import pandas as pd
        pd.set_option('display.min_rows', None)
        pd.set_option('display.max_columns', None)
        df = pd.read_excel('file.xlsx', sep=';', skiprows=6)
        df = df.drop(['Position', 'Swap'], axis=1)
        df.drop_duplicates(['Time'])
        df = df.groupby(['Symbol'])[['Profit', 'Volume', 'Commission']].sum()

        df['Comm. ratio'] = (df['Commission'] / df['Profit'])
        df['Comm. ratio'] = df['Comm. ratio'].round(2)
        df['Net profit'] = (df['Profit'] + df['Commission'])
        df.loc['Total'] = pd.Series(df['Commission'].sum(), index = ['Commission'])

    Symbol  Profit Volume Commission Comm. ratio Net profit

    BA
    FB
    NVDA
    US500
    USTEC
    Total                     -45

    df.loc['Total'] = pd.Series(df['Commission'].sum(), index = ['Commission'])

works on "commission" column, but I'd like to have a multiple column sum for "Profit, Commission, and Net profit" in the "Total" row.
I couldn't make it work.
Thanks!


texts: Setting input shape for an NLP task in R(Rstudio) using keras 1D convolution layer, when it expects 3 dimensional input (a tensor)

texts:
I am using R programming language and using Keras API to build a functional 1D CNN.

I have a matrix of my dataset of the following shape rows*features (6000*1024).

The input layer is set using the following code:

input_layer = layer_input(shape = 1024, batch_shape = c(nrow(train_matrix),1024), dtype = 'float64')

and then I am building a 1d conv layer as follows:

conv1 = input_layer %>% layer_conv_1d(filters = 32, kernel_size = 50, strides = 10, input_shape = 1024, batch_input_shape = list(NULL, 1024) ,dtype = 'float64', activation = 'relu' )

But I get the following error:

  Error in py_call_impl(callable, dots$args, dots$keywords) :
    ValueError: Input 0 is incompatible with layer conv1d: expected ndim=3, found ndim=2

I believe it is due to the fact that 1D cnn layer expects the input in the following form

  Input shape: 3D tensor with shape: (batch_size, steps, input_dim)

I understand that I have to reshape my data as (NULL, nrow(train_matrix), 1; as this has been suggested in various answer for the same issue arising for keras when used in Python.

If I am right,

what values should I provide to input layer
how should i reshape my training data?
does that mean I have to reshape the test data as well?

also if my understanding is wrong what should be done otherwise ?


texts: Box Jenkins, ADF and KPSS tests on a time series

texts:
this is my first question
I have the growth from Ecuadors GDP from 2000 to 2025 (Annually) and I have to predict Ecuadors GDP growth from 2026 to 2030 taking into account:

Ecuadors GDP Growth
Ecuadors Oil revenue
US GDP Growth.

I tried to follow Box-Jenkins methodology, so I did some Augmented Dickey Fuller tests and KPSS tests.
(where ecu_ts is Ecuadors GDP Growth)
I put my code and results here:(i use R)
adf_drift0<-ur.df(ecu_ts, type="drift", lags=5, selectlags="AIC")
#test statistic: -2.5454 | Critical Value (5%): tau2=-2.94 | Therefore I don't reject H0

adf_trend0<-ur.df(ecu_ts, type="trend", lags=5, selectlags="AIC")
#test statistic: -2.8878 | Critical Value (5%): tau3=-3.50 | Therefore I don't reject H0

adf_none0<-ur.df(ecu_ts, type="none", lags=5, selectlags="AIC")
#test statistic: -2.1466 | Critical Value (5%): tau1=-1.95 | Therefore I reject H0

And KPSS:
kpss_trend0<-ur.kpss(ecu_ts, type="tau", lags="long", use.lag=NULL)
#test statistic: 0.1232 | Critical Value (5%): 0.146 | Therefore I don't reject H0

kpss_cons0<-ur.kpss(ecu_ts, type="mu", lags="long", use.lag=NULL)
#test statistic: 0.2758 | Critical Value (5%): 0.463 | Therefore I don't reject H0

What should I do? Should I differentiate the series once more? (d=1) to reject H0 in the ADF test and don't reject H0 in the KPSS test?
The series looks like this: (Let me know if you need the data set)


texts: Understand the equations of quantile regression forest (Meinshausen)?

texts:
I am trying to implement a quantile regression forest (https://www.jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf).
But, I have some difficulties to understand how the quantiles are computed. I will try to summarize the part of interest in order to then explain exactly what I don't understand.
Let be $n$ independent observations $(X_i, Y_i)$. A tree $T$ parametrized with a realization $\theta$ of a random variable $\Theta$ is denoted by $T(\theta)$.

Grow $k$ trees $T(\theta_t)$, $t = 1, . . . , k$, as in random forests. However, for every leaf of every tree, take note of all observations in this leaf, not just their average.
For a given $X = x$, drop $x$ down all trees. Compute the weight $\omega_i(x, \theta_t)$ of observation $i \in \{1, . . . , n\}$ for every tree as in (4). Compute weight $\omega_i(x)$ for every observation $i \in \{1, . . . , n\}$ as an average over $\omega_i(x, \theta_t)$, $t = 1, . . . , k$, as in (5).
Compute the estimate of the distribution function as in (6) for all $y \in \mathbb{R}$.

Where the equations (4), (5), (6) are given below.
$$ \omega_i(x, \theta_t) = \frac{ 1 \{ X_i \in R(x, \theta_t) \} }{\text{#} \{ j : X_j \in R(x, \theta_t) \} } \ \ \ (4)$$
$$ \omega_i(x) = k^{-1} \sum_{t=1}^k \omega_i(x, \theta_t) \ \ \ \ (5)$$
$$ \hat{F}(y|X=x) = \sum_{i=1}^n \omega_i (x) 1\{Y_i \leq y\} \ \ \ (6) $$
Where $R(x, \theta_t)$ denotes the rectangular area corresponding to the unique leaf of the tree $T(\theta_t)$ that $x$ belongs to.
I can compute (4) and (5) but I don't understand how to compute (6) and then estimate quantiles. I would also add that I don't know where all observations in leaves (first step of the algorithm) are used.
Can someone give some elements to understand this algorithm ? Any help would be appreciated.


texts: In Spark, what's the role of each of Cluster manager and Spark Context?

texts:
What's the role of a cluster manager and the role spark context (Driver program) in terms of managing worker nodes.
How do they communicate with each other (and with the worker nodes) and when they do it and what purpose ?
Thanks for the enlightenment.


texts: What is the difference between Linear SVM and SVM with linear kernel?

texts:
I'm wondering whether there is a difference between Linear SVM and SVM with a linear kernel. Or is a linear SVM just a SVM with a linear kernel?

If so, what is the difference between the two variables linear_svm and linear_kernel in the following code.

from sklearn import svm
linear_svm = svm.LinearSVC(C=1).fit(X_train, y_train)
linear_kernel_svm=svm.SVC(kernel='linear', C=1).fit(X_train, y_train)


texts: Reinforcement Learning reward is not converging to zero

texts:
I am new to reinforcement learning. I am trying to build an RL algorithm which will predict cloud hardware capacity required for an org in terms of compute, storage, memory. The algorithm which i have developed uses boltzmann equation and a DQN. My reward is a simple calculation based on how much the prediction is off from utilisation threshold provided by user (For example if the user provides that he wants the compute utilisation to be 80%, my reward will be the difference between optimum value and predicted value). During training, the RL uses a random logic to predict next set of values.
The challenge i am facing is that the reward is never converging to zero is certain episodes even when the score reaches to 20000. Is it normal the reward might not converge to zero in certain episodes?
Also, do i need to reward mechanism during test? I think the reward mechanism is not needed for test as i would be saving my trained model and using the weights file during test.


texts: Performance measure: Why is it called recall and sensitivity?

texts:
precision is the fraction of retrieved instances that are relevant, while recall (also known as sensitivity) is the fraction of relevant instances that are retrieved.

I know their meaning but I don't know why it is called recall? I am not a native-speaker of English. I know recall means remember, then I don't know the relevance of this meaning to this concept! maybe coverage was better because it shows how many instances were covered...or any other term.

Moreover sensitivity is also insensible to me!

Could you please help me to associate these words to the concept and have a sense of them?


texts: How to pass linear regression weights to Xgboost regressor?

texts:
I'm trying to build an xgboost regressor or a catboost regressor for a task. I have a working linear regression model. I also trained an xgboost regressor model for the task but it was worse than the linear regression model. I am wondering if there is a way to pass the linear  regression weights (model parameters) as an initial set of parameters to the xgboost (or catboost) model to ensure performance gain?

E.g. if $w0*x0 + w1*x1 + w2*x2 + w3*x3 = y$ is the linear regression model, is there any way to tell xgboost to start by the same equation (and get better while training)?


texts: Tensorflow CNN not predicting correctly for a well trained model

texts:
So I have built a CNN and trained it to around 95% accuracy on training data, and 90% on testing data. The issue is, when I save this model, load it in again, and predict, it always predicts 1 or close to 1. I even used training data.

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import cv2
import os
import random
from keras.models import Sequential
from keras.layers import Conv2D,Dense,Flatten,Dropout,MaxPooling2D
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import ReduceLROnPlateau

image_height = 150
image_width = 150
batch_size = 10
no_of_epochs  = 10

model = Sequential()
model.add(Conv2D(32,(3,3),input_shape=(image_height,image_width,3),activation='relu'))
model.add(Conv2D(32,(3,3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(units=128,activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(units=1,activation='sigmoid'))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale=1./255,
                                   rotation_range=15,
                                   shear_range=0.2,
                                   zoom_range=0.2
                                   )

test_datagen = ImageDataGenerator(rescale=1./255)

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import cv2
import os
import random
from keras.models import Sequential
from keras.layers import Conv2D,Dense,Flatten,Dropout,MaxPooling2D
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import ReduceLROnPlateau
from keras import *
from keras.preprocessing.image import ImageDataGenerator
from keras.preprocessing import image

training_set = train_datagen.flow_from_directory('/content/hackmed2020/train',
                                                 target_size=(image_width, image_height),
                                                 batch_size=batch_size,
                                                 class_mode='binary')

test_set = test_datagen.flow_from_directory('/content/hackmed2020/test',
                                            target_size=(image_width, image_height),
                                            batch_size=batch_size,
                                            class_mode='binary')

# Updated part --->
val_set = test_datagen.flow_from_directory('/content/hackmed2020/val',
                                            target_size=(image_width, image_height),
                                            batch_size=1,
                                            shuffle=False,
                                            class_mode='binary')

reduce_learning_rate = ReduceLROnPlateau(monitor='loss',
                                         factor=0.1,
                                         patience=2,
                                         cooldown=2,
                                         min_lr=0.00001,
                                         verbose=1)

callbacks = [reduce_learning_rate]

history = model.fit_generator(training_set,
                    steps_per_epoch=5216//batch_size,
                    epochs=no_of_epochs,
                    validation_data=test_set,
                    validation_steps=624//batch_size,
                    callbacks=callbacks
                   )

model.save('path_to_my_model.h5')

model = models.load_model('path_to_my_model85.h5')
path_to_file = "/content/Pneumonia1.jpeg"

img = image.load_img(path_to_file, target_size=(150,150))
img = image.img_to_array(img)
img = np.expand_dims(img,axis=0)
prediction = model.predict(img)


texts: Google OR-Tools - Routing - Penalties - Adding Different Penalty to Different Location (Python)

texts:
I am using Google's OR-Tools for route optimisation. References can be found here.

I am performing an optimisation where certain pick-up locations are dropped based on a penalty at each location. The documentation gives an example and states: "In this context, a disjunction is simply a variable that the solver uses to decide whether to include a given location in the solution. In this example, the method adds the same penalty to each location, but in general you can add different penalties to different locations.", and provides the following chunk of code as an example of adding a constant penalty to each location:

penalty = 1000
for node in range(1, len(data['distance_matrix'])):
    routing.AddDisjunction([manager.NodeToIndex(node)], penalty)

where data is a dict containing a n x n distance matrix (distance_matrix). (This code runs and returns a sensible outcome).

Now, I would like to vary the penalty at each location, and I've used the following code to do so:

for node in range(1, len(data['distance_matrix'])):
    routing.AddDisjunction([manager.NodeToIndex(node)], data['penalty'][manager.NodeToIndex(node)])

where penalty is an array of length n containing penalties for each location. (In these examples, the depot is assumed to be at the first location and is penalty-free; range starts at 1).

When I run my code, I get the following error message:

TypeError: Wrong number or type of arguments for overloaded function 'RoutingModel_AddDisjunction'.
Possible C/C++ prototypes are:
operations_research::RoutingModel::AddDisjunction(std::vector< int64 > const &,int64,int64)
operations_research::RoutingModel::AddDisjunction(std::vector< int64 > const &,int64)
operations_research::RoutingModel::AddDisjunction(std::vector< int64 > const &)

Question: What am I doing wrong? How do I go about adding different penalties to different locations? Could someone please translate the C/C++ into Python?


texts: Visualizing results of a classification problem, excluding confusion matrices?

texts:
A classification system I built is going to go into production soon (it'll be part of a larger dashboard), and I'm looking for ways to better visualize and convey to business folks the results of a classification.

Basically, given "old" data on which the model was trained, I predict the classes of "new" data, with the goal being to show whether the class distributions for the new data are statistically indifferent from the class distributions observed in the old data. So, if there are three classes A, B, and C in the old data, with proportions of 50%, 30%, and 20%, respectively, I compare the classification distribution for the new data with those original, observed proportions.

Outside of a confusion matrix (which I think is probably inappropriate for most dashboard users), how else can I effectively present these results? I was thinking of a bar chart like this:


texts: How to interpret high loss value from model.evaluate() on test data

texts:
I'm collecting some metrics for my model's performance using:
# Evaluate the model on the test data using `evaluate`
print("Evaluate on test data")
results = model.evaluate(test_images)
print("test loss, test acc:", results)

I have an very high loss in this case:
test loss, test acc: [230, 0.769]

How do I interpret these results, is it the sum of errors made during the test data evaluation?


texts: Calculating RMSE AND R-squared from the confusion matrix

texts:
I have my confusion matrix as C.mat

8263    20    39
2       3826  14
43       7    4431

My predicted class labels are Ypred and actual labels are Ytest. Ypred size is 16000*1 and Ytest 16000*1.

I am trying to calculate the R-squared and RMSE. Is there a way to directly calculate RMSE and R-squared from the confusion matrix?

I tried this:

RMSE = sqrt(immse(Ypred, Ytest))

However, it didn't work.

I can use  either R or Matlab.

Any advice will be appreciated!


texts: Given a comment or a sentence and a list of profane words, How do I write a program to print the degree of profanity in that sentence?


texts: Why crossvalind and cvpartition results are different?

texts:
I did a Test on my data set with "cvpartition" and "crossvalind" functions (in MATLAB) with the same parameters and data set, but why the error results are different and which one is correct?

Codes are from MATLAB help just data set has been replaced.

crossvalind:

k=10;
indices = crossvalind('Kfold',lables,k);
cp = classperf(lables);
for i = 1:k
    test = (indices == i);
    train = ~test;
    class = classify(dataset(test,:),dataset(train,:),lables(train,:));
    classperf(cp,class,test);
end

cp.ErrorRate

Error Rate is: 0.3322

and cvpartition:

CVO = cvpartition(lables,'k',10);
err = zeros(CVO.NumTestSets,1);
 for i = 1:CVO.NumTestSets
     trainIdx = CVO.training(i);
     testIdx = CVO.test(i);
     ytest = classify(dataset(testIdx,:),dataset(trainIdx,:),lables(trainIdx,:));
     err(i) = sum(~strcmp(ytest,lables(testIdx)));
end
ErrorRate = sum(err)/sum(CVO.TestSize)

Error Rate is: 0.0342


texts: Is there any alternative to L-BFGS-B algorithm for hyperparameter optimization in Scikit learn?

texts:
The Gaussian process regression can be computed in scikit learn using an object of class GaussianProcessRegressor as:

 gp= GaussianProcessRegressor(alpha=1e-10, copy_X_train=True,
    kernel=1**2 + Matern(length_scale=2, nu=1.5) + WhiteKernel(noise_level=1),
    n_restarts_optimizer=0, normalize_y=False,
    optimizer='fmin_l_bfgs_b', random_state=None)

Here, we see that the L-BFGS-B algorithm has been used to optimize the hyperparameters. Is there any alternative (for example trust-region-reflective algorithm) to this algorithm available in sklearn?

EDIT: It provides some Constrained multivariate methods for optimization. Is there such functions available for other methods like trust-region-reflective algorithm and swarm intelligence (eg. Firefly algorithm)?


texts: Best learning automata reinforcement schema for solving grid world,help me

texts:
I have a gridworld puzzel , an agent and Target , I want find best path for reaching Target by agent.

gridworld example(with S as start point, G as goal point and black cells as cliffs):


texts: How to obtain the predictions of SVM model on single input?

texts:
So, I am trying to build a Spam detection model. It is trained on a dataset consisting of about 3500 messages. I used SVM to build a model. But, if I now wish to find out whether a message is spam or not for a single line of input message, how would I do it? I tried

testmessage = ['Discount ! Free ! guarantee']
x = vectorizer.transform(testmessage)
y = svm.predict(x)

But I am getting some error like :

  ValueError: X.shape[1] = 6 should be equal to 8272, the number of
  features at training time

import pandas as pd
import numpy as np
import nltk

data = pd.read_csv("/media/piyush/Disk/Code practice/SPAM_Detction/spam.csv",encoding ='latin-1')
data = data.drop(["Unnamed: 2","Unnamed: 3","Unnamed: 4"], axis = 1)

# data.loc[data.v1 == "spam", "v1"] = int(1)
# data.loc[data.v1 == "ham", "v1"] = int(0)

data = data.rename(columns = {'v1': 'label', 'v2': 'text'})

def message_lower(msg):
    msg = msg.lower()
    return msg

from nltk import stem
from nltk.corpus import stopwords
stemmer = stem.SnowballStemmer('english')
stopwords = set(stopwords.words('english'))

def alternative_review_messages(msg):
    # converting messages to lowercase
    msg = msg.lower()
    # removing stopwords
    msg = [word for word in msg.split() if word not in stopwords]
    # using a stemmer
    msg = " ".join([stemmer.stem(word) for word in msg])
    return msg

data['text'] = data['text'].apply(message_lower)

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test =  train_test_split(data['text'],data['label'], test_size=0.1, random_state= 1)
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(X_train)

from sklearn.svm import SVC
svm = SVC(C = 3000)
svm.fit(X_train,y_train)
X_test = vectorizer.transform((X_test)) y_pred = svm.predict(X_test)

from sklearn.metrics import confusion_matrix

print(confusion_matrix(y_test,y_pred))


texts: What is meaning of different symbols in image?

texts: I am learning about Deep Generative Models, tutorials all over the places use symbols and no one actually explaining the meaning of it. Can you please suggest the meaning of these terms?


texts: Stateful loss function and online regression

texts:
Working in tensorflow + keras I'm trying to define a custom loss function. N.B. I'm more interested in the value of the loss rather than the actual value of the predictions (this will be used for anomaly detection).

My network outputs predictions for the next n-steps ahead.

At time t, I receive an input which is the current observation (y_t), I have a history of x time steps into the past. I wish to compare the value y_t, to those predicted for y_t at each of t-1,...t-n.

To do this I envision a 'triangular' array of arrays ranging in length from 1 to n. I add the values predicted to the end their respective arrays and pop the first value from each to produce my y_pred tensor. y_t is then duplicated into a tensor of equal length (y_true) to be used in computing the loss at time t (probably MSE or the like).

The difficulty I am encountering is how to retain the states of these prediction arrays between batches.

The idea is for this to be used in an online fashion where a small update is made to the network at each time-step (batch size 1) without requiring significant delay between receiving an observation and reporting how accurate the model is (as would be required by saving up predictions for the n-steps and calculating after all have been received).