texts:
I trained a simple Handwriting model with Tensorflow and MNIST:
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train.astype('float32')/255
x_test =  x_test.astype('float32')/255
model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28,28)),
    keras.layers.Dense(100,activation = 'relu'),
    keras.layers.Dense(100,activation = 'sigmoid'),
    keras.layers.Dense(10,activation = 'sigmoid'),
])

As you can see, I flatted my first layer into 784 px One-dimensional array.
Now on paper I wrote a number :

After changing it's scale into 28*28 with image editor(GIMP), I loaded my image into my code:
img_width, img_height = 28, 28
img = image.load_img('rgb_seven.jpeg', target_size=(img_width, img_height))
x = image.img_to_array(img)

This is x result:
array([[[167., 170., 179.],
        [168., 171., 180.],
        [168., 171., 180.],
        ...,
        [174., 175., 180.],
        [174., 175., 180.],
        [167., 170., 175.]],

       [[173., 176., 185.],
        [172., 175., 184.],
        [166., 169., 178.],

and :
images = np.vstack(x)
images

And it's images result is :
array([[167., 170., 179.],
       [168., 171., 180.],
       [168., 171., 180.],
       ...,
       [166., 173., 179.],
       [166., 173., 179.],
       [166., 173., 179.]], dtype=float32)

Before to predict I have to do flat my images so I do this:
x_images_flattened = images.reshape(len(images),28*28)

But I got error:
ValueError                                Traceback (most recent call last)
<ipython-input-30-db6c299f19c6> in <module>
      3 images = np.vstack(x)
      4 # images
----> 5 x_images_flattened = images.reshape(len(images),28*28)

ValueError: cannot reshape array of size 2352 into shape (784,784)

Why I got cannot reshape array of size 2352 into shape (784,784) my image has 28*28 size.
And how can I predict that?


texts:
Can anyone help me about cost function in linear regression. As from the below plot
we have actual values and predicted values and I assumed the answer as zero but it actually is 14/6?

Can anyone help out please?


texts: Improving Naive Bayes accuracy for text classification

texts:
I am performing document (text) classification on the category of websites, and use the website content (tokenized, stemmed and lowercased).

My problem is that I have an over-represented category which has vastly more data points than any other (roughly 70% or 4000~ of my data points are of his one category, while about 20 other categories make up the last 30%, some of which have fewer than 50 data points).

My first question:

What could I do to improve the accuracy of my classifier in this case of sparse data for some of the labels? Should I simply discard a certain proportion of the data points in the category which is over-represented? Should I use something other than Gaussian Naive Bayes with tf-idf?

My second question:

After I perform the classification, I save the tfidf vector as well as the classifier to disk. However, when I re-rerun the classification on the same data, I sometimes get different results from what I initially got (for example, if previously a data point was classified as "Entertainment", it might receive "News" now). Is this indicative of an error in my implementation, or expected?


texts:
I am receiving the following error. I can't seem to figure out the exact change I need to make to my code to get solve this problem. I feel like it should be straightforward but I'm banging my head against the computer.
Here is the error message:
ValueError: Found input variables with inconsistent numbers of samples: [10000, 11200]
Here is my code:
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.ensemble import GradientBoostingClassifier

train_data = pd.read_csv("training/all.lst",sep=' ')
y_train = train_data["status"]
train_data.drop(labels="status", axis=1, inplace=True)

X_train = train_data.values[0:10000]
X_test = train_data.values[10000:]

scaler = MinMaxScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

state = 12
test_size = 0.30

X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=test_size, random_state=state)

lr_list = [0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1]

for learning_rate in lr_list:
    gb_clf = GradientBoostingClassifier(n_estimators=20, learning_rate=learning_rate, max_features=2, max_depth=2, random_state=0)
    gb_clf.fit(X_train, y_train)

    print("Learning rate: ", learning_rate)
    print("Accuracy score (training): {0:.3f}".format(gb_clf.score(X_train, y_train)))
    print("Accuracy score (validation): {0:.3f}".format(gb_clf.score(X_val, y_val)))
```


texts: regression network that will be optimised on a subset of the data

texts:
I am trying to optimise my network that is trying to perform regression. Currently, the dataset is dominated by values in a certain range. It is very good at predicting these values. I want to try to find a way to make it focus on the values outside of this bulk, these values outside the bulk are much higher. We thought about splitting the dataset into parts, and training multiple networks, but we really want the regression to be done by a single network.
An idea that we had was to adapt how the loss function calculated loss as we are training. So, after a certain number of epochs passed, or after a certain accuracy was achieved, we would only pass the loss function values that were being very poorly predicted. The motivation was that this may improve the performance of the network as training would have to focus on these points.
I have not found much literature, or examples, trying something like this out. Although, it seems like it could work. Naively, it seems like it's in the same vein as weighting a loss function, or when you perform transfer learning and retrain just the final few layers of the network to make it specific to your problem.
My questions:

Generally, how does this sound to people? Does anyone have any opinions on it or ideas on how to improve it?
Is there any work been done doing something similar that people know of?


texts: Not Access to Confusion Matrix in SVM.SVC.score Scikit-learn Python

texts:
I used SVM.SVC function to classify. But when I wanted to calculate the weighted and unweighted average accuracy I couldn't access the confusion matrix. Because of svm.SVC.score only provides a classifier accuracy percentage. How can I calculate WAR and UAR?

you can find part of my script below :

'''

    scaler = StandardScaler()
    scaler.fit(trainX)
    trainXsc = scaler.transform(trainX)
    testXsc = scaler.transform(testX)

    pca = KernelPCA(n_components=j, kernel="sigmoid", random_state=1)

    pca.fit(trainXsc)     # fit pca kernel with train data

    trainXtr    = pca.transform(trainXsc) # transform FV with PCA and dimension reduction
    testXtr     = pca.transform(testXsc)

    svmObject   = svm.SVC(C=2.0, kernel='rbf', degree=3, gamma='scale', coef0=0.0, shrinking=True,
                  probability=False, tol=0.001, cache_size=200, class_weight=None,
                  verbose=False, max_iter=-1, decision_function_shape='ovo', random_state=None)
                                        # SVM Kernel Function

    svmObject.fit(trainXtr, trainY)      # train SVM kernel with train FV

    result = svmObject.score(testXtr, testY)

'''


texts: Padding the sentences is consuming huge memory

texts:
I prepared a lstm model using tensorflow which has a max_sequence_length of 5000 and I'm padding the small sentences with 0's. I then deployed and tested the model with multiple calls at a time. I then encountered a memory error because the max_sequence_length is consuming a lot of memory.

Is there is any alternative to writing the model more efficiently using some thing as an alternative to padding?


texts: Parameters in GridSearchCV in scikit-learn

texts:
I am trying to build a model in scikit-learn. I used RandomForestClassifier as my method for classification. In order to improve the score and efficiency of my model, I thought about using GridSearchCV.

Here is the code:

import pandas as pd
import numpy as np
from sklearn.cross_validation import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score,roc_auc_score
from sklearn.grid_search import GridSearchCV

..................................... ## code for cleaning data

X_train, X_test, y_train, y_test = train_test_split(train,output, test_size=0.2, random_state =7)

# In[18]:

clf = RandomForestClassifier(n_estimators =100)
param_grid = {'max_depth' : [None, 10,20],
              'max_features' : ['auto',None],
              'n_estimators' :[100,200,300],
              'random_state': 7}
## This line is throwing the error shown below
validator = GridSearchCV(clf, param_grid= param_grid)
vaildiator.fit(X_train,y_train)

The error being thrown by my code is:

ValueError     Traceback (most recent call

last)
<ipython-input-22-3711af477b0c> in <module>()
      3          "max_depth" : [5,10,50],
      4          "random_state" : 7}
----> 5 grid = GridSearchCV(clf, param_grid=param, n_jobs=1)
      6 grid.fit(X_train,y_train)

C:\Anaconda3\envs\DeepLearning\lib\site-packages\sklearn\grid_search.py in __init__(self, estimator, param_grid, scoring, fit_params, n_jobs, iid, refit, cv, verbose, pre_dispatch, error_score)
    785             refit, cv, verbose, pre_dispatch, error_score)
    786         self.param_grid = param_grid
--> 787         _check_param_grid(param_grid)
    788
    789     def fit(self, X, y=None):

C:\Anaconda3\envs\DeepLearning\lib\site-packages\sklearn\grid_search.py in _check_param_grid(param_grid)
    326             check = [isinstance(v, k) for k in (list, tuple, np.ndarray)]
    327             if True not in check:
--> 328                 raise ValueError("Parameter values should be a list.")
    329
    330             if len(v) == 0:

ValueError: Parameter values should be a list.

Please help me in figuring out the above error and why is this happening?


texts: transform a supervised neural network to reinforcement learning?

texts:
I have a functional LSTM model that works with an acceptable performance. How can I now convert this supervised model to a reinforcement learning model for improving the performane? Is there any example about how to transform a supervised model to a reinforcement learning model?

Details: I have a multi-input multi-output system (since I can not share the actual problem lets assume weather forcast as example) where I need to predict the output (e.g temperature, wind speed, etc) in real time. I have a large dataset and I trianed a supervised learning model that can do the predictions pretty well in real time.

The problem is that sometimes there is big deviation between predictions and actual values. This means that probably there was a new trend in the inputs that has never been in the data sets. For such cases, I would like to gradually enhance my model predictions.

Does it mean that I have to train again my model with the old dataset and the new data or I can simply have a RL mechansim to gradually improve the model?


texts: Can be possible to solve Rubik's cube using DQN?

texts:
I'm trying to solve Rubik's cube using deep learning and I came across with DQN, so I decided to give it a try. I developed all the code and started training bu I got this results:

Loss goes up and test never get better results. I have tried to change learning rate, epsilon greedy decay, reducing scramble moves to one but it still can't solve it with just one move. That's why I would like to know if it just impossible to do it this way or I could miscalculating some values.


texts: How to plot the hyperplane for multiclass target variable in SVM?

texts:
Please suggest me how can I draw the hyperplane for a 7 class target variable.

I'm doing my project in python 3.7 in Spyder.


texts: How to fit integer output in neural network

texts:
I have a data set that contains 104 input features and 96 output values to be predicted, the input features are floating values normalized to [-1,1], and each out put value would be one integer among [-1,0,1].

I am basically new to data science and machine learning, and I need to know what kind of neural network model would fit best for this kind of data. I have googled this for a quite long time, can you guys help me ? Thank you!

我的每一条训练数据具有104个[-1,1]之间的浮点型输入量和96个整数型输出量，每个输出值为[-1,0,1]之中的一个整数。
我是机器学习的新手，我想知道应该采用何种神经网络模型来解决此问题，在post本问题之前我已经自行搜索了很久，希望大家能给我一些帮助，谢谢！


texts: Feature engineering one step at a time or in bunches?

texts:
Currently, I'm working on my very first classification project. If you want to know what dataset I'm working with, think "playing stairway to heaven in your local guitar store", and it will probably come to you. But whatever, I'm a newb and I have a question about best practices:

During feature extraction, do you:

extract one feature at a time, test the model performance and then
choose to keep it or discard it?

Or

extract a bunch of features at once and just throw them all
at the model because doing one at a time takes too long?

Currently, I've been doing one feature at a time, but my notebook is getting suuuper long (lots of failed strategies and only one feature has added value). I am running into the same issues expressed in this question.

His/her question is different than mine, but similar in nature.


texts: I am creating an image dataset of objects. We have 15 classes of objects and need to provide a color to the objects also. What colors(aroun 7-10) should we use? I am thinking of using VIBGYOR, but not able to find any justification as to why we should be using VIBGYOR and not anything else. Is there any standard practice? Or do people who create such computer vision or image processing datasets have some justification on which colors to use? Some links to research/papers or datasets with justification would be great!


texts: Why do we use a Gaussian kernel as a similarity metric?

texts: In graph-based clustering, why is it preferred to use the Gaussian kernel rather than the distance between two points as the similarity metric?


texts: Relationship between two continuous variables in time series data

texts:
I have a dataset that collects daily data based on transactions between two entities.
I wish to find the strength, direction, and kind of relationship between two continuous variables i.e. Number of transactions (No_of_transactions) and Error counts (Error_Counts) over the period of approximately two years.
Error counts are of degree 10²-10⁵ and the number of transactions in 10¹-10³.
I'm pretty sure the relationship is nonlinear because when I draw the scatterplot it is more towards inverse relationship (rightly so) and I believe a simple corr() function won't help much.


texts: What are some best papers on gradient descent for NN implementation?

texts:
I'm trying to implement GD for standard task of NN training :) The best papers for practioneer I've founded so far are:

1) "Efficient BackProp" by Yann LeCun et al.

2) "Stochastic Gradient Descent Tricks" by Leon Bottou

Are there some other must read papers on this topic?

Thank you!


texts: Open source data science projects to contribute

texts:
Contribution into open source projects is typically a good way to get some practice for newbies, and try a new area for experienced data scientists and analysts.

Which projects do you contribute? Please provide some intro + link on Github.


texts: Natural Language Processing: Identifying Words That Are Out of Place?

texts:
I want to make it easier to manually (or potentially automatically) correct AI transcription. I've noticed that one significant error that can be very easily picked up by human transcriptionists is words that are just out of place in a sentence.

I am wondering if there is a system (algorithmic or something complete like a node.js package) that can score words for out-of-placeness in a group of paragraphs. I imagine it would involve a lot of algorithmic learning.


texts: Does GPT-2 has pre trained for sentiment analysis?

texts: I tried sentiment analysis with 345M model of GPT-2. But it took a long time to train. So is there any other GPT-2 model available for sentiment analysis?


texts: Does torch.cat work with backpropagation?

texts:
I was wondering if it was okay to use torch.cat within my forward function. I am doing so because I want the first two columns of my input to skip the middle hidden layers and go directly to the final layer.

Here is my code: you can see that I use torch.cat at the last moment to make xcat.

Does the gradient propagate back? or does the torch.cat cover up what happened to my hidden variables?

class LinearRegressionForce(nn.Module):

    def __init__(self, focus_input_size, rest_input_size, hidden_size_1, hidden_size_2, output_size):
        super(LinearRegressionForce, self).__init__()

        self.in1 = nn.Linear(rest_input_size, hidden_size_1)
        self.middle1 = nn.Linear(hidden_size_1,hidden_size_2)
        self.out4 = nn.Linear(focus_input_size + hidden_size_2,output_size)

    def forward(self, inputs):
        focus_inputs = inputs[:,0:focus_input_size]
        rest_inputs = inputs[:,focus_input_size:(rest_input_size+focus_input_size)]
        x = self.in1(rest_inputs).clamp(min=0)
        x = self.middle1(x).clamp(min=0)
        xcat = torch.cat((focus_inputs,x),1)
        out = self.out4(xcat).clamp(min=0)
        return out

I call it like so:

rest_inputs = Variable(torch.from_numpy(rest_x_train))
focus_x_train_ones = np.concatenate((focus_x_train, np.ones((n,1))), axis=1)
focus_inputs =  Variable(torch.from_numpy(focus_x_train_ones)).float()
inputs = torch.cat((focus_inputs,rest_inputs),1)

predicted = model(inputs).data.numpy()


texts: Is there a way to combine both ties (nondirected edges) and wins/losses (directed edges) in a single social network?

texts:
I'm currently building social networks for small colonies of animals which I've observed, with the aim of comparing changes in social network structure in response to changes in certain environmental variables. Individuals in these colonies undergo dyadic dominance interactions in which one individual attempts to assert dominance over another. The result of a dominance interaction can be either win/loss (i.e. one individual successfully dominates another) or it can be a tie (neither individual successfully asserts dominance).

I want the nodes of my social networks to represents individuals, and the edges to represent these dominance interactions. I've been using the igraph package for R which works well in many ways, but I have an annoying issue: igraph allows the user to model win/loss interactions using directional edges, but it doesn't allow for ties to be modeled at the same time. That is, one cannot combine directional and non-directional edges in the same network.

Is there a standard way to deal with ties in this situation? I've considered modeling a tie as 'half a win' to each participating individual, but this seems wrong since a tie is neither a win nor a loss. The other possibility I've considered is just ignoring ties, but this is also unsatisfactory since there's reason to think that dominance interactions that conclude in ties are still significant (e.g. they still convey information between two individuals about relative status and dominance rank).


texts: What is the objective function that Latent Dirichlet Allocation (LDA) minimizes?

texts:
Latent Dirichlet Allocation (LDA) is a generative model which produces a list of topics. Each topic is represented by a distribution over words.
Questions:

What is the objective function that it tries to minimize? Can someone
explain it in simpler way? The mathematical formula and the intuition
behind it?

Secondly, After building the LDA model how we validate the model? I
have heard perplexity as one metric, but what is perplexity about?

Does bias, Variance problem also exists in LDA? If yes, when it can
occur?


texts: How to determine if a string is 'food' or 'drink'?

texts:
My data exists of several restaurant menus (in text form). I would like to detect the category of an item on the menu which can be 'food' or 'drink'.

How do I call this kind of machine learning? Text classification?

I know javascript isn't ideal for this kind of thing but I would like to use brain.js or tensorflowjs to implement a proof of concept.

Thanks in advance!


texts: Comparing AUC, logloss and accuracy scores between models

texts:
I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem:

  accuracy logloss   AUC
1   19%      0.45   0.54
2   67%      0.62   0.67
3   66%      0.63   0.68
4   67%      0.62   0.66
5   63%      0.61   0.66
6   65%      0.68   0.42

I have the following questions:

How can model 1 be the best in terms of logloss (the logloss is the closest to 0) since it performs the worst (in terms of accuracy). What does that mean ?
How come does model 6 have lower AUC score than e.g. model 5, when model 6 has better accuracy. What does that mean ?
Is there a way to say which of these 6 models is the best ?


texts: Ingredients, Recipes and recipe ratings. I would like to predict the rating based on combination of ingredients

texts:
I would really appreciate some help on the first steps to my problem, suggestions of modeling techniques i could use or relevant research (i could not find any).

I have a list of ingredients (150 in total) and a list of recipes (100 or so) with data on which combination of ingredients are present in each recipe (on average 4-5 ingredients per recipe).

My output variables are 4 metrics of ratings of those recipes (taste, smell, look, and texture) for each of those 100 recipes from a scale of 1-10 for each.

My aim is to predict the 4 metrics based on a new combination of ingredients (picked from the list) that i specify.

What modeling technique should I look at to help solve this prediction problem?


texts:
I am trying out the Sobel vertical operator to identify vertical edges in a picture. In each image there is one rectangle but the difference is that in 1 image the 2 vertical lines in the rectangle is 1 pixel apart whereas in the other it is more than 1 pixel apart as shown below.

Now when i run the sobel vertical filter my expectation is that in both cases i should get 2 edges. But to my surprise it returns 2 edges for the bigger rectangle but only one edge for the smaller rectangle as shown below.

My Python code is as follows.

from PIL import Image
import numpy as np
im = Image.open('Test.png').convert('L')
im.save('greyscale.png')
pix_val = np.array(list(im.getdata())).reshape(81,91)
for i in range(12,14):
    for j in range(12,78):
        pix_val[i,j]=255
for i in range(63,65):
    for j in range(12,78):
        pix_val[i,j]=255
for i in range(14,63):
    for j in range(12,14):
        pix_val[i,j]=255
for i in range(14,63):
    for j in range(76,78):
        pix_val[i,j]=255
Oim1=Image.fromarray(pix_val.astype(np.uint8))
Oim1.save('out1.bmp')
pix_val2=np.delete(pix_val,range(16,75), 1)
Oim2=Image.fromarray(pix_val2.astype(np.uint8))
Oim2.save('out2.bmp')
from scipy import ndimage
dy1 = ndimage.sobel(Oim1, 1)
Oim3=Image.fromarray(dy1.astype(np.uint8))
Oim3.save('Sobel1.bmp')
dy2 = ndimage.sobel(Oim2, 1)
Oim4=Image.fromarray(dy2.astype(np.uint8))
Oim4.save('Sobel2.bmp')

Is this a limitation of sobel operator that it cant detect edges that are very near to each other?


texts: Multiple Seeds Content Based filtering Recommender System

texts: I want to create a content based filtering recommender system which has multiple seeds. All that I have read about is having an initial seed from which the recommendations should be similar to. This is particularly helpful in situations where you want to create movie recommendations that should be interesting to a group of people. So I need a query seed which is a single row of movie features to feed to the recommender. So how do I create a query seed based from multiple rows of movie features?


texts: Multivariate Time Series Binary Classification

texts:
I have continuous (time series) data. This data is multivariate. Each feature can be represented as time series (they are all calculated on a daily basis). Here is an example.

Days    F1  F2  F3  F4  F5  Target
Day 1   10  1   0.1 100 -10 1
Day 2   20  2   0.2 200 -20 1
Day 3   30  3   0.3 300 -30 0
Day 4   40  4   0.4 400 -40 1
Day 5   50  5   0.5 500 -50 1
Day 6   60  6   0.6 600 -60 1
Day 7   70  7   0.7 700 -70 0
Day 8   80  8   0.8 800 -80 0

F1, F2, .. F5 are my features and Target is my binary classes. If I use a window size of 3, I can convert my features into time-series data. Then, I will have [10,20,30] for feat_1, [1,2,3] for feat_2 and so on. With the window size of 3, I have 5 feats * 3 window_size, a total of 15 features if written in the same vector.

The problem with this method is putting them into the same vector might cause some problems since the feature values are different

Example of multivariate time series (15 features in 1 network):

[10, 20, 30, 1, 2, 3, 0.1, 0.2, 0.3, 100, 200, 300, -10, -20, -30]
[20, 30, 40, 2, 3, 4, 0.2, 0.3, 0.4, 200, 300, 400, -20, -30, -40]
....
[60, 70, 80, 6, 7, 8, 0.6, 0.7, 0.8, 600, 700, 800, -60, -70, -80]

The other option is to create separate time series network (RNNs mostly, LSTM or CNN or their combination) for each of the features with the same target and then combine their results. In this scenario, I have 5 different networks and all of them are univariate time series binary prediction.

Example of different networks with univariate time series data (3 features in 5 networks):

[10, 20, 30]
...                            This is for network 1
[60, 70, 80]

[1, 2, 3]
...                            This is for network 2
[6, 7, 8]

...

[-10, -20, -30]
...                            This is for network 5
[-60, -70, -80]

The problem with this one is, I might lose information of the feature correlation even though I'm putting their results into another network.

My question is, which is the best way to use when dealing with multivariate time series problems? I want to use the first method but value differences worry me. Second method is easier but I worry that I might be losing some essential information.


texts: How are missing values treated in XGB RF Classifier?

texts:
I was exploring Random Forest Classifier in XGBoost listed here : https://xgboost.readthedocs.io/en/latest/python/python_api.html
I was wondering how the missing values will be handled in this algorithm. I don't see it mentioned in the documentation. I don't think sklearn RF supports missing values in the data.
How is missing values handled in XGBoost RF classifier?


texts: What is the difference between gradient descent and gradient boosting? Are they interdependent on each other by any way?

texts: What is the difference between gradient descent and gradient boosting?  Are they  interdependent on each other in any way ?


texts: How to choose best model checkpoint when training deep learning model on all the data?

texts:
When training a final model for production, it's often recommended to train on all available data (train + dev + test), as discussed here.

I'm training a deep learning model. I typically save and use the best-performing model checkpoint based on performance on a held-out data set. During hyperparameter tuning, I used the dev set performance to pick the best checkpoint in a given experiment. I also have an untouched test set.

For the final model, one option would be to train on the combined training and dev sets, and use the test set to choose the best checkpoint. But what I'd really like to do is train on ALL the data. However, without a held-out set, determining which checkpoint is the best becomes impossible. How can I decide which one to deploy? I could guess based on earlier experiments by assuming the best performance will occur after a similar number of training iterations, but looking over the earlier experiments, it's clear that the best performance doesn't consistently appear at the same point during training.


texts: Could a unique variable such as invoice_id be used as a target variable?

texts:
I am trying to create a machine learning algorithm that will match invoice numbers with un-applied accounts receivable receipts.

un-applied receipts are just receipts that do not match the details we have in our database such as the amount.

My problem is that I do not know what the target variable should be. Any insights would be greatly appreciated.


texts: Which ML classifier is appropriate for me if all of my features are categorical?

texts:
My dataset contains four features. All of the features are categorical. There are 150 categories in the value of 1st and 2nd features. There are 8 categories in the value of 3rd and 4th feature. I replaced the categories with numeric values and applied Random forest. However, performance is not still up to the mark.

What other Machine Learning Classification algorithms I can try?


texts: Difficulty understanding the difference between Poisson, Quasi-Poisson, and Negative Binomial models

texts:
I will try to keep this short.
As an assignment for my GLM course, we were given a dataset on the # of homicide victims a person knows, as well as the race of the person. The main idea is to answer the scientific question "Does race help explain how many homicide victims a person knows?". This same dataset, and actually nearly all the sub-problems are solved here: https://data.library.virginia.edu/getting-started-with-negative-binomial-regression-modeling/.
My issue is, I am struggling to understand the difference between the 3 models. My understanding is that the model is a tool that I use to help me predict something. In this example, the race is meant to help me predict the # of homicides someone knows. Therefore, if all 3 models give me identical coefficients, then my predictions will all be identical. This leads me to be very confused, as the models are said to not be identical, and have different residuals, etc...
I don't understand how this is possible, and don't understand what I could be missing. It seems like it should be very simple as the prediction is being made using only 1 binary variable, and so there are only 2 options for each prediction, and the identical coefficients should create identical predictions...
Thanks a lot for any help in advance.


texts: Practical difference between Bayesian Neural Networks and Feed Forward Neural Network with Gaussian Noise

texts:
To my understanding, a BNN's weights come from a Gaussian with trained mean and standard deviation, while a FFNN of the following form, comes from a learned weight, which acts as a 'mean', and is multiplied by an untrainable standard deviation
Dense()
GaussianNoise(stddev=0.3)

What is the practical difference between these two approaches?


texts: Sentiment Analysis Using Neural Networks with Python

texts:
I'm working on a task for an assignment for school. The course doesn't have a textbook and instead uses the tutorial on DataCamp. I've struggled severely using DataCamp to learn (been through it multiple times) and instead have found that written tutorials help me learn much better and quicker (at least with machine learning models). I usually use Medium but haven't found anything useful on there, so far.
I was curious if there's a tutorial that might help me with the following task.
You will build a neural network designed to learn word usage and context using NLP techniques. You will provide visualizations and a report, as well as build your network in an interactive development environment.


texts:
What is the Difference between TDIDT, ID3, CART, and C4.5?

My main concern is about TDIDT, Is it first ever algorithm that came with Decision trees?

Is it predecessor or successor of ID3, CART, and C4.5?

What are the differences to others(ID3, CART, and C4.5?)?


texts: Predicting Soccer: guessing which matches a model will predict correctly

texts:
I took on a project to predict the outcome of soccer matches but it turned out to be a very challenging task.
I tried out different models but I only got 50-54% accuracy on my test dataset. Some of the models were created in such
a way that a certain model would predict if a team will win, draw, or lose a match. That same model would also predict if
the opponent of that team will win, draw, or lose the match. Each model predicting with an accuracy of about 50% on each team distinctively. The second set of models I tried, takes the combination of data from both teams and predicts which class the match belongs to (home win, away win, draw). In the system,
only 10 matches are given everyday to be predicted. Meaning, if I predict the 10 matches using the second model, I have a chance of predicting 5
correctly. In this project, I only need to predict 3 matches correctly out of the 10 matches given in a day. Is there a system
of knowing the 3 matches which my models have the best chance of predicting correctly? I only need to get 3 correct predictions, I usually
get 5 correctly but I don't know how to select my 3 best matches.

Note: The first type of models uses about 50 features for prediction while the second uses 101. I've tried ensembles, they still give
me ~50% accuracy. I'm still about to set up a system that selects matches where the prediction for the home team does not
contradict the prediction for the away team using the first type of models.


texts: Object detection model using images which have single instance of class per image-can it detect multiple instances in a single image?

texts:
I tried to train Faster R-CNN to detect multiple instances of a single class (eg. pomegranate on a pomegranate tree) but the training data consisted of 200 images where each image had only one object labeled. The model is detecting an object but with very low probability. Is the model trying to find a single instance of the class but getting confused?

model name: Faster RCNN inception v2

Why does the model detect objects more efficiently when zoomed in closer?

Is there something wrong in the training process or something else?


texts:
I am using convolutional neural networks (via Keras) as my model for facial expression recognition (55 subjects). My data set is quite hard and around 450k with 7 classes. I have balanced my training set per subject and per class label.

I implemented a very simple CNN architecture (with real-time data augmentation):

model = Sequential()
model.add(Convolution2D(32, 3, 3, border_mode=borderMode, init=initialization,  input_shape=(48, 48, 3)))
model.add(BatchNormalization())
model.add(PReLU())
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(256))
model.add(BatchNormalization())
model.add(PReLU())
model.add(Dropout(0.5))

model.add(Dense(nb_output))
model.add(Activation('softmax'))

After first epoch, my training loss decreases constantly while validation loss increases. Could overfitting happen that soon? Or is there a problem with my data being confusing? Should I also balance my testing set?


texts: Overfitting for minority class after SMOTE w/ random forests

texts:
I used SMOTE to make a predictive model, with class 1 having 1800 samples and 35000+ of class 0 samples. Hence, as per SMOTE, synthetic samples were created and the random forest was trained.

However, I am now getting most results as class 1 when I test my model. I just tried to test it on the training set and this is what I got:

Without SMOTE

With SMOTE

I've also tried hyperparameter optimisation, but that hasn't worked

Thanks

PS: Used SMOTE implementation in pandas with UnbalancedDataset library


texts: How to predict the sentiment of the entities form the tweet?

texts:
I have a JSON file (tweets.json) that contains tweets (sentences) along with the name of
the author.
Objective 1: Get the most frequent entities from the tweets.
Objective 2: Find out the sentiment/polarity of each author towards each of the entities.
Sample Input:
Assume we have only 3 tweets:
Tweet1 by Author1: Pink Pearl Apples are tasty but Empire Apples are not.
Tweet2 by Author2: Empire Apples are very tasty.
Tweet3 by Author3: Pink Pearl Apples are not tasty.

Sample output:
Entities in the topics extracted: Share a dictionary with extracted entities as keys and the number of
tweets an entity has mentioned as values.
Example dictionary for the above example: {“Pink Pearl Apples” : 2,
“Empire Apples” : 2}

Now, objective 2 -

Now using Count Vectorizer I have completed objective 1, now how to get the sentiment based on the entities(Objective 2)
Thanks in Advance!!!


texts:
I'm working on the commaai speedchallenge. The goal of the challenge is to predict the speed of a car based on a dashcam video. So far all the examples that I found (example 1, example 2) use some kind of method that takes in the information at frames $x_{t-h}, \ldots, x_{t-2}, x_{t-1}$ when trying to predict the speed at frame $x_t$. I think this makes sense, because you can get an idea of the speed by looking at differences between two frames. Without a reference point (with just one frame) it would be very hard to predict the car speed..

Well apparently not! While looking for solutions I also found this repository. I'll summarise the approach that is used:

Rescale the frames to 50x50 images, and divide them into two sets: X_train and X_test.
Take X_train, X_test and pass them through the first two blocks of a convolutional network called VGG16 that is already trained, so that you get output X_train_features and X_test_features. The writer of the code explains that he wants to use transfer learning because his computer specs are not good enough to train a network himself.
Train a linear regressor based on the extracted features and compute the mean squared error.

If I understand the idea correctly, the author of this code just looks at each frame individually, passes the frame through the first two convolutional blocks of the network VGG16, and predicts the speed of the car based on the extracted features, without looking at earlier frames!

I've tested the code and it works, it has a mean square error just above 1!

Question: Am I missing something, or can you predict the speed of a car just by looking at the features of a single frame? If the answer is yes; then how?

I know that you can take the weights from the early layers of trained convolutional networks and use them for other purposes. This is possible because the early layers of convolutional networks usually just filter for very general/coarse shapes. What I just don't understand is how you could predict something that is relative to other frames based on just those low level features. If a human were to look at a single frame in the data, he would not be able to predict the speed of the car correctly, right?


texts:
Which and all Machine Learning algorithms needs the data to be standardised/normalised before feeding into the model. How do we determine whether the particular model/data needs to be  standardised/normalised.

Thank you.


texts: Can I use scipy.optimize module with PyTorch?

texts: I need to use scipy.optimize module after encoding some data with pytorch. However, scipy.optimize does not take torch.tensor as its input. How can I use scipy.optimize module with data with gradient path attached?


texts: Are CNNs reliable when dealing with distance between two objects in the image?

texts:
I am building a CNN application in which the output is a continuous value(regression). And this value is very much related to the distance between two specific objects in the input image. Detecting those specific objects should not be an issue. But carrying the distance information in fairly accurate manner into the network is doubting me.

These are the key points according to me:

I did read CNNs are translation invariant. That is to some extent kernels do not carry positional information of the features.
One approach would be to have a very large size kernel which can fit both the objects inside together. This seems to have two issues. One is of course large kernels. Another is, it might need kernels for every possible separation value between two objects. Hence this does not seem to be reliable.

So am I having a misconception? Or else what can be done to get along with this?


texts: How to implement feature selection for categorical variables (especially with many categories)?

texts:
I've been trying to get some ideas of how I could treat categorical variables when doing feature selection. Mainly I've been running Random Forest feature importance on Python for which preprocessing could play a big part.

For some variables which come with a large number of unique values such as State, Zip Codes, Cities...one hot encoding may not make sense since it explodes your vector length. Another issue is also how representative that category would be. Eg: If you have oly one observation from MI, does it really make sense to encode a feature for this state? What is considered the most appropriate way to treat variables like this?


texts: Continuous Bag Of Words (CBOW) network architecture?

texts:
Looking into word2vec like embeddings I found this exercise on PyTorch's website which prompts the reader to implement a CBOW network in PyTorch.
My question is about the architecture to implement this CBOW network.
Here is my understanding: From a number of sources, it seems that the network should have a single hidden layer (with weights and no biases) which is connected to an activation layer (most sources say softmax). Then the network will be trained to map one-hot encoded words to likely contexts. Finally, the hidden layer's weights will be used to as the embedding matrix.
My confusion is: I see a number of solutions like this first one off google where there are multiple hidden layers. In this example, there is a embedding layer and there are two linear layers connected by a relu. Here is another that uses one linear layer.
My questions are:

What is the proper architecture to train CBOW encodings?
If this multiple hidden layer approach is correct, how do you not lose semantic information when you only use one of the layers as the encodings?
If the single hidden layer approach is correct, does anyone have examples of this being implemented using this approach in PyTorch (fine if no)?

Note: Very new to ML so feel free to correct me even on nit picky things so I can learn!


texts: When text mining what consideration should I give to picking a class label

texts:
If I create a dataset and there are attributes that could be used to label it how do I pick which or what tests should I use in pre processing to identify the most suitable?

As example the attributes I'm considered are related to courses of study
1) a  descriptor of the subject  type of programme
2)  a descriptor of the length and award of the programme


texts:
I am training multiple linear Ridge regression using Python (sklearn). I want to train regression using my loss function. I found that my loss function in sklearn I can define using sklearn.metrics.make_scorer and then:
def my_custom_loss_func(predict, target):
...
    return loss

and call:
my_scorer = make_scorer(my_custom_loss_func, greater_is_better=True)
model = GridSearchCV(Ridge(),
                    scoring=my_scorer,
                    cv=5,
                    param_grid={"alpha": np.logspace(-6, 6, 13)})

I have a question, what loss function I should compose if for the forecast a very bad outcome if predict - target > 0 and not bad result outcome if predict  - target < 0 and the best result if predict - target = 0.  That is, it is important for me that there are as few cases as possible where predict - target > 0. Cases predict - target < 0 in general  suit me, but it is also better to minimize them to 0. Also, please, you can show how to do this using  sklearn.metrics.make_scorer.


texts:
I have a corpus of unstructured text that, due to a concatenation from different sources, has boilerplate metadata that I would like to remove. For example:

DESCRIPTION PROVIDED BY AUTHOR: The goal of my ...
Author provided: The goal of my ...
The goal of my ... END OF TRANSCRIPT
The goal of my ... END, SPONSORED BY COMPANY XYZ
The goal of my ... SPONSORED: COMPANY XYZ, All rights reserved, date: 10/21

This boilerplate can be assumed to occur in beginning or end of each sample. What are some robust methods for wrangling this out of the data?


texts:
I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution.
The problem is: first I'm wondering what are the main results from the HDP procedure (in the case of LDA we obtain two matrices that we can use to construct word clouds and graphs but in this case I'm not sure about the results) and what is the exact procedure?


texts:
I am trying to understand word-vectors and was reading this paper https://arxiv.org/pdf/1301.3781.pdf This paper proposes CBOW architecture which uses projection layer.
What is projection layer? I already searched and found this question on stack overflow.
But I still did not understand what is computed in projection layer.
For example:
In Feedforward NN there is activation layer which computes the activation function.
So likewise what is the function that is computed in Projection layer


texts: In Keras, the inverse of Conv2D is Conv2DTranspose. But how can I create the inverse of LocallyConnected2D? That is, using "local" connections, how can I map a small set of nodes into a larger set of nodes?


texts:
I've came across the following problem, that I recon is rather typical.

I have some large data, say, a few million rows. I run some non-trivial analysis on it, e.g. an SQL query consisting of several sub-queries. I get some result, stating, for example, that property X is increasing over time.

Now, there are two possible things that could lead to that:

X is indeed increasing over time
I have a bug in my analysis

How can I test that the first happened, rather than the second? A step-wise debugger, even if one exists, won't help, since intermediate results can still consist of millions of lines.

The only thing I could think of was to somehow generate a small, synthetic data set with the property that I want to test and run the analysis on it as a unit test. Are there tools to do this? Particularly, but not limited to, SQL.


texts: Considering the output of a BLSTM in pytorch, what's the order of the elements?

texts: I am currently using pytorch to implement a BLSTM-based neural network. I understand that the output of the BLSTM is two times the hidden size. However, I am currently unable to find out whether this is ordered as [forward_state_0, backward_state_n, forward_state_1, backward_state_n-1,..., forward_state_n, backward_state_0] or as [forward_state_0, forward_state_1,..., forward_state_n, backward_state_n, backward_stat_n-1,...,backward_state_0] or something else. I'd like to feed a pairwise maximum of the outputs to the next layer so the most import thing for me is which are the corresponding forward and backward states.


texts: Limits of Javascript on the implementation of AI algorithms

texts:
I have a basic understanding of javascript, and know hardly any other programming language. Am I bound to face some issues in the field of neural networks and machine learning because of this? Should I learn something else for the sake of avoiding some inherent weaknesses of the language? I am most worried about the capacity of javascript to handle data, rather than its possibilities regarding the textual implementation of the algorithms per se...

Thank you


texts: Help with breaking up a column in PowerBI to make a bar chart

texts: My data set is a list of documents. I have a column in my data set - 'Independent Variable'. This column can contain nothing, i.e. 'NA', or one variable, i.e. 'var1', or multiple, i.e. 'var1;var2;var3' and is essentially metadata for the documents. I'm attempting to create a visual, like a bar chart, that would show how many documents in my data set are associated with each independent variable. I have roughly 30 independent variables, and they are all housed in the column 'Independent Variable', hence when trying to make a chart i get a ton of entries for that so each bar is 'var1', 'var2', 'var1;var2' etc. How can I break it down so that only var1, var2, var3 are my buckets and if those variables exist in the column that document is counted


texts: Why are Machine Learning models called black boxes?

texts:
I was reading this blog post titled: The Financial World Wants to Open AI’s Black Boxes, where the author repeatedly refer to ML models as "black boxes".

A similar terminology has been used at several places when referring to ML models. Why is it so?

It is not like the ML engineers don't know what goes on inside a neural net. Every layer is selected by the ML engineer knowing what activation function to use, what that type of layer does, how the error is back propagated, etc.


texts:
I am building Seasonal Arima for more than 10k products. In all the tutorials and blogs mentioned, I need to do the exploration to find the p,d,q values along with seasonality value using the visualization plots.

But doing this analysis for 10k products is impossible. How can I automate the process? I am currently using the pmdarima package which has auto arima but when modelling for seasonal arima, it is expecting the seasonality time period.

How can I find the seasonality period with plots? Any statistical method or function which helps me here?


texts: How to model a supervised recommender system with varying data

texts:
Suppose there are 2000 movies and a company wants to recommend some movies (for example, at most 5 movies) to each visitor. The objective is to learn how to predict which movie will be selected if a specific set of movies is recommended.
   option-1  option-2  option-3  option-4  option-5  Selected-Movie
1. movie1    movie3    movie4                        movie4
2. movie3    movie4    movie100  movie1000 movie1001 movie1001
3. movie4    movie5    movie34                       movie34

Based on this data set, I want to learn when sample 1 is suggested to a customer, he will visit movie4. Because the number of features can be so high (here 2000 movies), I think it would not be a good option to use on-hot-encoding. Think at most 5 movies can be recommended, I thought it might be a good option to consider a vector with size 5 and if the number of recommended movies is less than 5, blanks will be replaced with 0. However, in this situation, the perturbation of movies will be important. For example, (1,2,3,4,5) will be different from (2,1,3,4,5) and I want to consider both cases the same. In other words, all 5! perturbations should be the same and there is no difference between them. Moreover, with this representation of data, I think it will not be possible to use Decision Tree whereas some algorithms like Catboost works.
My preference are algorithms that can generate rules like Decision Tree. I would be thankful if you have any recommendation for data representation and how features should be considered.


texts: What are some significance tests to rank features(multiple) before training the data

texts:
I have 8 features for a classification problem. The target value tells if there was an anomaly or not.

I want to run some significance tests to rank each feature, as being a distinctive feature of anomalies.

Could you please suggest some significance tests for this in Python?


texts: Is it correct to load weights from task Masked Language Modeling to train Causal Language Modeling

texts:
I intend to use 2 tasks of modelling including (a) Causal language modelling &
(b) Mask language modelling for training my new added tokens

My pseudo-code is below
##add new tokenizer
model_name = "vinai/phobert-large"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
with open("tokenizer_vocab.txt","r") as wr_file:
   new_tokens=wr_file.read().splitlines()
_________________
## Train the (a) task - Casual language modeling
added_tokens = tokenizer.add_tokens([tokens for i in new_tokens])
model.resize_token_embeddings(len(tokenizer))

## train the model following the hugging face released
https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb#scrollTo=JAscNNUD3l-P

##save_the_model
model.save_pretrained('weights_tokenizer/')
tokenizer.save_pretrained('weights_tokenizer/')
___________________
## Train the (b) task - Masked language modeling
model = AutoModelForMaskedLM.from_pretrained("weights_tokenizer/")
tokenizer= AutoTokenizer.from_pretrained("weights_tokenizer/")

My question here is it correct when I want to train the (b) task with weights from the (a) (because I think I can somehow more enrich the tokenizer).
Or there is any solution that I can train my tokenizer in both these two tasks. And I then can use the weights (from training on both two tasks above) to train my model with the task (C)
I do appreciate your time and sharing.


texts: AttributeError: 'numpy.ndarray' object has no attribute 'columns'

texts:
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import SelectFromModel
from scipy.io import arff

data = arff.loadarff("C:\\Users\\manib\\Desktop\\Python Job\\Project Work\\Breast\\Breast.arff")
df = pd.DataFrame(data[0])

df.head()

df["Class"].value_counts()

X = df.iloc[:,:24481].values
y = df.iloc[:, -1].values

from sklearn import preprocessing
label_encoder = preprocessing.LabelEncoder()

y=y.astype('str')

y= label_encoder.fit_transform(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)

sel = SelectFromModel(RandomForestClassifier(n_estimators = 100))
sel.fit(X_train, y_train)

sel.get_support()

selected_feat= X_train.columns[(sel.get_support())]
len(selected_feat)

print(selected_feat)


texts: What criteria use in order to select the best internal validation for clustering?

texts:
I am doing homework about how to evaluate a clustering algorithm both hierarchical and partitional.
For doing this I have a dataset that I can plot as you can see:

The clustering algorithms that I am using are K-Means, Gaussian mixture, and Agglomerative.
The problem is that I don't know which criteria use for selecting an internal validation for K-MEANS, for example, I read that silhouette is better for a convex cluster, but I am not able to see in the image if any of groups are or aren't convex.
I am read some papers like:

http://datamining.rutgers.edu/publication/internalmeasures.pdf
https://arxiv.org/pdf/1905.05667.pdf

but the problem is that I am not found any criteria to select A or B method for internal evaluation with facts, they're only a test about all vs all.
Where can found or what is found the criteria for choosing one or the other method for evaluation criteria?


texts: sklearn.naive_bayes VS categorical variables

texts: In a binary classification, how can I use sklearn.naive_bayes python module to predict the class of inputs with 5 categorical variables (not binary)?


texts:
For some binary image classification problems having close to 100% precision is super important and recall is much less important.

What are best practices for maximizing precision? Setting the probability threshold for model.predict() greater than 0.5 seems like a reasonable approach.

Are there any other recommended methods?


texts: Q-learning, state transition, immediate rewards (trading logic)

texts:
I've been thinking about how to correctly calculate rewards for several weeks now.

Here is a grid example:

##################################################################################
# | 0 | 1 | 2 |
---------------
0 |...|...|...|
1 |...|.A.|.G.|
2 |...|...|...|
##################################################################################
- the Agent(A) is currently at State(t) = [1,1]
- if he moves right -Goal(G)- he would get a Reward(t) of 100
- if the Agent(A) goes another way, he gets a Reward(t) of 0
- the selected Action(t) at State(t) depends ONLY on State(t) (no exploration)
- BUT the Reward(t) is calculated depending on State(t+1)?
- depending on Action(t) and based on State(t) we do the Transition to State(t+1)
##################################################################################

Is that correct up to here?

If not, then I have misunderstood something and therefore I am not making progress. I try to implement the whole thing in a trading logic, I am not sure if I am doing it right.

Imagine a simple time series and 3 Actions (B=BUY/H=HOLD/C=CLOSE):

# 0 1 2 3 4 5 6 7 8 9 . . . . . . . . . 19 (index/position)
##################################################################################
# 1 3 5 8 7 9 2 3 6 4 . . . . . . . . . 8  (data/value/price)
# . . B H H C . . . . . . . . . . . . . done
#    t^ buy open
##################################################################################

The Agent opens a BUY on Index(t)=2 at Price(t)=5, the Reward(t) is 8(t+1) - 5(t) = 3. I open a trade on State(t), the Reward(t) depends on the next value(t+1)?

Is that correct?

And if i close the trade? The next value doesn't matter, as I'm closing the trade at the current Price(t)?

In the examples here, the difference(entry.price - exit.price) is used, which is very confusing:

https://github.com/miroblog/deep_rl_trader/blob/master/TraderEnv.py#L94

https://github.com/edwardhdlu/q-trader/blob/master/train.py#L36

I see 2 problems, the drawdown while holding the positions is ignored and the sparse rewards are more difficult to learn, what do you think?

Thanks in advance!

EDIT: The main question is:

In a grid world, the Action(t) is done at State(t), BUT the Reward(t) depends on the next State(t+1), because only if this is the goal, it gets a reward.

In a trading environment, i can find a lot of different versions, some guys are using future values, some are using the difference of entry/exit/current close price or other stuff like sharpe/sortino/...

What is correct in a trading environment and why?

or

How do i calculate the rewards for a trading bot?


texts: How to handle unseen class feature in test dataset

texts:
feature   Definition    Key
survival    Survival    0 = No, 1 = Yes
pclass   Ticket class   1 = 1st, 2 = 2nd, 3 = 3rd
sex       Sex           M/F
Age     Age in years

feature notes given below

pclass: A proxy for socio-economic status (SES)
1st = Upper
2nd = Middle
3rd = Lower

age: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5

Suppose I have dataset same as mentioned above.

How to handle a case where I can have a unseen value for a class. For example : In our dataset we have 3 different value for pclass. But how to handle a case where pclass value may have 4rth class say "elite" which did not appear in training dataset but appeared in the test dataset


texts:
Logistic and Linear Regression have different cost functions. But I don't get how the gradient descent in logistic regression is the same as Linear Regression.

We get the Gradient Descent formula by deriving the Squared Error cost function. However in Logistic Regression we use a Logarithmic Cost function instead. I think I am lost here.


texts:
I've written the code below to try word2vec implementation of gensim. I've two questions:

Even though I've removed stop words, the word "the" is listed as one of the most similar words of "friend".
The most similar words of "friend" is not satisfying (at least according to my subjective evaluation). Should I try a larger text (austen-emma.txt file contains 192427 words) or the problem is something else?

Thanks.
import nltk
from nltk.tokenize import sent_tokenize
from nltk.corpus import gutenberg
import gensim
from gensim.models import Word2Vec
from gensim.parsing.preprocessing import remove_stopwords
from nltk.tokenize import RegexpTokenizer

text = gutenberg.raw('austen-emma.txt').
text = remove_stopwords(text).
tokenizer = RegexpTokenizer(r'\w+').
data = [].

for i in sent_tokenize(text):
    temp = [].
    for j in tokenizer.tokenize(i):
       temp.append(j.lower()).
    data.append(temp).

model = gensim.models.Word2Vec(data, min_count = 1,
                          size = 32, window = 2)

model.wv.most_similar(positive='friend', topn=10)

[('mind', 0.9998476505279541),
 ('present', 0.9998302459716797),
 ('till', 0.9998292326927185),
 ('herself', 0.9998183250427246),
 ('highbury', 0.999806821346283),
 ('the', 0.9998062252998352),
 ('place', 0.9998047351837158),
 ('house', 0.999799907207489),
 ('her', 0.9997915029525757),
 ('me', 0.9997879266738892)]


texts: Why is the derivative of the product of a vector and its transpose cTc2 c?

texts:
I'm learning principal component analysis and i have to derive the result that will minimize the error $$c^* = \arg \min{[x-g(c)]^2} = \arg \min{[x-g(c)]^\mathsf{T}[x-g(c)]}$$.

What I have done so far is to simplify the error by expanding the argument, replacing $g(c)$ by its definition $g(c)=Dc$, and use the fact that $D\mathsf{T}D=I$:

$c^* = \arg \min [{x^\mathsf{T}x - 2x^\mathsf{T}g(c) + g(c)^\mathsf{T}g(c)}]$

$c^* = x^\mathsf{T}x - \arg \min {[ -2x^\mathsf{T}g(c) + g(c)^\mathsf{T}g(c)}]$

$c^* = x^\mathsf{T}x - \arg \min {[ -2x^\mathsf{T}g(c) + (Dc)^\mathsf{T}(Dc)]}$

$c^* = x^\mathsf{T}x - \arg \min{[ -2x^\mathsf{T}g(c) + c^\mathsf{T}D^\mathsf{T}Dc}]$

$c^* = x^\mathsf{T}x - \arg \min{[ -2x^\mathsf{T}g(c) + c^\mathsf{T}Ic]}$

$c^* = x^\mathsf{T}x - \arg \min{[ -2x^\mathsf{T}g(c) + c^\mathsf{T}c]}$

$(−2x^\mathsf{T}Dc + c^\mathsf{T}c) = 0$

$-2D^\mathsf{T}x+2c=0$

$c=D^\mathsf{T}x$


texts: In which CRAN mirror is the quadprog package available?

texts:
I was trying to find the accuracy of some predicted data and I came to know that I can use the forecast package to do this. When I installed it and gave library(forecast), it said it needs quadprog package to work with forecast package.

I searched it in many CRAN mirrors [US(CA), US(IN), IRAN, MALAYSIA] and in all these only quadprogXT is available and with this package the forecast package is still not working.

In which CRAN mirror is this quadprog package available?


texts: What kind of neural network would work best for loosely-defined data, like video game RAM?

texts:
I'm trying to build out a network layer map for a neural network to use in an NES AI.  Most networks I run across on web searches are CNNs that use image data to identify things.  Miles and miles and miles of papers, questions, and tutorials about image-based CNNs.  Even the video game machine learning AIs are generally using rendered video frames as inputs to a CNN.

I played around with SethBling's MarI/O a few years ago.  But that's used NEAT, a 16-year-old algorithm, and MarI/O seems more like an interesting experiment than something that could actually beat human play.  It gets stuck once the level is no longer going right, and learns incredibly slowly.

My goal this time is to ditch the video feed and go straight for the NES RAM.  I don't want something that can match a 4-year-old, but an AI that could potentially beat TAS speedruns or find new tricks.  There's plenty of useful hidden details in the RAM, and it will contain a map of the enemy locations already.  Fortunately, the NES RAM footprint is only 2048 bytes, so feeding that in as input isn't a challenge.

The real challenge is creating a NN map that would work for this data.  Data like this is only very loosely-defined.  While an entire map could be built for each game, that's not very practical, and could miss out on memory locations that people haven't discovered yet.  (The reward/penalty side will need some hard-coded locations for score, death triggers, timers, etc.)

However, they can be broken down into a few various types:

Integers
Signed Integers
Bitmaps
Status values

I'm starting to develop a NN layer map as best as I can.  But I don't have a template to lean off of, so I have no idea how well this will work:

Layer 1: 2048x10
    Byte representation of RAM (ByteTensor)
        Times 10 frames

    + torch.nn.RReLU

Layer 2: 2048x266x10
    torch.stack(
        Copy      of Layer 1 ( torch.view + mm(1/255) )           # integers
        Copy      of Layer 1 ( convert to signed + mm(1/255) )    # signed integers
        Bitmap    of Layer 1 ( torch.bitand(a, $bit) for 2**0-7 ) # bitmaps
    Value map of Layer 1 ( torch.eq(a, $val) for 0-255 )      # status values
    )

    + torch.nn.RReLU

Layer 3: Need something to reduce the valid activators???

Layer 4:
    nn.LSTM

Layer 5: 512
    Fully-connected layer to outputs

A few questions:

Is this the right approach?  Are there better types of networks that would work here?
With an expanded input set of 2048x266=544K inputs, what kind of layer would I use to reduce that down to only usable sets?  There's a lot of wasted zeros for memory locations that may only have statuses up to 0x0A.
If I decide to add the video feed back in, how do I tie that together with a completely separate network?  Does that get tied into the fully-connected layer?
Should this be much much bigger, like AlphaGo or Leela Chess Zero's humongous NN stack?  Processing time isn't a factor, since it's not running in real time, but a single game shouldn't take months to process, either.


texts: High multi class classification with small data set

texts:
I am working on a multi-class classifier for data set with 240K samples and ~1880 classes, the most  populated class is 4% of the dataset and large number of classes are less then 1% of the samples.

I am trying to find a method to decide for which class I have enough information to correctly predict or to reject the prediction due to classifier performance for that particular class.

Also, are there any good methods to deal with that kind of situations of high multi class with very low sample size?


texts: Finding best match/score between 2 datasets

texts:
I am looking for a potential matching algorithm to apply to 2 datasets (DS1,DS2), that would provide a score for all DS1 x DS2.
To exemplifies the problem: imagine a pool of job seekers are looking for jobs. What would the jobs that match the job seeker profile, and what could the job seekers that match a job.
Any pointer or documentation you could direct me to help me with this?

inputs:
jobseekers (DS1)
ID | name | skills | bio | ...
u1 | alex | C#     | candidate bio blah ...
u2 | john | JVM,AWS| ...
u3 | emma | AWS,CSS| ...

jobs (DS2)
ID | name            | spec                | skills needed | ...
j1 | C# engineer     | a job spec blah ... | C#,AWS
j2 | Java developer  | a job spec blah ... | JVM,AWS
j3 | VueJS developer | a job spec blah ... | CSS

outputs:
scoring matrix
   j1  j2  j3
u1 0.2 0.1 0.3
u2 0.3 0.4 0.5
u3 0.6 0.1 0.3


texts: Outlier detection by unsupervised algorithm: Fraud Detection

texts:
I have set of 300,000 set of rows with credit card transactions and my job is to find outliers (suspicious transactions) in those dataset.

I have created around 5 features (All continuous data, with 1 column as transaction id)

I need to return list of all transaction id,which looks suspicious

What have I tried

I have tried using K means algorithm, but it does does not fit my laptop's memory (8 GB) and it crashes.

I wanted to try 1 class SVM, but I do not see any good tutorial to get me started. I tried scikitlearn official tutorial, but it already has outliers added to it and they are just plotting it.

http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py

http://scikit-learn.org/stable/auto_examples/covariance/plot_outlier_detection.html#sphx-glr-auto-examples-covariance-plot-outlier-detection-py

How do I automatically detect the outliers and return those observations ?


texts: What is the relationship between clustering and association rule mining?

texts:
Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).

Source: Wikipedia

  Association rule learning is a method for discovering interesting relations between variables in large databases.

Source: Wikipedia

So both, clustering and association rule mining (ARM), are in the field of unsupervised machine learning. Clustering is about the data points, ARM is about finding relationships between the attributes of those datapoints.

However, I wonder if there are more relationships. For example, given a clustering, can this enhance / simplify ARM or vice versa?


texts:
I am currently practicing my data visualization skills with ourworldindata dataset. I noticed there is a negative value of new_cases in Benin country. Out of curiosity, what does that mean? I mean, why would the new_cases has a negative value?
Here is where the negative value is found.


texts: Any case studies using Bayesian Networks for system design trades?

texts:
I am exploring using Bayesian Networks to identify the best parameters within a system design, to improve its performance. I'm trying to find any case studies where this has been used successfully or unsuccessfully.

As an example consider an automatic assisted anti-collision system on a car.  It could use some combination of ultrasonic, radar & IR seekers to sense and measure the closing speed with the car in front.  These all have different costs / reliabilities and perform differently in different weather conditions.  I am interested in understanding how effective Bayesian Networks can be in evaluating the trade space and identifying the best solutions to these types of problems.


texts: How to calculate precision at K and NDCG for ranking algorithms

texts:
I am ranking a filtered item list as per user's metadata and historical behaviour.

Now how to calculate metrices like precision at K?

One approach could be -
Divide historical data in training and test dataset(e.g. 90% and 10%). Now pick all the items for a user from test data and randomize the order and pass it to the ranking algorithm.

Now compare actual items for the user from test data to the ranked output.


texts: Keras model with LSTM quantization aware training

texts:
I would like to run quantization aware training with a keras model which has an LSTM layer. However, just the LSTM layer seems to not be supported. Alan Chiao seems to suggest here that it is possible to use the Quantize Config to allow this layer to be used. However, I can't seem to make the right Quantize Config which allows for the use of the LSTM layer.
I attach below both the code for my model and the Quantize Config.
Could anyone confirm either

how to properly make a quantization aware model with an LSTM layer with keras
if LSTM quantization simply isn't supported with keras

Thanks for the feedback!
from tensorflow.keras import layers as L
quant_layer = tfmot.quantization.keras.quantize_annotate_layer
quant_apply = tfmot.quantization.keras.quantize_apply
quantize_scope = tfmot.quantization.keras.quantize_scope

######################  Simple Quantized Model   ####################

i = tf.keras.Input(shape=(32,32,1))
x = (L.Conv2D(1, (5, 1), activation='relu', padding='same'))(i)
x = L.BatchNormalization()(x)
x = L.Lambda(lambda q: K.squeeze(q, -1), name='squeeze_last_dim')(x)
x = quant_layer(L.LSTM(64, return_sequences=True),DefaultLSTMQuantizeConfig())(x)
x = tf.keras.layers.Flatten()(x)
x = L.Dense(64, activation='relu')(x)
o = L.Dense(12, activation='softmax')(x)

simple_model = Model(inputs=[i], outputs=[o])
simple_model.compile(optimizer='adam', loss=['sparse_categorical_crossentropy'], metrics=['sparse_categorical_accuracy'])

with quantize_scope(
    {'DefaultLSTMQuantizeConfig': DefaultLSTMQuantizeConfig}):
    simple_model_quant = quant_apply(simple_model)

simple_model_quant.summary()

##################### Quantize Config  #################
LastValueQuantizer = tfmot.quantization.keras.quantizers.LastValueQuantizer
MovingAverageQuantizer = tfmot.quantization.keras.quantizers.MovingAverageQuantizer

class DefaultLSTMQuantizeConfig(tfmot.quantization.keras.QuantizeConfig):
    # Configure how to quantize weights.
    def get_weights_and_quantizers(self, layer):
        return [(layer.layer.recurrent_kernel, LastValueQuantizer(num_bits=8, symmetric=True, narrow_range=False, per_axis=False))]

    # Configure how to quantize activations.
    def get_activations_and_quantizers(self, layer):
        return [(layer.activation, MovingAverageQuantizer(num_bits=8, symmetric=False, narrow_range=False, per_axis=False))]

    def set_quantize_weights(self, layer, quantize_weights):
      # Add this line for each item returned in `get_weights_and_quantizers`
      # , in the same order
        layer.kernel = quantize_weights[0]

    def set_quantize_activations(self, layer, quantize_activations):
      # Add this line for each item returned in `get_activations_and_quantizers`
      # , in the same order.
        layer.activation = quantize_activations[0]

    # Configure how to quantize outputs (may be equivalent to activations).
    def get_output_quantizers(self, layer):
        return []

    def get_config(self):
        return {}


texts: Action selection in actor-critic algorithm:

texts:
I have an action space that is just a list of values given by acts = [i for i in range(10, 100, 10)].
According to pytorch documentary, the loss is calculated as below. Could someone explain to me how I can modify this procedure to sample actions from my action space?

   m = Categorical(probs)
   action = m.sample()
   next_state, reward = env.step(action)
   loss = -m.log_prob(action) * reward
   loss.backward()```


texts: Generate sentences from keywords by adding formal word

texts:
For example, I have a list of key word like : I , hungry => output: I am hungry or I , author , poem => output: I am the author of this poem.
Can someone please suggest the simplest way to achieve this?
I am newbie, please tell me which knowledge I must have to solve this problems.


texts: 'int' object is not iterable using panda python

texts:
Here I want to calculate time interval in between row by row in time column import from csv file. my start time 6.00 a.m and my end time is next day 6.00 a.m. In between this time periods how to find time interval period. Can anyone help me to solve this problem?
Here I upload my code and csv file.

def time_diff(start, end):
start.append(pd.to_datetime(data['time'][0],formt = '%H:%M:%S').time())
end.append(pd.to_datetime(len(data['time']), format='%H:%M:%S').time())
if isinstance(start, datetime_time): # convert to datetime
    assert isinstance(end, datetime_time)
    start, end = [datetime.combine(datetime.min,i) for t in [start, end]]
if start <= end:
    return end - start
else: # end < start
    end += timedelta(1) # +day
    assert end > start
    return end - start
for time_range in range (len(data['time'])):
start = [datetime.strptime(t,'%H:%M:%S') for t in time_range]
end = [datetime.strptime(t,'%H:%M:%S') for t in time_range]
print(time_diff(s, e))
assert time_diff(s, e) == time_diff(s.time(), e.time())

my csv file :-

I got an error:-

TypeError                                 Traceback (most recent call last)
 <ipython-input-28-be40ae41d284> in <module>()
 12         return end - start
 13 for time_range in range (len(data['time'])):
 ---> 14     start = [datetime.strptime(t,'%H:%M:%S') for t in time_range]
 15     end = [datetime.strptime(t,'%H:%M:%S') for t in time_range]
 16     print(time_diff(s, e))

 TypeError: 'int' object is not iterable


texts: What loss function should I use for auto encoders for text

texts:
I'm trying to implement an autoencoder for text. But I don't know which loss function I should use ? I tried using the mse but I get a huge loss 1063442. I'm trying to build a very simple autoencoder using only LSTM layers in Keras. On the keras-blog they use binary-crossentropy but I think the reason for this is because they use black and white images.

Thanks


texts: Training loss in CNN oscillates but not due to too large learning rate

texts:
I am training a CNN (SSD Inception V2) and I get a strange shape of the training loss:

At first, I thought it was a too large learning rate (suggested in Question Oscillating loss in CNN
). But after taking a closer look, I don't think that's the case. When I zoom in to where the increase in loss happens, it looks like this:

My interpretation is that this cannot be caused by learning rate because the individual steps made in the training are relatively small. But there is something the causes the training loss to start increasing at some point.
Question: What might be the cause of this? I am not looking for a technical answer or a fix but rather for an answer that would help me understand the behaviour on a conceptual level.
Thank you!


texts: Is Neural Network Architecture independent of Data?

texts: If I change my dataset (let's say it is always images), should I change the architecture of my neural network?


texts: What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2?

texts:
I am a newbie to data mining. I am trying to find associations between two categorical variables. Since more than 20% of my expected frequencies are less than 5, I wanted to use Fisher exact test but it turns out it is generally used for contingency tables 2x2 but my variables have more than two values. Would really appreciate recommendations on the best course of action for me now. Here are some options I found after some search:

Use Freeman-Halton extension to Fisher's Exact test for more than 2x2 table.
Merge multiple attributes values so that I end up with 2x2 contingency table and then use Fisher's Exact test.
Merge multiple attributes values so that I end up with expected counts > 5 and then use chi square test for independence.
Use the Crammer V test.

I would like to know what is the standard practice in this case when your categorical variables with >2 possible values end up with less than 5 expected counts?
Thanks,


texts: I am struggling on object detection in an image, but not sure which model to use. I tried to design a model from scratch, CNN, but it could not meet my requirements. I need a right direction or step by step approach to help me achieve it.


texts: Does label encoding an entire dataset cause data leakage?

texts:
I have a dataset on which one of the features has a lot of different categorical values. Trying to use a LabelEncoder, OrdinalEncoder or a OneHotEncoder results in an error, since when splitting the data, the test set ends up having some values that are not present in the train set.
My question is: if I choose to encode my variables before splitting the data, does this cause data leakage?
I'm aware that I shouldn't perform any normalization or educated transformations on the data before splitting the dataset, but I couldn't find a solution for this problem inside scikit-learn.
Thanks in advance for any responses.
Edit: This particular features has very high cardinality, with around 60k possible values. So using scikit-learn's OneHotEncoder with handle_unknown set to ignore would introduce too many new columns to the dataset.


texts: Why do people use different Batch Sizes for training and validating?

texts:
For example, I see people often use a train batch size of 32 and a validation batch size of 16.

Are there any particular reasons for that?


texts:
I am reading this book http://infolab.stanford.edu/~ullman/mmds/ch9.pdf
there is an exercise 9.3.2 a) it says

  Exercise 9.3.2 : In this exercise, we cluster items in the matrix of
  Fig. 9.8. Do the following steps. (a) Cluster the eight items
  hierarchically into four clusters. The following method should be used
  to cluster. Replace all 3’s, 4’s, and 5’s by 1 and replace 1’s, 2’s,
  and blanks by 0. use the Jaccard distance to measure the distance
  between the resulting column vectors. For clusters of more than one
  element, take the distance between clusters to be the minimum distance
  between pairs of elements, one from each cluster.

I am not able to understand how clustering is done?If I replace all 3,4,5 with 1 and 1,2, with 0 I am left with a matrix

  a b c d e f g h
A 1 1   1 0   1 0
B   1 1 1 0 0 0
C 0   0 1   1 1 1

What is meant by cluster formation here? How do I solve this exercise or any solution available online please give a link.


texts: How can one use a validation set to reduce overfitting Naive Bayes?

texts:
What is the correct procedure for using a validation set to reduce overfitting?

Say I split the data 80:10:10 (training: validation:test). I train on the training set then get 90% accuracy. I apply this model to the validation set then get 20%. What do I do then?

How can the validation set be used to reduce overfitting especially with reference to Naïve Bayes?


texts: Problem in Recommendation for categorical data?

texts:
I have been building a recommendation model to recommend certain questions in an interaction platform to users to help each other.
I have calculated an affinity score between categories to find which top categories are to be recommended. But each category has questions by users in itself. The amount of the questions increases with every new post a user posts in a certain category. Now how can I choose which of these questions to recommend when I have chosen the category through my affinity score ? Do I make it random ? Do I display the questions which come first in the data base ? Or is there any better alternative ?


texts: How to evaluate the quality of representation for variables and individuals of a PCA in scikit-learn?

texts:
I just looked at the PCA in scikit-learn, but I didn't find a way to evaluate the quality of representation for variables and individuals like I usually do using the squared cosine.

The squared cosine between the vector and the projection of the individual or the variable on the axis is used to determine the quality of the representation.

For instance with the FactoMineR package in R to get those results for the variables you can simply do :
res.pca$var$cos2

Where res.pca are the results of your PCA.

So how would I do something similar with scikit-learn ?


texts: I have a big amount of customer journeys, where CJ is a sequence of user interactions with the site. It looks like: read an article in the blog -> checked the special offer -> watched video about the product -> made a purchase. Some actions may be repeated a few times. I want to represent each journey as a chain of input variables, with output 1 when the purchase was made and 0 when it wasn't. My goal is to evaluate the influence of each interaction on the final result. What is the best method or algorithm for this?


texts:
I don't really know if I'm posting in the right place but I hope so.

I'm writing my bachelor degree, and I collected some data about fonts and language support.

At the moment, all my data is stored in an excel spreadsheet, in the form of:

Column: language support (e.g. Cyrillic, Cyrillic extended, Arabic, Vietnamese, Devanagari, etc)
Rows: names of the fonts that supports that language, up to 890

Note that some of there fonts, even though they occur only one per columns, they may appear several times in different columns (e.g. Noto Sans will appear in Latin, Latin Extended, Cyrillic, Cyrillic Extended, Simplified Chinese and many others; while others e.g. Padauk will only appear under Myanmar since that font may not support all Latin characters).

What I would like to do is to put these data in visual form, ideally via a Venn Diagram that displays how much each language is supported (area) and how many language are supported at the same time by a given font (opacity overlapping). I don't need to process this plotting step inside excel, I just need to get the data of how many fonts per intersection are there (e.g. latin and cyrillic gives 100, latin and arabic gives 80, latin and arabic and cyrillic gives 10 ... up to latin and cyrillic and arabic and devanagari and [n] gives 0).

I thought I could have parsed these data with javascript and json but I can't get to properly convert my excel file to json in a fashion that I can really use.

Any suggestion in warmly welcomed and I really thank you in advance


texts: ImageDataGenerator - trained with model.fit instead of model.fit_generator

texts:
I am a beginner in using the ImageDataGenerator from Keras and I accidentally used model.fit instead of model.fit_generator.

def gen_Image_data():
   gen = ImageDataGenerator(
           width_shift_range=0.1,
           horizontal_flip=True)
   return gen

train_gen = gen_Image_data()
test_gen = ImageDataGenerator()

train_samples = train_gen.flow(X,y, batch_size=64)
test_samples = test_gen.flow(X_val, y_val, batch_size=64)

history = model.fit(train_samples, steps_per_epoch = np.ceil(len(X)/64),
              validation_data=(test_samples),
              validation_steps=np.ceil(len(X_val)/64),
              epochs=300, verbose=1, callbacks=[es])

Many thanks for every hint


texts: ValueError: Tensor Tensor("activation_5/Softmax:0", shape=(?, 2), dtype=float32) is not an element of this graph

texts:
There seem to be an issue with predicting using my keras model. I had trained it using the following keras code:

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(150, 150,3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(32, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(64, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(2))
model.add(Activation('softmax'))

model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['accuracy'])

However when i predict it on my local system after training with the shape (1,150,150,3) . It predicts accurately with an accuracy over 90%. however when i load my model on my raspberry pi and input the image of the same shape (1,150,150,3) it returns an error. Below is the code loaded on the raspberry pi to predict from the keras model.

    data = numpy.fromstring(stream.getvalue() , dtype = numpy.uint8)
    image5 = cv.imdecode(data , 1)
    print(image5.shape)
    #cv.imwrite('uhhu.png',image5)
    img = cv.resize(image5,(150,150))
    x = img_to_array(img)
    x = x.reshape((1,) + x.shape)
    x = x/255
    x = numpy.float32(x)
    print(x.shape)
    score = loaded_model.predict(x)
    print(score)


texts: why variational inference in variational autoencoder if dealing with simple graphical model

texts:
As far as I am reading about VAE, I always see a graphical model of Z --> X,

I know P(X) can be intractable, but that would be if we would have many dependencies. But here is only Z. So why variational inference at first place?:)


texts: How to build a network that detects common object in images?

texts:
I have a group of images. All contain some common object (let's assume all at the same size and no other alterations). I want to train a CNN that will learn the filter for the common object and gives me back a boolean flag of whether it exists in a given image or not.

(for simplicity, I assume the "object" is smaller than 5x5 pixels)

I thought of something like:

A Conv layer with 1 filter of 5x5
A MaxPooling layer with pool_size = input size (meaning - returns a single value: the maximal activation of the filter)
Output max activation (or a variation of it)

Some questions:

Would you output the raw max activation and use a threshold to decide? sigmoid of it? or force the network to output "object" and "no_object" classes and do softmax?
Quite related - what loss function?
Is it a problem to have a large MaxPooling layer (one that reduces all activation to a single one)? What happens to the gradient on backprop?
It seems like it would just go randomly to one of the activations and I'm not sure that it can "learn" the object that way..

I'm wondering how to look at this problem as it's not really a 2-class classification problem. Even if I can come up with images that doesn't contain the object, there is nothing to "learn" from this class except of the fact that it doesn't contain something (which is a valuable information here, I must agree).

I've tried playing with it on simulated small black-white images and a cross "object" that I randomly added. Couldn't get the filter to learn the cross pattern although on some tries it got the horizontal line..

Any idea would help! Thanks!


texts: numer.ai: how does their leaderboard system work?

texts:
There is a data mining competition website called numer.ai.

Presumably, behind the website is a hedge fund which makes use of the predictions that people send. People within the 100th top places continuously make money, until the next dataset is revealed, and the competition resets.

What I don't understand is that websites like Kaggle avoid overfitting by having a public and a private leaderboard. The private leaderboard is only revealed at the end of the competition and only then are prizes handed out.

Numerai says under rules that it uses the same approach. Quoting:

  If models overfit the public leaderboard and perform poorly on the
  private leaderboard, those users will suffer penalties potentially
  eradicating all gains. This discourages overfitting.

What do they mean by "penalties" being applied? I know someone who is making money on that website by continuously playing the public leaderboard. Does this mean that if he does not withdraw his bitcoin he is in danger of losing what he has earned in case he is overfitting?


texts:
So I have a DataFrame which consists of order data from customer on an exchange.

I have a column Dollars which is the dollar value of all trades conducted by customers in different currency pairs. I want to train a classifier on this data, so I'm normalizing the Dollar column to get the values into the same range, but I think the normalized values seem quite.. odd. This is a snippet of my DataFrame:

Dollars   | Normalized dollars
181447.50 | 9.10975e-06
281885.00 | 1.41523e-05
290786.00 | 1.45992e-05
70923.00  | 3.56076e-06
1121169.54| 5.62894e-05

These values seem tiny.

I've used sklearn.preprocessing.normalize for this.

It's worth mentioning that the lowest Dollars value is 0.06 while the largest is 5,605,847,772.52, which I assume is the explanation, but I was expecting a 0-1 range, but my largest normalized value is 0.157. Should I do more filtering on the dataset to remove extreme outliers as a general rule?

UPDATE:
Applying the base 10 logarithm to dollars yields some more manageable results, though the normalized values are still fairly small. ScaledDollars is the result of using sklearn.StandardScaler on Log10Dollars:

Dollars   | Normalized dollars| Log10Dollars | LogNorm   | ScaledDollars
181447.50 | 9.10975e-06       | 5.258751     | 0.00384193| -0.761916
281885.00 | 1.41523e-05       | 5.450072     | 0.00398171| -0.573336
290786.00 | 1.45992e-05       | 5.463573     | 0.00399157| -0.560028
70923.00  | 3.56076e-06       | 4.850787     | 0.00354388| -1.16404


texts: Statistical kernels for image similarity

texts: I am looking for statistical kernels that could outperform the Gaussian kernel for comparing the similarity of a single image to a batch of other images.


texts: Back propagation and Structure of a Neural Network in scikit-neuralnetwork

texts:
I am trying to learn Neural Networks using scikit-neuralnetwork framework and I know basics about Neural Networks and now trying to implement it with scikit-learn. but I am confused on 2 points.
1- what is the structure of this NN given below? Somehow, in some examples felt to me, some people don't put input layer as a layer. Otherwise, I am thinking this as a 2 layer NN has input layer with 100 nodes and 1 node at the ouput layer.
from sknn.mlp import Classifier, Layer

nn = Classifier(
   layers=[
    Layer("Maxout", units=100, pieces=2),
    Layer("Softmax")],
learning_rate=0.001,
n_iter=25)

nn.fit(X_train, y_train)

2- Does scikit-neuralnetwork do back propagation within the code that I put above?
Thank you!


texts: How to append new numerical features to an embedding from word2vec, such that KNN on embeddings is not biased for one feature?

texts:
I am working on similarity calculation between entities of similar type. For each entity I am able to make a vector that comprises of multiple vectors itself.

A = 50*1 vector
B = 100*1 vector
C = 50*1 vector
D = [age, gender, x-feature, y-feature, z-feature]

Entity = [A B C D], basically concatenating all of those.

My problem is that: since they come from different spaces, each would have different orders of magnitude.
If I run KNN on these vectors, I doubt that results are going to be governed by features of one space only. What should be done to get best KNN results? What "normalisation" would be best here?


texts: Standard Scaler drops accuracy significantly in Scala Spark

texts:
I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly.

Without the accuracy is ~90% (on training set), 77% (on testing set)

With Normalizer or Standard Scaler is 19% (on training), 0% (on testing)

I have the feeling that it's not related to the normalization per se, but more like a code bug.

Here is part of the code:

val labelIndexer = new StringIndexer()
  .setInputCol("label")
  .setOutputCol("indexedLabel")
  .fit(assemblerDF)

val featureIndexer = new VectorIndexer()
  .setInputCol("features")
  .setOutputCol("indexedFeatures")
  .setMaxCategories(4) //if column has more than 4 categories, then use it as continious instead
  .fit(assemblerDF)

val labelConverter = new IndexToString()
  .setInputCol("prediction")
  .setOutputCol("predictedLabel")
  .setLabels(labelIndexer.labels)

val scaler = new StandardScaler()
  .setInputCol("indexedFeatures")
  .setOutputCol("scaledFeatures")
  .setWithStd(true)
  .setWithMean(false)

val ml = new LogisticRegression()
  .setMaxIter(15)
  .setFeaturesCol("scaledFeatures")
  .setLabelCol("indexedLabel")

val pipeline = new Pipeline().setStages(Array(labelIndexer, featureIndexer, scaler, ml, labelConverter))

Where then I pass the Pipeline into a CrossValidator. Any idea what is going wrong here?


texts: Is there a case that random forest data (a bunch of trees) consumes MB or GB of memory?

texts:
Is there any case that random forest (a bunch of trees) consumes too much memory in practice?
I'm wondering why my scikit-learn consumes large amount of memory.


texts: Reducing High generalization-error on industrial fault data

texts:
I have a industrial dataset containing labeled machine data for fault classification(3 classes: 1 ok, 2 for faults). The problem is that i have less (~16) different machines, thus iam currently having instance shift problems: The accuracies on the training set is perfect but validation on holdout instances fails. As background information, the machine data is time series, where i extracted statistical (domain specific) features from (14 in total). This features are my dataset for classification. I tried different model types, like SVM and MLP.
My question to you is: I tried to reduce the generalisation error with methods like Dropout and L1L2 regularisation - But this does not work well as it causes the training accuracy to stay low.
It would be very helpful if you could get me some hints and tips would i could try.
Thanks in advice.


texts: Why can't ARIMA model large lags and/or long range dependence?

texts: ARIMA cannot model large lags(obtained from autocorrelation plot) and long range dependency(hurst exponent H > 0.5). Why is it so?


texts: RMarkdown 2*5 Matrix somehow multiplying by itself?

texts:
There is a matrix m:
m <- matrix(1:10, nrow = 2, ncol = 5)

Somehow when I attempt m * m it gives a values, instead of arguing that these dimensions will not allow for multiplication to occur. The value is that each element of this matrix is squared. Do not understand why this was happening.


texts:
Given a question, with some model answers and their grades, and possibly other 'learning material' content.

What tools or technologies can be used, to implement a system that evaluates a student answer?

I am new to machine learning.

Thanks.