texts:
I got question about preparation data for my ML algorithm.
Raw data has format similar to:

{
  "finances": [
    {
      "assets": 1230.39,
      "investments": 3245.39,
      "netProfit": 8765.45,
      "year": 2017
    },
    {
      "assets": 111.11,
      "investments": 222.22,
      "netProfit": 333.33,
      "year": 2016
    },
    {
      "assets": 1111.11,
      "investments": 2222.22,
      "netProfit": 3333.33,
      "year": 2015
    }
  ],
  "someValue": 123.45,
  "title": "Hello!"
}

And I am wondering what is best way to pass data about finances to my algorithm (Number of years can differ from 0 to 8).

I was thinking about making every parameter for every year new attribute, but this would lead to have many not null values for some cases, because not every record has 8 years of financial history.

My question is: What is best way to handle such data in Input data (In fact every of finances elements would have more values in it (around 10-15).


texts: How to fill missing latitude and longitude values in time series data?

texts:
I have time-series data like this:

date
longitude
latitude

01/01/2010
-5.42766
107.5784

02/01/2010
-6.42728
104.5245

07/01/2010
-7.42702
105.5816

14/01/2010
-4.42728
99.57834

17/01/2010
-6.41523
103.5562

...
...
...

31/12/2013
-4.42728
99.57834

This is location data (longitude and latitude).
Some of the data is missing (ex. there is no 03/01/2010 to 06/01/2010).
I want to fill in the missing values, what is the best method to fill the missing values?
Since this is location data, I don't think using the mean value is make sense to fill the missing values.


texts: Trained vs randomly initialised cnn model. Difference in inference speed

texts: We have 2 models written in Keras. We then convert them to coreml format and test the inference speed of each one on iPhone 7. The model architecture is the same for each of the two. The 1st model is trained on a set of images. the 2nd model's weights are randomly initialized. The number of params is the same for each model, however the speed (in fps) is different (21 vs 13 fps). How's this possible? Why is it so that the trained model is slower, despite the fact that the number of weights and the structure is the same?


texts: can machine learning/Deep learning used to minimize an objective function?

texts: I have data of construction site and am wondering if i can use machine learning to reduce the cost it takes to build a building. But, as far as i know, Machine learning can only does function approximation.  But, i remember that google used AI to reduce the costs of the data center. This and  This are the article.


texts: Is Categorical Crossentropy Loss always bounded in the [0, 1] interval?

texts:
Is Categorical Crossentropy always bounded between 0 and 1, or is it possible that during training of a Neural Network it can get higher values?

More specifically, I'm referring to the TensorFlow 2.0 function.


texts: Binning of Continous Predictor and Predicted Variables

texts:
My problem has three categorical variables C1,C2, C3 and one continous variable X, predicting a continuous outcome Y. I can visualize the problem with the following reproducible code (apology for the badly written code):

library(data.tree)
i = expand.grid(c("A","B"),c("C","D"),c("E","F"))
i = i[order(i[,1],i[,2],i[,3]),]
i[,4] = c(1:8)

t = expand.grid(c("A","B"),c("C","D"),c("E","F"),seq(0,1,0.1))
t = t[order(t[,1],t[,2],t[,3]),]
t = join(t,i, by = c("Var1","Var2","Var3"))
t$Var5 = runif(nrow(t))
t$pathString <- with(t, paste("Tree", Var1, Var2, Var3, V4, sep="/"))
plot_tree <- as.Node(t)

plot(plot_tree)
ggplot(data = t, aes(x=Var4, y=Var5)) + geom_line() + facet_grid(~V4)

The tree categorical variables show 8 possible path combinations:

And within each path there is a distribution of Y depending on a continuous variable X:

I would like to bin both the continuous Y and X variables such that I am left with a more concise decision tree. There is method to this madness as in my actual problem, some categorical paths will become insignificant due to no movement in the predicted Y beyond the established volatility threshold of 5%.

I can manually obtain bins by brute force, but is there a binning algorithm that supports binning of both predicted and predictor variables? I have looked at smbinning and my knowledge in this area of algorithms is limited. My actual problem has many categorical and continuous variables resulting in a more complex structure, but I would like to convert it to a decision tree result which will help me understand the significance and movement of variables better.


texts: How to choose a model for this cross-validation curve?

texts:
I'm using GridSearchCV to tune hyperparameters for a Logistic Regression multiclass model.

I read on Kaggle that you should choose the hyperparameter that results in the lowest discrepancy between the CV-score and the training score, but in this case this leads to a very low score.

How should I choose the proper C value to ensure generalisability of the model but also high model performance based on the CV-curve below?

From my understanding opting for low discrepency between the two scores ensures the ability of the model to be generalised to unseen data. But on the other hand I want a score as high as possible on unseen data.

Thanks for any help!


texts: Football match prediction using regression

texts:
I am trying to predict goal difference of football matches in keras using a single layer Neural Network. I used mse as metrics and its a low value aroung 0.05 but some predictions has huge difference.

I tried sklearn LinearRegression too but results were almost same. Also I have tried multiple layers network too but results were almost same too.

Any ideas to lower the predicted difference?


texts:
I was reading about One Class Classification but had two doubts -

1) How does One Class Classification work because the training data is of only a particular class so in that case, during testing, the model would always predict that any test data belongs to that class only.

2) And since it does work then is it not perfect for Spam Detection because we can have data for spam emails but how could we find any amount of training data on Non Spam to possibly cover all cases for Non Spam E-mails.
As any general email is a non spam one and only certain mails are spam.


texts: I have created a prediction model and on the one hand I have to compare it with other baseline models, and on the other hand, I have to compare it with the ideal approach (supported by additional data), so I would like to know how I can call it (antonym of baseline) in the research paper.


texts: Size drastically increased when images converted to hdf5 format

texts: I converted Kaggle Dog vs Cat dataset from image to hdf5 format, in order to fasten the learning time. To my surprise, 540MB training data converted to  more than 30GB of data in hdf5 file format. Why is this so. Also I have not much space in my laptop, is there any other method to fasten my deep learning process without consuming too much space.


texts: Why does image segmentation benefit from fuzzy clustering?

texts:
In many image processing papers, I've seen that they used fuzzy logic for segmentation
I wonder how fuzzification impact the result that made Fuzzy-C-Means better than ordinary K-Means.

PS. If possible could you provide me the sample data sets for case study


texts: predict multiple letters in pixels matrix

texts:
I have a multilayer perceptron model that is trained to recognize handwritten English letters from an image. In the training set each image matrix had 784 pixel values. The labels of these images corresponded to the place in the alphabet. For example: A is 1, B is 2 and so on. In total there were 26 classes. From the pixel values and the corresponding label the model is now capable to predict a letter.
Now I am trying to predict corrupted images with the model. Each corrupted image has 4 letters in it. Each corrupted image has a dimension of (30, 140). I have tried to come up with a solution, but I am at a loss at this point.
So let's say I have a corrupted image with pixel values that represent the letters VRLX, I want to predict with my existing model that the label is 22181224 (v = 22, r = 18, l=12, x=24). The current model is capable to predict a letter independently. However, I fail to fit my model with these corrupted images. Does anyone have some suggestions that can send me in the right direction?


texts: Predicting a numeric variable from many numeric variables, how to choose a proper structure?

texts:
I need to estimate the value of a one numeric variable from the values 8 numeric variables

y = f(x_1, ..., x_8).

I have an historical dataset where I can relate thousands of y values to the corresponding x, that will grow with time potentially to millions of records.

Of course I don't know the relationship between x and y, for sure I know it is not linear with x (I'm guessing y depends quadratically on some xs) and in principles it could be that there is no analytical expression at all for this relationship.

For these reasons I guessed to solve the problem with a Neural Network but I'm pretty new to this topic so I'm trying to figure out (with very little success) which model should be suitable for my problem, i.e. how many hidden layers to choose and how many neurons for each layer? Should I put ReLUs or other functions between layers?

Of course I'm not interested just in solving this particular situation but in understanding the possible choice criterias!

Also I would be interested in this: if I know that y probably depends on combinations or functions of the x_i such as y = f(x_1 - x_2, x_3**2, ..., x_8) should I put directly those guesses as inputs in the net or plain x_8s?

If it's useful in any way, I'm pretty familiar with python and I would like to implement this solution with PyTorch for my personal learning benefit. An hint for the Model class structure would be most appreciated.

Thank you


texts: How to handle a regression problem with skewed target and only few high values?

texts:
I'm currently tackling a regression problem with skewed target variable (presented below).

Naturally, my first idea was to transform the target with natural logarithm as it'll probably help both linear regression or decision-tree-based algorithms. The second idea is to prepare a validation scheme similar to stratified k-fold cross-validation with target binned into n groups. However, my concern is that I have only few highest values:

Therefore, my test set and all validation sets error are highly dependent if one of these 4 extreme values are drawed placed within them or not. That makes it hard to obtain reliable true error estimate.
Is there anything more I can do to handle that issue?


texts: I have dataset with two label class (good and bad), I want to apply K Means on my dataset using python, should I use that label dataset or I have to delete the label class column ?


texts:
I'm trying to do hyperparameter tunning for random forest regression model.
My question is- is there any order I should do it? like starting with specific parameter and then move on on the other?  should I check the model each time with one paramter only or for each parameter i'm  checking I should add the ones I have found already?
Exmaple: if I run model with only n_job -1 and min_sample_leaf, and I got the best value for min_sample_leaf and now I want to check the max depth, and after that more parameters, should I put already the best value I found for min)sample_leaf? or I should check it without any other parameter and in the end create model with the best values from each one?


texts: I think a learning rate schedule would be counter-productive with AdaBelief. Am I wrong?

texts:
I am inclined to believe the concept of a learning rate schedule is overcome by the improvements of Adabelief over Adam.
My code is on Github; please check it out and attempt to replicate my results, which I will probably post as an answer if I do not hear anything from the community by the time my experiment is completed.


texts: Is there any text similarity databse available for phrases?

texts:
I want to train my application for phrase similarity. I want my model to predict similarity score for phrases as shown in below examples.
ex-

International Business Machines = I.B.M
Synergy Telecom = SynTel
Beam inc = Beam Incorporate
Sir J J Smith = Johnson Smith
Alex, Julia = J Alex
James B. D. Joshi = James Joshi
James Beaty, Jr. = Beaty

Is there any dataset available to train this type of model?


texts:
I have some data that I would like to cluster with k-means.

One of the features is the hour of the day.

The problem is that the hour '23' is considered far from the hour '0'.

How can I map the data so that the boundary will be cyclic?


texts: Predicting Intent to do X with a confidence score or intent percentage score?

texts:
I have a data set like:

did_purchase  action_1_30d action_2_20d action_2_10d ....
   False            10          20            100
   True            ....etc

Where did_purchase shows whether the customer purchased or not, and the columns indicate the volume of actions taken before the purchase (or non-purchase) event.

So, for the first row the customer did 10 of action_1 within 30 days of the purchase event, but didn't purchase in the end.

I have been using sklearn's LogisticRegression to predict the did_purchase false/true, and can get about 89% accuracy, which is nice.

However, I'd like a percentage intent score instead. So it could say user-321 has a 46% chance of purchasing in the next 10 days.

What would be a good algo/approach for this?


texts:
While running decision tree, I have unbalanced data. The data balance is 93%(Class 0) to 7% (Class 1).
Now when I am plotting decision tree to understand the factors contributing to class 1, then I find that most boxes are for 0 class (as it is 97% of the data).
Also, after pruning, very few contributors are there which are contributing to class 1. How can I get the factors contributing to class 1?


texts: Text classification analysis based on similarity

texts:
I have been reading a lot of literature regarding text classification and different approaches/models, especially using Python language, but probably I am still missing something on how to build the models and the steps involved.

I have multiple datasets, each of them on a particular topic. These datasets include news and fake news, manually flagged at the moment.
I have collected texts from different sources about similar topics (using keywords) and now I would like to try to build a model that can allow me to classify a news as real or fake automatically.

I have thought that it could have been useful to study the frequency of words and punctuation, and also similarity, trying to group similar texts based on same conditions (for example, similarity between texts as in plagiarism).
I have used similarity (Jaccard or cosine) for comparing texts rather than words, but I do not know if this is the right approach and how I should create a model based on that.

Probably the best method would be to build a logistic regression model using a binary variable, but I also read a lot of naive bayesian modes and nn models.

Do you have any example on how I should apply text similarity for classification or further relevant information about that?


texts: Approximate Text compression by training model?

texts:
I have a key-value form of dataset. Basically two sets of integers in many-to-one mapping. Like f(123) corresponds to 4652, f(24) corresponds to 12 and so on. Size of this dataset is say around upto 10^7.

This is too large. So my Idea is to train this dataset on some model and store trained model's parameters only, so when I have to answer say f(24), I can pass 24 as input to my model and print model's output. My thinking was overfitting some model on the dataset could possibly result in high accuracy of say greater than 99%. I strictly want size of parameters to be less than 50,000 bytes.

I tried making vanilla neural network but error percentage is way too high.
I found an interesting blog trying same approach as mine. Kindly let me know if something is unclear to you in comments.

Concrete Example Problem
To give you a concrete example of what I am trying. Here is the DataSet, ith line stores f(i) in this file. I am trying to make a code which is of 50k bytes and can answer queries f(i). Ofcourse while training there are no limits, but after training it should be a file of 50k length at max.


texts:
Say I want to do supervised learning of response variable $y_k$ (continuous) and feature variables $(x_k, z_k, a_k, b_k)$ where $k$ is the sample index.
Instead of learning the general form
$$ y \sim f(x, z, a, b),$$
I would like to learn the form
$$ y \sim A(a,b) x + B(a,b) z + C(a,b). $$
This is because the coefficients $A$ and $B$ are more critical than the general (non-linear) form, and I know that the structure of $y, x, z$ is linear, but the coefficients may vary depending on the values of $a$ and $b$.
I am wondering if there is any learning method suitable for this situation.
If I use neural networks for learning $A(a,b)$ and $B(a,b)$, is it possible to optimize the RSS with tensorflow? If I design the model with tensorflow, note that the last layer is not a typical NN layer in the sense that the products, $(A, B)$ and $(x, z)$, depends on the feature variables.
Much appreciate your help in advance.


texts: Linear transformations making independent features dependent

texts:
I read about features and some relevant topics recently. I ran into an easy but very advanced question:

Why can two independent features become dependent after applying a linear transformation?

I think this is false, because:

For $x_1$ and $x_2$, after linear transformation $y_1=ax_1+b$ and $y_2=cx_2+d$, $y_1$ and $y_2$ are independent.


texts: How should I improve my Vectorized Gradient descent linear regression model?

texts:
I wrote a vectorized Gradient descent implementation of the linear regression model. The Dataset looks something like:

It's Not Working properly as I am getting negative R Squared error I don't understand why ?? Should I decrease Alpha or No. of Iterations or is there any problem in my implementation what should I do?

My Regression plot looks something like below I don't know why I am getting such a line.

My Cost Function Error plot with respect to the number of iterations in Gradient descent looks something like below

R Squared Error is: -3.744682246118262

My Code Snippet:

import numpy as np
from sklearn.model_selection import train_test_split
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

def CostFunction(Theta,DataMatrix):
    Size = DataMatrix.shape[0]
    Error = 0
    for i in range(0,Size):
        Feature = np.vstack(([1],np.array(DataMatrix[i][:-1]).reshape(-1,1)))
        Error += (np.transpose(Theta).dot(Feature) - DataMatrix[i][-1]) ** 2
    return (1/(2*Size))*Error

def GradientDescent(Theta,Alpha,DataMatrix,Iterations):
    Progress = []
    Iterate = 0
    Size = DataMatrix.shape[0]
    Error = np.zeros((DataMatrix.shape[1],1))
    while(Iterations):
        for i in range(0,Size):
            Feature = np.vstack(([1],np.array(DataMatrix[i][:-1]).reshape(-1,1)))        #Last Entry is Label Thats Why
            Error += (np.transpose(Theta).dot(Feature) - DataMatrix[i][-1])*Feature
        Theta -= Alpha*(1/Size)*Error
        if(Iterations % 10 == 0):
            Progress.append([Iterate,CostFunction(Theta,DataMatrix)])
            Iterate += 10
        Iterations -= 1
    return [Theta,Progress]

def ProgressCurve(Progress):
    Progress = [[i[0],i[1].ravel()[0]] for i in Progress]
    sns.lineplot(x = np.array(Progress)[:,0],y =  np.array(Progress)[:,1],marker = '*')
    plt.show()

def Prediction(Theta,Test):
    Predicted = []
    for i in range(0,Test.size):
        Feature = np.vstack(([1],np.array(Test[i]).reshape(-1,1)))
        Predicted.append(np.transpose(Theta).dot(Feature))
    return Predicted

def Error_Metric(Actual,Predicted):
    Actual = np.array(Actual,dtype = 'float64').reshape(-1,1)
    Predicted = np.array(Predicted,dtype = 'float64').reshape(-1,1)
    Error = (Actual - Predicted) ** 2
    Variance = (Actual - np.mean(Actual)*np.ones((Actual.shape[0],1))) ** 2
    return (1 - np.sum(Error)/np.sum(Variance))

def RegressionLine(X,Y,Orig_X,Orig_Y):
    Y = [i[0].ravel()[0] for i in Y]
    sns.scatterplot(x = Orig_X,y = Orig_Y,color = "blue")
    sns.lineplot(x = X,y = Y,color = "red")
    plt.show()

X = 2*np.random.rand(1000)
Y = 4 + 3*X + np.random.randn(1000)
X_Train,X_Test,Y_Train,Y_Test = train_test_split(X,Y,test_size = 0.3,random_state = 0)
DataFrame = pd.DataFrame()
DataFrame['X'] = X_Train
DataFrame['Y'] = Y_Train
DataMatrix = DataFrame.as_matrix()
ThetaParams = np.random.randn(2,1)
Theta,Progress = GradientDescent(ThetaParams,0.001,DataMatrix,50)
Prediction_Out = Prediction(Theta,np.array(X_Test))
Error = Error_Metric(Y_Test,Prediction_Out)
ProgressCurve(Progress)
RegressionLine(X_Test,Prediction_Out,X,Y)
print(Error)


texts: Don't understand why I get an inverse ROC curve for SVM (Python)

texts:
I build an SVM classifier but get an inverse ROC curve. The AUC is only 0.08. I've used the same datasets to build a Logistic Regression classifier and a Decision Tree classifier, and the ROC curves for them look good.
Here are my codes for SVM:
from sklearn.svm import SVC
svm = SVC(max_iter = 12, probability = True)
svm.fit(train_x_sm, train_y_sm)
svm_test_y = svm.predict(X = test_x)
svm_roc = plot_roc_curve(svm, test_x, test_y)
plt.show()

Could anyone tell me what is wrong in my codes?


texts: Has Gradient decent any effects on the actual predictions or designed for practical convenience?

texts: If we consider Batch Gradient, Stochastic Gradient, Mini-batch Gradient, will they affect on the actual predictions? as we know they always attempt to reach a local minimum.


texts: Predict ratings for Item Based Collaborative Filtering


texts: R: Producing multiple plots (ggplot, geom_point) from a single CSV with multiple subcategories

texts:
I have a collection of bacteria data from approximately 140 monitoring locations in California. I would like to produce a scatterplot for each monitoring location with the Sampling Date on the Y-axis and the Bacteria Data on the X-axis. The Sampling Date, Bacteria Data, and Monitoring Location all reside within their own column.

I've come up with the below code:

## Create List of Files ##
filenames <- list.files(path = "C:\\Users\\...")

## Combine into one CSV ##
All_Data <- ldply(filenames, read.csv)
All_Data$SampleDate <- as.Date(All_Data$SampleDate, origin="1899-12-30")

## Save CSV for possible future use ##
write.csv(All_Data, file= "C://Users//...", row.names = FALSE)

## Construct Plots ##
ggplot(All_Data) + geom_point(mapping =aes(SampleDate, Total.Result)) + facet_wrap( ~ Identifier) +ylim(0,20000)

but this produces the below plot where every single location is crammed into the plot frame. Is there a way to print each plot individually to a folder?

I tried to incorporate the subset function like so

ggplot(All_Data) + geom_point(mapping =aes(SampleDate, Total.Result)) + facet_wrap( ~ subset(All_Data, Identifier)) +ylim(0,20000)

but received the error

Error in subset.data.frame(All_Data, Identifier) :
  'subset' must be logical

Alternatively, is it better to do this through some sort of loop through the original 15 csvs that I've combined together? I would still have the challenge of creating one plot per monitoring location. Thanks in advance for any suggestions!


texts: How would you build a big production ready image training dataset from scratch?

texts:
How would you most likely create a large production ready image training dataset from scratch including annotations for a image classification task?
We will take a large amount of images (~1 million) with industrial cameras and save them in a S3 bucket. Do you think a data lake infrastructure is necessary?
In your opinion, what are the most suitable methods for annotating the images in the shortest possible time (bounding boxes not needed).
Solutions that I have been able to find so far are the following:

Use an open source web based image annotation tool like make-sense or LOST (Problem: who will annotate the images? These tools doesn't seem to be perfect for big amount of image data). See also awesome-data-labeling
Build a gamificated web application and let users annotate images and earn discount codes to motivate them
Use third party tools with annotation workforces like Playment, Labelbox, Amazon Mechanical Turk

Are there any options I missed? In principle, it would be possible to pay for the annotation, but should be avoided or kept as small as possible.
Are there things that should be considered architecturally with such a large database?


texts: How do I learn encoding of a text that is encoded at character level?

texts:
Encoded text at character level:

  tvletwgzkrqvuhtwamuluhpkskpmpmiwtvuhamqvmviwlrvikquhtwamuluhqgvipmmvulkriwpmqvtwleuhamqvmviwlrvikquhtwamuluhqgqvqvtwviezlemvxeuhamqvmviwlrvikquhtwamuluhpkskvieniwlrvikquhqvmvuhqgpmpmiwletwulenokuhxepmuhtwiwululentvuhtwamuluhvimvuhsktwlemvezskenuhtwtvuhulqvkrezuhamypmvamdfuhulenamguuhraskvipmyptwqvuhtwamuluhxepmuhvimvenulgzenypuhenuhsatvuhvipmdfuhqgletwsklepmuhulqvlemvxeuhtwamuluhxepmuhtwiwululentvuhenuhqvmvuhpmpmiwletwulenok

Map to one of 10 classes: 0-9.


texts: High Bias in 1dimensional CNN with high training Loss

texts:
I am working on a project where I have to classify the good and bad packets. I am modelling my data with 1 dimensional CNN u where i have to predict good packet(0) and malicious packets(1,2,3), i have upsample the data with class_weight parameter while training the model. I have reshaped my data(6,1) and i have used a Conv1d layers where the first one has 124,second one has 64 and the third one has 48 layers. In each middle layer I have used a kernel_size = 5, padding =same, and activation function relu. Afterthat I have used a Flatten, with BatchNormalization and I have added a two Dense layers with 128 and 64 respectively with activation function RELU. The output layer has the number of classes(4) and activation function softmax. However the model has a high Bias and the training loss is much higher than the validation loss. I really need help...
Thanks in advance!
My original data has 168000, however i am first trying to get the model working on small data
Results of the model.


texts: What variance threshold to consider in feature selection?

texts: Consider a numerical dataset with continuous variables, that has been scaled to end up with values in the [0,1] range. How can I compute a reasonable variance threshold for all the variables?


texts: Prerequisites for Elements of Statistical Learning

texts:
I am working through Elements of Statistical Learning, and unfortunately have found great difficulty in following the math. I have taken the standard series of Calculus courses (i.e., up to multi-variable calculus) and linear algebra, and have taken introductory (non-calculus-based) statistics and econometric courses.

Please  let me know if there are any courses or textbooks you would recommend in order to get my mathematics/statistics knowledge up to speed. For example, I am considering working through the relevant portions of Statistical Inference (Casella, Berger) as a prereq.

Thank you!


texts:
Language model(LM) is the task of predicting the next word.

Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.

Or both with-encoder and without-encoder can do the LM task?


texts: How to use weights from our data set properly? Should we use them at all?

texts:
The census data set I'm using: https://archive.ics.uci.edu/ml/datasets/Adult

So, I'm currently using this census data to make observations and predict whether someone is married or not. However, when I plot a categorical graph for the race, it's overwhelmingly white and I was going to just not take race into account at all. Before I did that, though, I realized that I didn't take into account the "final weights" in the data set, which shows the number of people the census believes that the entry represents.

So, I have a few questions:

Should I use these weights when making observations? Am I interpreting this correctly? Or should I just ignore them?

If I were to use these weights, how exactly would I go about using these weights? I want to be able to predict whether someone is married or not based on a few of these factors given, but I'm having trouble figuring out how I should incorporate these weights.

Also, would it be dumb to divide all the final weights by 100 or something similar? All of these weights are in the hundreds of thousands, and I was thinking that if we divide the weights by some constant, then we can just add that number of rows (for each row based on its weight) to our data set.

I'm new to ML in general so forgive me if this is a newbie question.


texts: Adding hand-crafted features to a convolutional neural network (CNN) in TensorFlow

texts:
Let's say I want to add a few hand-crafted features to a convolutional neural network CNN in TensorFlow.

The CNN can be a simple one as described here.

Naturally I'd like to add these features right after the second pooling and right before the first fully-connected layer (FC1 in the example).

Is that easy to express my method in code? I'd have to append my features to the h_pool2_flat vector/tensor:

h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)


texts: When is centering and scaling needed before doing hierarchical clustering?

texts:
I am working on a clustering project where we have collected protein data from over 100 patients samples. This data is normalized and log transformed. The goal is to cluster samples based upon their similarities, I am using hierarchal clustering and trying out combinations of distance metrics and clustering algorithms. (We haven't made a decision on distance method or clustering algorithms) My question is related to the centering and scaling, Is it absolutely necessary to both scale and center the data?, even in scenarios where all the data is coming from the same platform and with same units of measurement.

Appreciate your input on this one.

Thanks


texts: Data calculation/normalisation /data summarisation

texts:
Clustering is done based on houses data and say suppose cluster 1 contains 5 houses.
Appliance 1 is used in 2 houses out of 5 and appliance 2 is used in 4 out of 5 houses.Each appliance produces an error . Need to find sum of the errors of each appliance in each cluster. Summing up of errors at appliance level would not give us the correct picture as appliance 1 is used in 2houses but appliance 2 is used in 4houses. How do we handle such data? How we get the total errors produced from each appliance for a particular cluster?


texts: What is an example of normalization (making the row unit norm)?

texts:
I am learning about standardization and normalization concepts for feature engineering.
Standardization is done for example using z-score where based on the mean and std deviation we re-calculate the values so that the mean is 0 and std deviation is 1. This is a column wise operation.
Where as Normalization is performed row wise (to make the entire row unit norm) – thus making it good for calculating things like cosine similarity. I am looking for an example of what it means to make the row unit norm.


texts: which algorithm will be good for detecting and recognition of faces from variety of angles

texts: i am building a face recognition app for my class attendance system , i collect training data from social website like facebook, instagram and other, as you can see the images i got from there is not usually front facial but at variety of angle. i was using haar_cascade for face detection , but it is not good for tilted face, can anyone suggest me the good algorithm for face detection through which i can detect face at variety of angle.


texts: Optimum way to train a keras nn by a dataset

texts:
I have a training dataset with all the images and no sub -directories , also the images are named as random long ids and the target labels are stored in another file labels.txt with ids and the corresponding  labels.
How should I implement a generator using keras to pass these images to the model without loading the dataset in ram.

edit:

One way I see it as

sorting the image names and labels so that i can use ImageDataGenerator.flow


texts: Is it necessary to cap test and control metric at same percentile to treat outliers?

texts: I am working on an AB test and the metric I am using to compare test and control is revenue. Both of my test and control have outliers. I am trying to treat outliers and have found that to do so, I need to cap test at 99.98 percentile and control at 99.95 percentile. I need help to understand if by capping test and control at different percentile values, am I introducing bias?


texts: Why not use more than 3 hidden layers for MNIST classification?

texts:
Many works use 2-hidden-layer neural networks for classifying MNIST handwritten digits sets.

To improve accuracy, other techniques (dropout, ReLU.. etc) have been used without increasing the number of hidden layers.

Is there any reason not to use more than three hidden layers?
for example, overfitting?


texts: I'd expect that for a precision-recall curve, precision decreases while recall increases monotonically. I have a plot that is not smooth and looks funny. I used scikit learn the values for plotting the curve. Is the curve below abnormal? If yes, why and how can I correct it considering scikit learn automatically sorts the true and predicted labels. If the plot is OK, how best do I explain this behaviour?


texts: scikit-learn sample weight interpretation

texts:
I'm trying to use scikit-learn to plot a confusion matrix from raw data I have obtained (contains just predictions and ground truths).
The data contains a total of 4 classes: [0, 1, 2, 3]
One of the parameters of the confusion matrix is sample weight, and I noticed the shape has to be equal to the number of samples in the data.
Considering that the classes are imbalanced, and given the class ratio to all of: [0.5, 0.3. 0.15, 0.05], what would I need to pass to sample_weights to account for the class imbalance in evaluation, when I'm also trying to normalize the matrix?


texts: Predict a label based on multiple rows each one case?

texts:
I need to do a classification of binary labels. The dataset represents a loan procedure. So one row is a certain moment of a process for a certain case.

The dataset consists of several cases more than 9000 (attribute CaseID)

Example:

CaseID 1: consists of a process of 8 rows (each row represents a moment) and in the end the loan is accepted or refused.

CaseID 2: consists of 20 rows

etc etc

I was already able to combine these rows in one row per case and then predict. But I would also like to keep the order of these processes as maybe it might increase the performance of the models (random forest and XGboost).

My question is: How can I say to the model use the first 8 rows of this case to  tell me for this case what its label is and use the first 20 rows for the next case and predict its label etc etc?

I hope the question is clear?


texts: Should I set higher dropout prob if there are plenty of data?

texts:
I have some excessive amount of data for the size of NN I am able to teach in a reasonable time.

If I feed all the data into the network it stops learning at some point and a resulting model shows all signs of being overfit.

Intuitively if I increase dropout prob the model should learn less aggressively from data and gain from more data being fed into it.

Is my logic sound?


texts: Differentiable loss function for ranking problem in regression model

texts:
In regression problem, we may need a loss function to measure the relative ranking accuracy between targets $y$ and predicted values $y_{pred}$. Abviously, the simple MSE does not consider such ranking relations.

A straight choice is the so-called IC (Information Coefficient)

$$IC\propto corr(\text{rank}(y), \text{rank}(y_{pred}))$$

which use the correlation between two ranks. However, rank function is not differentiable, thus it can't be used in loss function for regression which uses gradient propagation to update the parameters.

Another choice might be a modified pairwise ranking loss designed for regression

$$\text{Loss}=\sum_{i, j} \text{max}(0, -(y_{pred}^{(i)}-y_{pred}^{(j)})\cdot(y^{(i)}-y^{(j)}))$$

which is proposed in this paper [https://www.cc.gatech.edu/~zha/papers/fp086-zheng.pdf]. This function is differentiable but also hard to train.

My questions are: (1) Is there any other differentiable loss function that can be applied in ranking problem with regression model;
(2) Can rank function be approximated by a differentiable function.


texts: How to plot the K-means and print the points outside the cluster

texts:
How to plot the K Means for the below data

no,store_id,revenue,profit,state,country
0,101,779183,281257,WD,India
1,101,144829,838451,WD,India
2,101,766465,757565,AL,Japan

My Code is below

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
df = pd.get_dummies(df, columns=['state','country'])
clusters = 7
kmeans = KMeans(n_clusters = clusters)
print(kmeans)

How to scatter plot for the cluster
How to get the print points away from the cluster


texts: How to read first row data as start time , end of the day last row as end time and continue to next day import from csv file using python

texts:
Here I have time series with date from csv file. So in time column first time I want to read it as start time and it will continue with one hour delta time till to end of the day time. Then again it will continue to next day first time as a start time. This is a continue process. Can you help me to solve this problem?
 Here is subset of my data set,

I wrote one code just display date and I don't know how to proceed my code?

data = pd.read_csv('data9.csv')
data['Date']=pd.to_datetime(data['Date'], format="%m/%d/%Y")
print(data['Date'])
data['Time']= data['Time'].str.split(':').apply(lambda x: int(x[0]) * 60 + int(x[1]))
print(data['Time'])

Then I want to give time duration one hour first time to next time to predict value with time.

def make_dates(start_date, number_of_days, start_time, end_time, interval):
if isinstance(start_date, datetime.datetime):
    start_date = start_date.date()
start_date = datetime.datetime.combine(start_date, start_time)
cur_date = start_date
num_days_passed = 0
step = datetime.timedelta(seconds=interval*60)

while True:
    new_date = cur_date + step
    if new_date.time() > end_time:
        num_days_passed += 1
        if num_days_passed > number_of_days:
            break
        new_date = start_date + datetime.timedelta(days=num_days_passed)
    ret_date, cur_date = cur_date, new_date
    yield ret_date

Time = make_dates(datetime.datetime.now(), 3, datetime.time(hour=6), datetime.time(hour=23), 60, None)

Output should have to come like this:-

2018-09-01 06:00', '2018-09-01 07:00', '2018-09-01 08:00', '2018-09-01 09:00', '2018-09-01 10:00', '2018-09-01 11:00', '2018-09-01 12:00', '2018-09-01 13:00'..

So here date is should have to includ csv file date time start should have to time column first time on that day and end time also and this is continues to next day.


texts: Can tensorboard be used after training has finished, to compare models?

texts:
I've been running a few experiments, on top of fairseq/pytorch, and have a separate log file for each experiment. It is just the default log format; if I grep for test results, across all model log files, I have a line like this for each epoch:
model_name/out.log:2021-02-22 16:11:59 | INFO | test | epoch 002 | valid on 'test' subset | loss 10.191 | nll_loss 9.478 | ppl 713.32 | wps 224771 | wpb 14883.3 | bsz 662.8 | num_updates 304 | best_loss 10.191

I can pull out a similar line for train data; with a bit of awk use I could make the formats identical.
I've seen that tensorboard can be made to work with pytorch, but all the examples I see are for plotting live charts to monitor model progress. Can it also be used for comparison charts after a set of models have been trained? Is that something that can be automated/scripted, or is it all done interactively.
(I'm not really familiar with tensorboard, apart from following along with tutorials when I played with TF1 2-3 years ago; my first instinct is to manipulate this log data into a csv format, and then use R or Python/matplotlib to make charts. But I'm wondering if tensorboard can save me that job?)
What I mainly want to know is if tensorboard is suitable, so I know if it is worth installing and learning it; but links to good tutorials on this topic would also be very welcome.


texts: Model Selection using Bias Variance Trade Off

texts:
I have a Regression Model with Train MAPE as 6% and Test MAPE as 15%.
This appears to me as a clear case of over fitting. But can I still use this model assuming 15% Error is not a bad number after-all. Is this there a flaw in this thinking?


texts: Sequence pattern mining on continuous dataset

texts:
I have a dataset on broadband usage and bills for a set of customers for an year. For every metric, like Upload usage, I have 12 values corresponding to each month.
Below are the columns and the format.

  CustomerId,Upload_Jan,Upload_Feb...Upload_Dec,
  Download_Jan,Download_Feb....Download_Dec,
  bill_Jan....bill_Dec

The values in all columns are continuous.
I want to see if there are some sequences appearing in the data frequently.

Can we do sequential mining on continuous data set?
Is it mandatory to bin each column to separate bins? How to decide the number of bins?
What is the best way of pre processing the data set for tools like SPMF which requires transactionized data?

I looked at TraMineR package in R and wrote a few scripts, but I could not make out much from the sequences generated.


texts: Parallel hyperparameter optimization techniques?

texts:
Most hyperparameter optimization technique want to evaluate points one by one.
I have an expensive optimization problem, but i can run hundreds of evaluations in parallel.
The dimension of the problem is around 20-30.
My variables are mostly continuous.

Is there any technique with open source, documented implementation available for this kind of problem?


texts: Error when checking target: dimensions error in CNN-LSTM model for multivariate time series forecasting

texts:
I'm making a CNN-LSTM model to forecast multivariate time series:
       model = Sequential()
       #model.build((10,7,1))
       model.add(Conv1D(filters=64,  kernel_size=2, activation='relu',input_shape=(10,7),strides=1))
       model.add(Conv1D(filters=128, kernel_size=2, activation='relu',strides=1))
       model.add(MaxPooling1D(pool_size=2))
       #model.add(Flatten())
       model.add(LSTM(200,return_sequences=True, activation='relu', recurrent_activation="sigmoid"))
       model.add(Dense(32, activation='sigmoid'))
       model.add(Dense(1))
       model.compile(optimizer='RMSprop', loss='mse',metrics=['accuracy'])
       model.summary()
       print('a new model has been created')

I have as input 7 features ("Time series") and a single output.
I made a function (make_samples to sample the data into sliding window size 10 in code called as n_steps
def make_samples(self,file, n_steps):
    X, y = list(), list()
    data=pd.read_csv(file)
    for i in range(len(data)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the dataset
        if end_ix > len(data):
            break
        # gather input and output parts of the pattern
        seq_x = data[self.lista].values[i:end_ix]
        seq_y = data["Volume"].values[end_ix-1]
        X.append(seq_x)
        y.append(seq_y)

    return array(X).astype("float32"),array(y).astype("float32")

When I pass this data to the model I got the following error:
Error when checking target: expected dense_30 to have 3 dimensions, but got array with shape (659, 1))
The question is, why does this error arise? And, how do I go about fixing this?
Here is the summary of

Layer (type)                 Output Shape              Param
conv1d_38 (Conv1D)           (None, 9, 64)             960

conv1d_39 (Conv1D)           (None, 8, 128)            16512

max_pooling1d_18 (MaxPooling (None, 4, 128)            0

lstm_18 (LSTM)               (None, 4, 200)            263200

dense_29 (Dense)             (None, 4, 32)             6432

dense_30 (Dense)             (None, 4, 1)              33
Many thanks in advance


texts:
I recently started working on RNN and LSTM and got confused with the terminologies.

What is the meaning of RNN layer ?

When I say RNN has 2 layer - does that mean

Two RNN cell connected horizontally ?
Two RNNs connected vertically. In that case what we call the connection of multiple RNN cell horizontally ?


texts: xgboost GridSearchCV take too long or does not goes to the next step

texts:
just strange

%%time
xgb = xgb.XGBRegressor(n_estimators=500, learning_rate=0.07, gamma=0, subsample=0.75, colsample_bytree=1,
                            max_depth=7, tree_method='gpu_exact')

this code takes around Wall time: 866 ms.

but when I do the gridsearchCV it does not goes to the next step
even though I gave only one parameter

    %%time
    xgb = XGBClassifier(tree_method='gpu_exact',verbose_eval=True, silence=False)
    kfold = StratifiedKFold(n_splits=10, random_state=0)
    xgb_param_grid = {
                    'learning_rate': [0.08,0.09],
                    'random_state': [0],
                    'max_depth': [8,9],
                    'n_estimators': [400,500]
    }
    xgbGrid = gsRFC = GridSearchCV(xgb,param_grid = xgb_param_grid, cv=5, scoring="neg_mean_squared_error", n_jobs= 10, verbose = 1)
    xgbGrid.fit(X,y)
    xgb_best = xgbGrid.best_estimator_

for my understanding, this should not take that long.<br/>
it d

does not go to the next step I do not sure this even working or not
it stop with

  Fitting 5 folds for each of 8 candidates, totalling 40 fits
  [Parallel(n_jobs=10)]: Using backend LokyBackend with 10 concurrent
  workers.

data set size is  (15035, 22)
am I doing something wrong?


texts: Difference between text-based image retrieval and natural language object retrieval

texts: I'm working on creating a model that locates the object in the scene (2D image or 3D scene) using a natural language query. I came across this paper on natural language object retrieval which mentions that this task is different from text-based image retrieval in the sense that natural language object retrieval requires an understanding of objects in the image, spatial configurations, etc. I am not able to see the difference between these two tasks. Could you please explain it with an example?


texts: What is a distribution-wise asymmetric measure?

texts:
I was trying to understand KL-Divergence, $$D_{KL} \langle P(X) \Vert P(Y) \rangle,$$ and was going through its Wikipedia article. It says the following

In contrast to variation of information, it is a distribution-wise asymmetric measure and thus does not qualify as a statistical metric of spread - it also does not satisfy the triangle inequality.

What is the meaning of distribution-wise asymmetric measure? Is there a symmetric measure? What are the rules that a quantity should follow to be qualified as a statistical metric of spread?


texts:
I have 10 classes and 10-15 questions in each class .
Given a new question, I want to find the class to which the question is most similar?


texts: error length > 1 and only the first element will be used?

texts:
I'm new i data mining and R, and i'm working with a dataset and trying to remove the "white space" elements. i tried to create a function to do that.

elim.fal=function(v){
contador=0
for (i in length(v)){
if (as.factor(v[i])== "" && contador<=length(v)){
  a=v[-i]
}
else {
  a=v

}
contador= contador+1
return(a)
}

but appear that error


texts: image based example for Bayes error/irreducible error

texts:
I have a hard time intuitively understand the Bayer error in the context of supervised learning. We have an input X and an output Y. We want to find the function f(X) = Y.

I feel like I don't understand why we model X as a random variable in the first place. I think we construct the function f not based on the distribution of X but rather on actual values of X. Why should we care if X is stochastic or not?

Example I:
Let's say X = [1,2,3], Y = [1,2,3] and f(x) = x.
When X=1 => Y=1, X=2 => Y=2, X=3 => Y=3, I will never make an error.

Example II: Why should it be different if X = [picture of dog, picture of cat, picture of house] and Y = [dog, cat, house]. I can still find a function which does the mapping. It is obviously more complex but doable.

Where did the Bayes error get lost in my examples?

I am looking for an intuitive explanation of the Bayes error preferably in the context of image classification.


texts: I am having trouble in understanding the generalized likelihood ratio test (GLRT). Can anyone explain what it is to me, or point me toward an easy-to-understand reference? Is it a supervised or unsupervised method? How GLRT relates to Bayesian approaches?


texts: pros and cons of lexical vs machine learning methods for text mining

texts:
I wanted to know what are the pros and cons are of using lexical methods and machine learning methods for classifying texts based topic.

I have used a simple method of mining documents related to a specific topic based on a keyword list. Basically, if the document contains one of the words from the keyword list it will retrieve it. If that particular word could be used in a different context it checks the post again for other associated words which would usually be found in similar types of documents. This is a simple method but seems to work well, and can be applied to any topic quickly and easily. The main detractor seems to be the keyword lists need to be created and maintained which can be time consuming and inefficient.

In recent times machine learning methods have been used for this type of document classification. It seems this method is able to judge "context" better in documents but requires large datasets to be trained on and also require continual training if new data needs to be classified.

It feels like people dismiss lexical methods since the emergence of machine learning methods but is this warranted? It seems like lexical methods can still get good results, especially on small documents which don't contain much context.

What are the pros and cons of each?


texts: Cleaning input data with pd.get_dummies()

texts:
What is the advantage of converting a series like

>>> df
   Color
0    Red
1   Blue
2  Green
3    Red

To a multiple series like the below?

>>> pd.get_dummies(df)
   Color_Blue  Color_Green  Color_Red
0           0            0          1
1           1            0          0
2           0            1          0
3           0            0          1

One could as well have a hot encoded values for the Color Column as below?

>>> labels=list(set(df.Color))
>>> pd.DataFrame(df.Color.map({x:labels.index(x) for x in labels}).rename('Color_Code'))
   Color_Code
0           1
1           2
2           0
3           1

I know syntactically pd.get_dummies looks much simpler, but somehow I want to lean towards a lesser number of features than more number of features...


texts:
Below is text output obtained after ocr image to string of medical discharge summary report.

XXXXXXXX  T D.0.A'. 20.05. 2017
13.0,? ; 20.05.2017
AGE / sax; 43 YEAR(S] / MALE
CODE: IP1’7- 14041] FHL33350709 D.0.D:22.05.2017
ROOM NO: 1309F
CONSULTANT: XXXXXXXXX [CARDIOLOGIST]
CO v CONSULTANT: XXXXXXXX [GEN . PHYSICIAN]
DIAGNOSIS : ACUTE CORONARY SYNDROME
UNSTABLE ANGINA
MILD CORONARY ARTERY DISEASE
PLAN : MEDICAL MANAGEMENT

The output I obtained is not accurate enough as the input image.I want to cleanse the text output and extract the values of the fields such as DOA,DOD,AGE/SEX,CODE,ROOM NO,CONSULTANT,DIAGNOSIS etc.
For ex , i need the extract the value of the field name "consultant" as "XXXXXXXXX [CARDIOLOGIST]" as output in excel format.

Below is my attempt :

import pytesseract as pt
import re
import pandas as pd
import cv2
import numpy as np
from matplotlib import pyplot as plt

from PIL import Image

img1 = Image.open('/root/regex_try/DS_Sample.jpg')

text = pt.image_to_string(img1)

print(text)

Fieldnames_dict = {'Hospital Name':['Hospital Name', 'Hospital_Name',
'hospital name'],
'Hospital Address':['Address', 'city', 'Hospital Address'],
'Patient Name' : ['Name', 'Patient_Name', 'Patient Name', 'patient name'],
'Address' : ['Patient Address', 'Address','Amy“'],
'Age' : ['Age', 'age'],
'Sex': ['Sex', 'sex', 'gender', 'Gender'],
'Doctor Name' : ['Doctor Name', 'doctor name', 'doctor', 'treating doctor',
'Consultants','CONSULTANT','Consultant Name'],
'Inpatient Number' :['Inpatient Number', 'IP Number'],
'Admission Date' : ['Date of Admission', 'Admitted Date', 'D.0.A'],
'Discharge Date' : ['Date of Discharge', 'Discharged Date', 'D.0.D'],
'Diagnosis' : ['Diagnosis', 'Final Diagnosis', 'Principal/Secondary
 Diagnosis'],
'Treatment Given' : ['Treatment Given', 'On Examination', 'Examination'],
'Follow up' : ['Follow Up', 'Follow up', 'Review & Advise', 'Condition on
 Discharge', 'Advise on Discharge'],
'Summary' : ['Summary', 'Past Treatment Given'],
'Bed No' : ['Bed Number', 'Bed / Ward Details'],
'SS Number' : ['SS Number', 'SS No.'],
'UHID Number' : ['UHID Number', 'UHID No'],
'MR Number' : ['MR No', 'MR Number','MRNO '],
}

def preprocess_formtext(field_dict,text):
   values_dict=field_dict
   values_dict=values_dict.fromkeys(values_dict, '')
 for i in field_dict.keys(): #reaching the keys of dict
    for x in field_dict[i]: #reaching every element in tuples
        q=0
        p=[x.upper()+r":(.*) ",x.upper()+r" : (.*) ",x.upper()+r": (.*)",x.upper()+r": (.*)"]
        for j in p:
            #print(j)
            match = re.search(j, text)
            if match:
                result = match.group(1)
                q=1
                values_dict[i]=result
                break
            else:
                result = ""
        if q==1:
            break
return values_dict

values_dict=preprocess_formtext(Fieldnames_dict,text)

formdata_df=pd.DataFrame([values_dict])


texts:
Can anyone suggest a machine learning algorithm that would be useful for identifying the organization a document belongs to? I'm working with a relatively large number of organizations (on the order of 1000s). Naturally, the input data is unstructured, but most documents look almost, but not quite, the same. Also, there are some organizations that follow more then a one template, again, in that "almost but not quite" manner.

Documents are in PDF format, and are reports of some kind and can contain several rectangular regions of text and tables. Most of them are not more than five pages long.

New organizations can eventually appear - how does interfere with algorithm?


texts: How do I find the common values in two different dataframe by comparing different column names?

texts:
If the data frames are as follows :
df1
column names = UniqueID  Age Gender
values = 213,21,F
         145,34,M
         890,23,F
df2
column names = ID Occupation Bill
               145 Teacher 45RS
               890 Student 70RS
               456 Teacher 90Rs

I want to use a function which finds the common values in "UniqueID" from df1 and "ID" from df2 and gets stored in df3. Also, the only columns which exist in df3 should be of "common ids" and the columns of df2.


texts: How to read Several JSON files to a dataframe in R?

texts:
I have a folder with 30,000 plus JSON file. A sample file with contents is posted below.

{
"name": null, "release_date_local": null, "title": "3 (2011)",
"opening_weekend_take": 1234, "year": 2011,
"release_date_wide": "2011-09-16", "gross": 59954
}

However, I need the data in a df in a structure as given below:

name relase_date_local title    opening_weekend_take year release_date gross
NA   NA                3 (2011) 1234                 2011 2011-09-16   5994

Here is my code snippet to get all the files as a list:

path = "./Week1/jsonfiles"
temp = list.files(path, pattern = "*.json")
filename = paste(path, temp, sep = "/")
movies = c()
for (i in filename){
  movie = fromJSON(file = i)
  movies = c(movies, movie)
}

Please advise, how can I read all 30,000 files as rows of a df?


texts: How to plot data points and not centroids using sklearn k-means?

texts:
I have issues finding a way to plot data points colored by cluster with k-means.

I have a very long list of strings.I managed to plot the centroids but not the data points;

import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(cleans)

true_k = 5
model = KMeans(n_clusters=true_k, init='k-means++', max_iter=100, n_init=1)
model.fit(X)
print("Top terms per cluster:")
order_centroids = model.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(true_k):
    print("Cluster %d:" % i),
    for ind in order_centroids[i, :10]:
        print(' %s' % terms[ind]),
    print()

print("\n")

plt.scatter(order_centroids[:, 0],order_centroids[:, 1], marker="x", s=150, linewidths=5, zorder=10)
plt.show()

I expect the output to be the data points colored by cluster, and not the centroids


texts: Theoretical and practical comparison of CTC and seq2seq loss in Tensorflow

texts:
Tensorflow has built-in implementations for both, the Connectionist Temporal Classification (CTC) loss  and a special seq2seq loss (weighted cross-entropy).

Since CTC loss is also intended to deal with seq2seq mappings, I wonder about how the two loss functions compare. I went through an excellent explanation of CTC loss, finding out that the target sequence is restricted not to be longer than the input sequence whereas this restriction does not exist for seq2seq loss.

Some questions I have (ranked by relevance):

Are bidirectional LSTMs suited for seq2seq loss? I have only found b-LSTM implementations with CTC or even simpler loss functions. There is a post from 2016 saying that this was not possible at that time. Any updates?
Are there conceptual differences apart from the many-to-one restriction in CTC loss that seq2seq does not have? E.g. does the seq2seq loss make a similar conditional independence assumption like the CTC loss?
Do the loss functions have preferences regarding the optimizer (RMSProp, ADAM, Momentum etc.)
Do the loss functions work better for particular sets of tasks?

Theoretical arguments and practical experiences are both appreciated.


texts: DTW (Dynamic Time Warping) requires prior normalization?

texts:
I'm trying DTW from mlpy, to check similarity between time series.

Should I normalize the series before processing them with DTW? Or is it somewhat tolerant and I can use the series as they are?

All time series stored in a Pandas Dataframe, each in one column. Size is less than 10k points.


texts: Search and Replace in specific column of csv with python

texts:
I created a program that search and replaces over an entire csv file but I need to make so it is column specific. Is there a way to only search via a column. Here is my code, I am pretty new to python so I apologize if this is an easy fix.

import csv

ifile = open('testbook.csv', 'rb')
reader = csv.reader(ifile,delimiter='\t')
ofile = open('output.csv', 'wb')
writer = csv.writer(ofile, delimiter='\t')

findL = ['Australia', 'Mexico', 'United States', 'hob']
replaceL = ['Kangaroo', 'Spider Monkey', 'Eagle', 'Test']

rep = dict(zip(findL, replaceL))

def findReplace(find, replace):
    s = ifile.read()
    for item, replacement in zip(findL, replaceL):
        s = s.replace(item, replacement)
    ofile.write(s)

for item in findL:
    findReplace(item, rep[item])

ifile.close()
ofile.close()


texts: Reporting test result for cross-validation with Neural Network

texts:
I have a small dataset, so I have to use cross validation to report the test result to get a better estimate of the classification result. For some reason, I have to use neural networks to do this.

Because neural networks have their unique quirks e.g finding hyper-parameters, I am using a nested cross-validation. I am dividing up my dataset into 10 folds for cross-validation. Then I am dividing the 9 folds that are for training, again in 10 folds. From those 10 folds, I am using 9 folds to train with different hyper-parameters(in my case it is number of hidden units and dropout rate), and using the other fold to to get the accuracy with different hyper-parameters (kind of like a validation set in the deep learning literature).

Then I am training my model again on all of the 9 folds of the first division of data with the best hyper-parameters I found. Because I am missing out on some data from the initial 9 folds for using as a validation set. Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result? Just to make it clear, I am not tuning any hyper-parameters at this stage. I am just setting the network to stop training on the training data when it reaches the best test result. I think this is a really subtle problem, if it is a problem at all. That is why I am confused.

I am doing this whole thing 5-6 times with different seeds for different divisions of data, and I am only reporting the mean of all of these runs.


texts:
I am working on a data set that has multiple traffic speed measurements per day. My data is from the city of chicago, and it is taken every minute for about six months. I wanted to consolidate this data into days only, so this is what I did:

traffic <- read.csv("path.csv",header=TRUE)
traffic2 <- aggregate(SPEED~DATE, data=traffic, FUN=MEAN)

this was perfect because it took all of my data and averaged it by date. For example, my original data looked something like this:

DATE        SPEED
12/31/2012   22
12/31/2012   25
12/31/2012   23
...

and the final looked like this:

DATE        SPEED
10/1/2012    22
10/2/2012    23
10/3/2012    22
...

The only problem, is my data is supposed to start at 9/1/2012. I plotted this data, and it turns out the data goes from 10/1/2012-12/31/2012 and then 9/1/2012-9/30/2012.

What in the world is going on here?


texts: How to encode positional information of a string in a NN effectively

texts:
So I want to compare strings and find out there relation to one another. For this I figured out I also have to encode the positional info. For eg:
aaaaassssssbbbbbb should be more relevant to aaaasssssbbbbrather than bbbbaaasssss. The output will be a relative measure and 1 and 2 should match more compared to 1 and 3. Also I will represent each character by a number.

So how can I re-design my feature vector (feature transformation) to be input in a NN to capture this information effectively?


texts: How do I get the feature importace for a MLPClassifier?

texts:
I use the MLPClassifier from scikit learn. I have about 20 features. Is there a scikit method to get the feature importance? I found

clf.feature_importances_

but it seems that it only exists for decision trees.


texts: How to determine key frames in a video for video classification?

texts: How to detect changes between when the changes between 2 frames of videos that are significant enough to be counted for video classification? Thinking of the problem as analogous to edge detection in an image where the change between 2 adjacent pixel is threshold-ed to detect an edge.


texts: Extract keywords from set of similar sentences by using python

texts:
I have a list of similar sentences, and I would like to automatically extract top-n important keywords [single word length] from a whole set of those sentences by using python. These sentences are already in normalized form as they do not contain stop words.
Those top-n important keywords should be a representation of those similar sentences.
Which technique should I follow such as summarization technique, keyword extraction technique, or topic modeling technique?
I know TF-IDF, RAKE etc but I how can I utilize them to achieve my goal? Can you please suggest some code snippet which can work well to achieve that goal?


texts: Why can't continuous variables be used for the estimator of learning curves, when using StratifiedKFold to split the dataset?

texts:
I want to produce learning curves for three regression models run on data containing 200 samples, 10 features and 1 target variable.

The target variable contains two clusters/peaks, making it imbalanced between and within the clusters, so I applied a stratified split to divide the data into training and test sets, using train_test_split, in the following manner:

# Stratified Split dataset into Training and Testing
bins = np.linspace(0, 1.01, 10)
y_binned = np.digitize(Y_scaled, bins)

X_train, X_test, Y_train, Y_test = train_test_split(X_scaled, Y_scaled, stratify=y_binned, test_size=0.3, random_state=0)

I then trained and tested each models, using stratified K-fold cross-validation for hyperparameter optimisation.

I would now like to use leaning curves to examine verify that my models are not overfitting, and that the training and test scores have converged.  For this I am turning to sklearn.model_selection.learning_curve.

In the documentation for sklearn's learning curve it says the following:

  Learning curve.

  Determines cross-validated training and test scores for different
  training set sizes.

  A cross-validation generator splits the whole dataset k times in
  training and test data. Subsets of the training set with varying sizes
  will be used to train the estimator and a score for each training
  subset size and the test set will be computed. Afterwards, the scores
  will be averaged over all k runs for each training subset size.

I implemented the learning curve using the function defined in Plotting Learning CUrves (with an additional 'scorer' variable) and the following code:

cv = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)

title = "Learning Curves (Linear Regression)"
estimator = model_1
plot_learning_curve(estimator, title, r2_score, X_train_0, Y_train_0, train_sizes=np.linspace(0.1, 1.0, 10), \
                    ylim=(0.1, 1.01), cv=cv, n_jobs=-1)

title = "Learning Curves (Ridge Regression)"
estimator = model_2
plot_learning_curve(estimator, title, r2_score, X_train_0, Y_train_0, train_sizes=np.linspace(0.1, 1.0, 10), \
                    ylim=(0.1, 1.01), cv=cv, n_jobs=-1)

title = "Learning Curves (Random Forest - Extra Trees)"
estimator = model_3
plot_learning_curve(estimator, title, r2_score, X_train_0, Y_train_0, train_sizes=np.linspace(0.1, 1.0, 10), \
                    ylim=(0.1, 1.01), cv=cv, n_jobs=-1)

plt.show()

According to the code:
train_sizes=np.linspace(0.1, 1.0, 10
and
cv = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
I expected that the training dataset (X_train_0) would be split into 10 sub-sets, that each sub-set would be stratified according to the whole (as it was during training and testing), and that each of the ten sub-sets would be then split into stratified training/test set, with the training set undergoing 5-fold stratified k-fold cross validation.

I passed the training data X_train_0 and targets Y_train_0 to the learning curve function, expecting to obtain mean training and testing scores for each sub-set, however I received the following error:

  ValueError: Supported target types are: ('binary', 'multiclass'). Got
  'continuous' instead.

At this point, I can't figure out how I can pass continuous variables for the target of the learning curve the estimator, while producing stratified sub-sets of the training dataset.


texts: What is the name of non-neural network algorithms

texts:
i am writing papers for my academic degree on machine learning.

everytime I get confused when I try to put a name for standard algorithms (KNN, Linear Regression, Random Forest, etc) ( the non-neural network algorithms)

are they called standard algorithms?

or classic algorithms?

or generic algorithms?

is there a name that is used for these kind of algorithms?


texts: Prediction ability by small sample size datasets

texts:
All the datasets i have for modelling are relativ small - ca. 80 rows and there is no way how to increase the sample size. If there is no significant or dominant parameter / input (usually there is any), the model performance differs (R2 from 25% to 70%) as i play with session_ID or split.
They are two approaches possible:
Approach1:
Achieve the best R2 or an error (in train, test and unseen) with trying to find optimal split and session ID - the result is one model, which will be fine-tuned and deployed.
Approach2:
Record the whole performance (R2 or an error) from 30 experiments with several splits and sessions_ID's, then caluclate an average R2 and the range or std.dev from result. From all 30 experiments i take the model with an average performance and this is going to be fine-tuned and deployed. This is the way how to find out, how the model will behave on the new data. Something like "confidence interval".
Which of the two approaches is the right one?


texts: Why does Catboost outperform other boosting algorithms?

texts:
I have noticed while working with multiple datasets that catboost with its default parameters tends to outperform lightgbm or xgboost with its default parameters even on a tabular dataset with no categorical features.
I am assuming this has something to do with the way catboost constructs the decision trees but I just wanted to confirm this theory. If anyone could elaborate on why it performs better on non categorical data then that would be great! Thanks in advance!


texts: Using Information from the rest of a Sequence to Predict the Label for any one Item

texts:
I have a dictionary of variable-length sequences:

[(file_name[-10:], len(tag_is_header_list)) for file_name,
 tag_is_header_list in HEADER_PATTERN_DICT.items()]

[('37bd1.html', 25),
 ('0bcce.html', 40),
 ('90364.html', 28),
 ('8f9c7.html', 24),
 ('d12d4.html', 73),
 ('46837.html', 37),
 ('adb92.html', 53),
 ('0a1e7.html', 69),
 ('da077.html', 43),
 ('9366a.html', 21),
 ('6ae4d.html', 37),
 ('f62ee.html', 19),
 ('73aee.html', 33),
 ('e090a.html', 35),
 ('8b093.html', 44)]

These contain a label for each item as to whether or not they are a subject heading:

HEADER_PATTERN_DICT[sorted([(file_name, len(tag_is_header_list)) for file_name,
                            tag_is_header_list in HEADER_PATTERN_DICT.items()],
                           key=lambda x: x[1])[0][0]]

[(None, True),
 ('<div', False),
 ('<div', False),
 (None, True),
 (None, False),
 ('<li', False),
 ('<li', False),
 ('<li', False),
 (None, False),
 (None, False),
 ('<li', False),
 ('<li', False),
 ('<li', False),
 (None, True),
 (None, True),
 ('<li', False),
 ('<li', False),
 ('<li', False),
 ('<div', False)]

Every item in the sequence is an instance for which the label should be predicted. So, what is the best way to use some variable-length sequence vectorization to train a model to predict the label?


texts: Neural network: does bias equal to zero, is the same as, a layer without bias?

texts:
Question as in the title. Does bias equal to zero, is the same as, removing bias from the layer? Here's a pytorch implementation to showcase what I mean.
class MLP_without_bias(torch.nn.Module):
    def __init__(self):
        super().__init__()

        # Bias set as False
        self.linear = torch.nn.Linear(5, 3, bias = False)

        # Xavier initialization
        torch.nn.init.xavier_uniform_(self.linear.weight)

    def forward(self, x):
        return self.linear(x)

class MLP_with_bias_zero(torch.nn.Module):
    def __init__(self):
        super().__init__()

        # Default bias set as True
        self.linear = torch.nn.Linear(5, 3)

        # Xavier initialization
        torch.nn.init.xavier_uniform_(self.linear.weight)

        # Bias initialized as zero
        torch.nn.init.zeros_(self.linear.bias)

    def forward(self, x):
        return self.linear(x)


texts: Can you recommend a machine learning challenge that is suitable for novices?

texts:
I am looking for a challenge that is suitable for a group of novices who want to learn the basics of data science and machine learning. The challenge should match the following criteria:

is based on a real application or is at least realistic
has a clearly defined goal and partial progress is measurable
includes a machine learning component, but also other aspects of data science
should be doable within 3 to 6 weeks
is suitable for novices
it should be an actual challenge in the sense that you cannot just look up near-optimal solutions from the internet


texts:
Theoretically, is this possible: When you have a given input string, there are a set of permutations and bit operations to do.

The problem is, when you choose an approach like reinforcement learning, the model could do these kind of actions again and again , and could only reach its destination by chance.

When there is an upper limit of steps (timesteps), you cannot tell how far the model is from reaching its goal because you just don't know what to do next. The reward is always 0 when the goal has not been reached and 1 when it has been reached. Thus, since the reward is basically always 0, the algorithm cannot learn.

Any ideas as to why it is so?


texts:
I need an opens source solution to classify a document. I do not want to use NLP i need only to check the look and feel.

I tried OpenCV. I have a template and i need to match it.

import cv2

template = cv2.imread(template_file,0)
template = cv2.normalize(template, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)

 method = ['cv2.TM_CCOEFF_NORMED']

 img = img2.copy()
 method = eval(meth)
 # Apply template Matching
 res = cv2.matchTemplate(img,template,method)

 min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
 if (max_val > threshold):
    print('match OK')

this method doesn't seems so robust and I get a lot of false positive.


texts: How to load a saved model in TensorFlow?

texts:
This is my code in Python:

from __future__ import absolute_import, division, print_function, unicode_literals
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np

I checked if the saved model is there using the following code:

tf.compat.v1.saved_model.contains_saved_model(
    '/Link_to_the_saved_model_directory/'
)

which returns True
and I can use the following code to further make sure the model is saved correctly, as  far as I understood:

tf.saved_model.Asset(
    '/Link_to_the_saved_model_directory/'
)

which returns this:

<tensorflow.python.training.tracking.tracking.Asset at 0x2aad125e5710>

So, everything looks fine. But, when I use the following script to load the model I get an error.

LoadedModel = tf.saved_model.load(
    export_dir='/Link_to_the_saved_model_directory/', tags=None
)

Error:

OSError: Cannot parse file b'/Link_to_the_saved_model_directory/saved_model.pbtxt': 1:1 : Message type "tensorflow.SavedModel" has no field named "node"..

The files in the /Link_to_the_saved_model_directory/ look like this:

['saved_model.pbtxt',
 'saved_model.ckpt-0.data-00000-of-00001',
 'saved_model.ckpt-0.meta',
 'checkpoint',
 'saved_model.ckpt-0.index']

Any suggestion is greatly appreciated on how to load the model so that I could reuse it for transfer learning. It might also be the case that the model is partly written in a previous version of TensorFlow (e.g. TensorFlow 1.x) and this error is thus due to compatibility issue, but I could not find a solution for that yet.

Update: I tried the following method but it does not work (tf was imported using the compatible version import tensorflow.compat.v1. as tf):

with tf.Session() as sess:
    saver = tf.train.import_meta_graph('/dir_to_the_model_files/saved_model.ckpt-0.meta')
    saver.restore(sess, "/dir_to_the_model_files/saved_model.ckpt-0")
    loaded = tf.saved_model.load(sess,tags=None,export_dir="/dir_to_the_model_files",import_scope=None)

It returns the following warnings and errors:

WARNING:tensorflow:The saved meta_graph is possibly from an older release:
'metric_variables' collection should be of type 'byte_list', but instead is of type 'node_list'.
INFO:tensorflow:Restoring parameters from /dir_to_the_model_files/saved_model.ckpt-0
<tensorflow.python.training.saver.Saver object at 0x2aaab4824a50>
WARNING:tensorflow:From <ipython-input-3-b8fd24f6b841>:9: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.

OSError: Cannot parse file b'/dir_to_the_model_files/saved_model.pbtxt': 1:1 : Message type "tensorflow.SavedModel" has no field named "node"..


texts: Genarate one hour time interval array using pandas in python (import from csv) to predict next value

texts:
I am trying to generate one hour one hour time interval to predict next value according to my data set imported from csv file. Here according to the time it will give outputs include in x column. This output is actual value and  I want to predict next value at 12.40 p.m (time) according to the past actual values and the time. For that I need to write an array for one hour to one hour continuously to predict the value with next time . I don't know how to proceed the time array after converting it.

subset of actual dataset;

Can you give me any suggestion to me solve this code?

My code is,

data = pd.read_csv('data4.csv')
data['Time'] = (pd.DatatimeIndex(data['Time']).astype(np.int64)//10**9)%86400
print(data['Time'])


texts: Validation accuracy is always close to training accuracy

texts:
I am trying to tune the hyperparameters of a LSTM I have to do time series forecasting.  I have noticed that my validation accuracy is always very close to my training accuracy.  I am not sure whether or not this is good or bad or what it implies in general?

At the moment I have kept all the hyperparameters the same and have only varied the number of units in the LSTM layer from [1, 5, 26].

I expected 26 units to give me good results and have since added units 1 and 5 to help me investigate.  I was also expecting to see that my validation accuracies are worse than my training accuracies but this does not seem to be the case.  My validation accuracies track the training accuracies very well.  Is this something to be concerned about?

The plot below shows the average loss, average training and validation accuracies.

You can see from the plot that each of the units training and validation accuracies stay very close together.  Why is this?  Does this mean that in general my model should generalise well on unseen data as there isn't much of a discrepancy?

Obviously the accuracies of the model are not very good yet in general and it requires some tuning but to do the tuning process I was more expecting to see some differences between the training and validation data and not that they would stay largely similar to each other.

EDIT:

Further information as requested.  The hyperparameters used for each the model can be seen in the legend of the plots. Here is an updated plot which shows a greater variation in accuracy from varying the number of neurons and batch size:

From the plot it seems as if the model accuracy converges the quickest with a batch size of 1 and 39 neurons.  However it is characteristic amongst all that the training and validation accuracies track each other closely.  I hadn't expected that varying two hyperparameters independently would lead to a consistent result like that.

In this problem I am working with, the model is to provide forecasts for one time series.  I am using it as a 'toy' problem to try and learn what kind of models work well for my particular problem.

I have 315 data points in total for the time series.  I have left the last 52 out as a hold out test set which I haven't ever looked at to this point. 52 points as I am making weekly predictions. I then left the next last 52 points out to use as a validation set.  Due to the time dependency I am unable to use a validation method such as KFold cross validation.  Therefore I am using something called rolling origin analysis which is explained well here (see under predictive performance).  this simulates well how the model would be used in practice and my accuracies are measurements using a modified sMAPE formula for a multi-step forecast.

Essentially what this means is that I train (in this case) 27 separate models.  I have a forecast horizon of 26, so what I do is take my train data and take the first 26 points of my validation set.

I train my first model on the train data (which is 131 samples) and this is validated against 1 sample.

I train the next model which uses the same training data except it is shifted one along so the very first point from train is dropped and the first point from the validation set is appended to the end of the train set.  The validation set is then also shifted along by one.  As a quick example (numbers represent indices of the time series):

Model1 train: [1, 2, 3, 4, 5], val: [6, 7, 8]

Model2 train: [2, 3, 4, 5, 6], val: [7, 8, 9]

The average accuracy shown in the plots is therefore the average of the accuracies that each model computed against its own validation set.

So each model is trained on the same amount of data and each model is completely fitted anew, that is Model1 doesn't share any information with Model2.

The accuracies are computed in this way because I do not have much data and this is the best way I have found that allows me to get more than two validation sets and more than two test sets.


texts: What does $\mathbf{w^Tx}+w_0$ graphically mean in the discriminant function?

texts:
I found a post explaining the discriminant function very detailed. But I am still confused about the function $g(\mathbf{x})=\mathbf{w^Tx}+w_0$ in 9.2 Linear Discriminant Functions and Decision Surfaces. What does it represent graphically? Could anyone explain it, probably with figure 9.2?

Does it mean the distance between the origin and the hyperplane?


texts: How to factorize the Matrix in TensorFlow? (Recommender System)

texts:
Given a user ratings matrix which is $n \times p$, where $n$ users rate $p$ movies, I already have a row matrix $n \times 10$ which characterises the user.

I ideally wanted to use the TF was method for optimisation, https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/factorization/WALSMatrixFactorization
but it looks like it creates the row matrix itself.

What I need is to create the column matrix - which is $10 \times p$ (not both), containing the relationship between hidden characteristics (10) to the movies (p).

How can I do this in TF?


texts: What is the scalability of linear regression and decision trees?

texts: Recently I'm studying machine learning algorithms among them linear regression and decision tree so I have a question regarding the scalability of both algorithms. Can anyone provide what is the scalability of both algorithms and examples?


texts: Feature scaling for MLP neural network sklearn

texts: I am working with a dataset that has multiple scales for my features. Before running sklearn's MLP neural network I was reading around and found a variety of different opinions for feature scaling. Some saying you need to normalize, some saying only standardize, others saying in theory nothing is needed for MLP, some saying only to scale training data and not testing data, the sklearn documentation says MLP is sensitive to feature scaling? This has left me very confused on which route I should take for my dataset before running the MLP model. Any clarification on these and how I should proceed would be much appreciated.


texts:
I use data from https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test, W value is 9.
But, for the following code W value is 18,
what is the reason?

# Wilcoxon signed-rank test
from numpy.random import seed
from numpy.random import randn
from scipy.stats import wilcoxon
# seed the random number generator
seed(1)
# generate two independent samples

a=np.array([125,115,130,140,140,115,140,125,140,135])
b=np.array([110,122,125,120,140,124,123,137,135,145])

# compare samples
stat, p = wilcoxon(a,b)
print('Statistics=%.3f, p=%.3f' % (stat, p))
# interpret
alpha = 0.05
if p > alpha:
    print('Same distribution (fail to reject H0)')
else:
    print('Different distribution (reject H0)')


texts: How much data to use for feature selection?

texts:
Working on my master's thesis, this is a problem I'm unable to find good resources about.

I'm working with data of 18 participants, who are either active or passive. Each participant is then subjected to a 3 x 3 experiment and results in a total of around 676 trials per participant (around 12.168 trials in total). There are 100 data points in each trial but cannot be used separately from the trial (since its an EEG epoch).

My data consists of 579 features, so I need some sort of feature selection as literature shows that most of them are irrelevant, but I want to use a bottom-up machine learning approach (to verify this).

Is there a rule of thump/literature to use for the amount of data needed for feature selection?


texts: How to compare number of occurrences in two data frames for a column and extract the similarities

texts:
I'm trying to find the same number of occurrences in both data frames
This is a follow-up question for my previous question
I got 2 data frames
df1=pd.DataFrame([[1,None],[1,None,],[1,None],[1,'item_a'],[2,'item_a'],[2,'item_b'],[2,'item_f'],[3,'item_e'],[3,'item_e'],[3,'item_g'],[3,'item_h']],columns=['id','A'])
df2=pd.DataFrame([[1,'item_a'],[1,'item_b'],[1,'item_c'],[1,'item_d'],[2,'item_a'],[2,'item_b'],[2,'item_c'],[2,'item_d'],[3,'item_e'],[3,'item_f'],[3,'item_g'],[3,'item_h']],columns=['id','A'])

 df1
        id  A
    0   1   None
    1   1   None
    2   1   None
    3   1   item_a # id 1 has 1 occurrences in total in df1
    4   2   item_a
    5   2   item_b
    6   2   item_f #id 2 has 3 occurrences in total in df1(id 2 has 3 occurrences here)
    7   3   item_e
    8   3   item_e
    9   3   item_g
    10  3   item_h #id3 has 4 ccurrences in total in df1

df2
    id  A
0   1   item_a
1   1   item_b
2   1   item_c
3   1   item_d
4   2   item_a
5   2   item_b
6   2   item_c
7   2   item_d
8   3   item_e
9   3   item_f
10  3   item_g
11  3   item_h

I got an answer on how to find similarities by using
previous result:
d=pd.merge(df1,df2,how='inner')
        id  A
3   1   item_a # id 1 has 1 occurrences in total in d
4   2   item_a
5   2   item_b # id 2 has 2 occurrences in total in d(id 2 has 2 occurrences here which does not match all the occurrences(3) in df1)
7   3   item_e
8   3   item_e
9   3   item_g
10  3   item_h #id 3 has 4 occurrences in total in d

What I've tried to find same number of occurrences in both data frames:
d[d['id'].value_counts()==df1['id'].value_counts()]
Which gave me an error:Can only compare identically-labeled Series objects
I've also tried different things using rename to put a column name for value_counts and merge them but failed. Match: Count of occurrences in df1 for an id match count of occurrences in result data frame d
        cnt_in_df1|cntin_d
for id1:     1    | 1  count #match => id 1 should be in the desired output.
for id2:     3    | 2  count #mismatch=> id 2 should not be in the desired output
for id3:     4    | 4  count #match => id 3 should be in the desired output.

My desired output for this question:

        id  count
    0   1    1
    1   3    4


texts: What is the proper way to use time-series data for classification?

texts:
I have a time-series dataset for a classification problem. The data contains brain signals collected via EEG eletrodes along 2 seconds in frequency (Hz). The classes are divided in different files (so each file contains only one class). This is a typical BCI (Brain-computer interface) classification problem, and as far as i know it is the usual way of separating classes for classification.

I want to know the proper way to prepare data for the classifier algorithms, since i have no X,Y values such as a common classification problem should have and the data is time-dependant. Any explanation/material is welcome.

I'm considering using scikit-learn MLPClassifier as well.


texts: Gaussian Mixture Models EM algorithm use average log likelihood to test convergence

texts:
I was investigating scikit-learn's implementation of the EM algorithm for fitting Gaussian Mixture Models and I was wondering how they did come up with using the average log likelihood instead of the sum of the log likelihoods to test convergence.

I see that it should cause the algorithm to converge faster (given their default parameters), but where does that idea come from ?

Does anyone know if they based this part of the implementation on a specific paper or if they just came up with it and used it ?

In most explanations of the EM algorithm I have come across, they would have used log_likelihoods.sum() instead of log_likelihoods.mean().


texts:
I'll jump right to the structure of the data, and then I'll ask the question(s):

For a mass X ranging from 200 to 500 units, i have 100 seconds worth of 3 output_values.
So, the first few rows of the data set look like:

t =   1, m = 200, out_1 = a, out_2 = b, out_3 = c
t =   2, m = 200, out_1 = d, out_2 = e, out_3 = f
...
t = 100, m = 200, out_1 = h, out_2 = i, out_3 = j
t =   1, m = 201, out_1 = m, out_2 = n, out_3 = o
etc.

This is how the distribution of the outputs for a mass = x for 100 seconds looks:

Now, my goal is to test wether an LTSM could predict 100s worth of output for every mass =

x, x + 1, ..., x + 100

after seeing

x - 300, x - 299, ..., x - 1

In my case, should I also use the time as an input_variable?

I couldn't completely figure out a way to structure my data but "Multiple Parallel Input and Multi-Step Output" from https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/ seems to be right But I don't know how to prevent it from treating cases like:

...
input from second 98, m = 200
input from second 99, m = 200

output from second 1, m = 201
output from second 2, m = 201
...

which have no correlation and shouldn't be treated as an example.

How would you approach this problem?

I'm more than happy to get links to similar problems. I've played around with various approaches today, but the results weren't satisfactory.

Thank you all!
Andreas

Update (what i tried) for seeing if the first 50 seconds of a mass can predict the next 50 seconds:

Using this notebook as a skeleton:
https://www.kaggle.com/anshuljdhingra/time-series-data-analysis-using-lstm-tutorial/data

Pass the data to series_to_supervised(df.values, 50, 50)
50 input seconds, 50 output seconds

(size of data before 30100 x 4, size of data after 30001 x 400)
n_train_time = 150 * 100
(first 150 examples, times 100 seconds each)

size of train_X = (15000, 200)
size of train_y = (15000, 200)

size of test_X = (15001, 200)
size of test_y = (15001, 200)

train_X after reshape: (15000, 50, 4)
test_X  after reshape: (15000, 50, 4)
same simple LSTM model, with:

model.add(LSTM(100, input_shape=(train_X.shape1, train_X.shape[2])))

and batch_size=100

Should the batch_size pe equal to the time window i'm training on?
How does my logic look?

Thanks again!
Andreas


texts: (Feature Selection) different results from L2-based and Tree-based

texts:
I am doing feature selection using Sklearn:

Tree-based feature selection : RandomForestClassifier.feature_importances_
L2-based feature selection: LogisticRegression.coef_

Target variable is binary classes. The training set is standardized.

How should I interpret when a certain feature shows significant importance in Random Forest estimator, but negative coefficient in Logistic Regression?


texts:
I want to do facial recognition on wide varieties of images captured at various ages of my family members. Below are some of the questions I have.

If a person uses glasses of different types, do I need to feed images with each glasses type in the training dataset ? What if the eyes are not visible after wearing sunglasses. ?
There are many pics which ranges from childhood to current age of approx 30. Do I need to train my model with pics of various ages to get good accuracy ?
Do I need to align the images of train dataset for better accuracy ?
Do I need to crop the background of the images in the training dataset just to keep the faces OR background doesn't matter ?

The idea behind my requirement is to scan through all the images on my hard disk (around 100 GB images) and classify the images in different folders for each person. I know similar thing is done on iphone and other mobile phones but I wanted to do it for the offline images stored on my external hard disk.

I was exploring facenet model from google, opencv face recognition. But stuck in the first step of dataset creation.


texts: Multi-valued categorical features in LIME

texts:
I am working with the LIME implementation by Marco Ribeiro (https://github.com/marcotcr/lime). Specifically, I am utilizing the LimeTabularExplainer as I have a mixture of numerical and categorical features in my dataset. How would I represent categorical features that may take on ≥ 0 values in a single example? I understand that the API requires categorical features to be converted
to an integer representation, but how would I represent combinations of values for one categorical feature? To illustrate the circumstance, see the example dataset attached as an image and consider the "Comorbidities" feature.

One approach I investigated was to consider the presence of each value as its own binary categorical feature; however, the number of features scales up fast since I have multiple features with examples taking on many combinations of values. I am concerned that my approach does not enable efficient sampling around an example to be explained. I understand that LimeTabularExplainer requires that values in categorical columns be integers, but how would I encode these "multi-valued" categorical features as integers? Thank you for considering my question!


texts:
Suppose you train Neural Network with SGD and see that it overfits data. Which of the following actions can help you to regularize model?

Change optimization method to Adam.

Insert (or increase rate of) Dropout layers inside NN.

Add (or increase) Weight Decay.

Reduce number of parameters (e.g. remove some layers)

Add more layers.


texts: Neural networks with non-negative weights

texts: Could you tell me, are there any techniques for building neural networks with non-negative weights?


texts: Is data science only for substantive specialist?

texts: Right now there is a lot of hype for data science. Everyone is learning it. There are lots of courses and boot camps being offered. Unlike software engineering, however, are data scientists required to have a specialization in a specific domain? For example, if you want to be a data scientist and like to work on topics in biology, would you need to know a lot about biology?


texts: clarification on back-propagation calculations for a fully connected neural network

texts:
I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.
Let's say my fully connected neural network looks like this:

Notation I will be using:
X = Matrix of inputs with each row as a single example,
Y = output matrix,
L = Total Number of layers = 3,
W = weight matrix of a layer. eg: $W^{[2]}$ is weight matrix of layer 2,
b = bias of a layer. eg: $b^{[2]}$ is bias of layer 2,
Z = Linear function of a layer. eg: $Z^{[2]}$ is linear output of layer 2,
A = Post-activation output of a layer. eg: $A^{[2]}$ is Activation of layer 2,
and Loss/Cost = Cross entropy cost after a Gradient Descent Iteration,
. = matrix multiplication and * = element-wise multiplication of a matrix.
So, during Forward Propagation, our calculations will be:
at first layer:
$Z^{[1]} = W^{[1]} . X + b^{[1]}$
$A^{[1]} = relu(Z^{[1]})$
at second layer:
$Z^{[2]} = W^{[2]} . A^{[1]} + b^{[2]}$
$A^{[2]} = relu(Z^{[2]})$
at third and output layer:
$Z^{[3]} = W^{[3]} . A^{[2]} + b^{[3]}$
$A^{[3]} = \sigma(Z^{[3]})$
Now the cost:
$\mathcal{J} = \mathcal{L} = -\frac{1}{m} \sum\limits_{i = 1}^{m} (y^{(i)}\log\left(a^{[L] (i)}\right) + (1-y^{(i)})\log\left(1- a^{[L](i)}\right))$
Now the back-propagation (this is where my confusion starts and I may have got these equations wrong, so, correct me if I am wrong):
at every layer:
$l$ = a given layer, = 1,2,3
$\partial Z^{[l]} = \partial A^{[l]} * g'(Z^{[l]})$
$\partial W^{[l]} = \frac{\partial \mathcal{J} }{\partial W^{[l]}} = \frac{1}{m} \partial Z^{[l]} A^{[l-1] T}$
$\partial b^{[l]} = \frac{\partial \mathcal{J} }{\partial b^{[l]}} = \frac{1}{m} \sum_{i = 1}^{m} \partial Z^{[l](i)}$
$\partial A^{[l-1]} = \frac{\partial \mathcal{L} }{\partial A^{[l-1]}} = W^{[l] T} \partial Z^{[l]}$
And now we use dW and db at a respective layer to update weights and bias at that layer. That completes a Gradient Descent iteration. My primary aim from this question is to know, how to start backpropagation after having computed the cost? What do I compute first and the final layer? And then compute what?
And Where am I wrong and what have I missed? It would be really helpful if you shed some light and help me understand calculations that take place in each iteration of back-propagation.
This is more of a clarification or doubt than a question. Please do not downvote this. I am a beginner trying to grasp concepts of neural networks.