texts: Solution of quadratic optimization in support vector machines

texts:
In support vector machines, the minimization problem with inequality constraints can be converted to a minimization problem of Lagrange multipliers with equality constraints by KKT condition and Lagrange duality. The Lagrange multipliers have to be positive definite (or negative depending on the definition of the Lagrange multipliers).

How to solve this minimization problem of positive definite Lagrange multipliers with equality constraints? Maybe I can add additional Lagrange multipliers to get rid of the equality constraints, but how to deal with the fact that the original Lagrange multipliers are positive definite?


texts: Fixing autocorrelation not caused by time series

texts:
I have data with features that looks like this
I am using ML regression in Sklearn to predict a final cost (in a separate df). Before fitting a linear regression I went to test the assumptions of linear regression and have a problem with autocorrelation. Running an autocorrelation test gave me this
As you can see the Durbin Watson is too low. Someone suggested to me since some of my variables may be affected by time, to use ARIMA models to fix it. So to test if any of my variables were non-stationary I used an ADFULLER test and got these results.
The P-value of 0 for all of the variables indicates that the variables are already stationary meaning time shouldn't be the issue. Does anyone have a suggestion to get rid of this autocorrelation in my data?


texts: Computing Item-to-Item Similarity Using Cosine

texts:
I have a "User x Item" matrix as below:

user item1 item2 item3
u1   2     0     3
u2   1     2     0
u3   4     3     1
u4   0     2     2

I want to computer the similarity between items based on users.
For example, to calculate similarity between items i1 and i2, I only choose users who have assigned values to both of these items.

The cosine similarity between two items (i1 and i2) is as follows:

cos(v1,v2) = (1*2 + 4*3)/sqrt[(1 + 4)*(16+9)]

My question is for item-to-item similarity, should I consider all users, or just common users who assigned values to the items?


texts: Learn and sample from a generative model with 2000 boolean features

texts:
My data is a set of about 1 million training examples, where each example is represented by about 2000 boolean features. Examples aren't labelled - it wouldn't really have any meaning in this domain. The features fully define the example. My goal is to learn a generative model that I can use to generate more examples. I would like to capture some of the dependencies between the features (e.g. if feature 10 is on, feature 14, 15, and 16 are usually on).

I don't have much background in ML, so I'm not sure what a simple way to do this would be (both algorithms and software packages). My initial thought was to find a structure learning package to learn a Bayesian network and then sample from it, but I figured I'd ask here to see if anyone had any recommendations.

Thanks


texts: What tasks you train with one set of features and predict with another?

texts:
The most common scenario in supervised learning is to have data points with a set of features and train a model to make classification predictions afterward.

Usually, for predictions to make sense with new data points, these new data points need to have the same features and come from the same distributions.

My questions is: what are scenarios in which the data points you want to predict do not necessarily have the same set of features from the data points you trained your model with?


texts: Which classification algorithms are negatively affected by class imbalances?

texts:
I've seen a few posts and papers floating around the web (mostly those related to over/undersampling, SMOTE, and cost-sensitive training) that, when discussing class imbalance, specify that certain algorithms are negatively impacted by class imbalance.

Which algorithms are those? Which are not? How can we figure out whether or not an algorithm or approach will be negatively affected by class imbalance?


texts: How to visualize trending over geographical area?

texts:
I would like to visualize how a percentage evolves in time over a geographical area.

In particular I have the percentage of products delivered in different regions and I want to show the weekly trend for each of the 21 regions. Currently I show a heatmap with the values of the current week and three line plots showing the trends of the 3 worst regions. I would like to show the full picture. Unluckily I can't use animations as I have to put the visualization on a static powerpoint.

I'm working with d3, sql and excel.
Any idea?


texts: the feasibility of image processing techniques for physics based images

texts:
When building deep learning models for image analytics-related applications, we sometimes apply various types of operations to enhance the image, such as an image denoising operation.

In my study, we have images generated by physical simulations. In other words, the physical simulations generate matrices of dimensions such as 256*256, which can be visualized as an image as well.

I am trying to apply a deep learning model to perform some analysis over these physics-based images. In the pre-processing steps, I can always apply those image analytics related techniques to pre-process my images, but I am not sure whether it makes sense. For example, denoising or some other contrast enhancement operations can be used to improve the quality of images, like photos. But would it make sense to use them process the images generated by physics simulation?


texts:
I want to use Natural Language Processing to analyze traffic accident reports and from the text determine two things:

Direction of vehicle travel (just compass directions like north, southeast, etc.)
Vehicle movement descriptions (e.g. backing, turning left on a red, stopped in traffic, turning right, parked).

A fragment of an accident report narrative looks something like this:

  TU2 was parked and attended on the west side of Main St, facing west,
  engine turned off. TU1 was parked on the east side of Main St, facing
  east and exited the parking spot, driving backwards. TU1's rear
  collided with TU2's rear. TU2 was still parked with the engine off
  when rear ended by TU1.

The accident reports are in thousands of files, with all accident reports for one year in one city in a single file.  The "answers" (labels) to vehicle travel direction and movement descriptions are provided, along with the narrative of the accident report.  So I have a decent training dataset.

I was thinking about starting with an approach like an n-gram bag of words and a simple classifier for vehicle direction (north, southwest, etc.).  Would that be a good start?


texts:
Kmodes is for categorical Data. But sometimes I read it somewhere that it is applicable on Mixed Data i.e., Numeric and categorical.

So whats right?


texts: Implicit Training Models in Spark MLlib?

texts:
I have something of a ground floor question I’d like to ask.  I’m looking at various options for recommendation engines using Spark.  I feel that I have a decent grasp of the basics of collaborative filtering and have been able to replicate the simple examples provided by the Spark docs and AMPlab.  However, I am seeking to investigate how to create an effective Product Recommendation Algorithm using Implicit Feedback where there is an abundance of user level data available.  Let’s say that I have a large dataset with individual user attributes such as a unique user ID, Product ID, user demographic data, user behavioral data, etc.  What this data set does not contain is an explicit rating that the user gave the product.  In the absence of that explicit feedback, is it still possible to train a Recommendation model using:

ALS.trainImplicit(ratings, rank, numIterations, alpha=0.01)

Furthermore, I found this statement from the Spark website to be a bit nebulous: “If the rating matrix is derived from other source of information (i.e., it is inferred from other signals), you can use the trainImplicit method to get better results.”  Does this mean that the large table from my hypothetical example would need to be compacted or reduced to a singular value for the ‘rank’ argument in the trainImplicit function? I have looked everywhere I can think of, but cannot find an accessible example describing how this would be accomplished.  If anyone knows of a good resource, please let me know.  Lastly, if anyone provides a code example, please do so in Python since that is my preferred Spark API.

Thanks in Advance.


texts: Deep learning based Resume Parser and Scoring

texts:
I want to know if Deep learning can be used for Resume Parsing and scoring of the resume.
Currently what I am doing is extracting the text from pdf or image using OCR/tesseract and finding the features like Email, Mobile No, Skills, Tenure, No of Companies, Awards etc from the text. So I have close to 100 features which are important for scoring the resume.
Can we do similar thing using Deep learning and will the accuracy be better ?
Any starting point/document/blog/github link which can help me get started on this.
I have gone through this link but this doesn't not have code to start with.


texts: Detect multicollinearity in real-life, non-normally distributed data

texts:
I am currently trying to figure out whether my data (consisting of thousands of rows, some is numerical, and some are categorical, and some are ordinal) has multicollinearities or not.

One thing I have noticed is that my data is not normally distributed, based on the Shapiro-Wilk test. As is the case with mostly (if not all) real world data, as answered here

But based on several posts, including this one, many suggests the ANOVA (Categorical vs Numerical) or the Chi-Squared (Categorical vs Categorical ) tests to detect whether or not there are multicollinearities, without implying (at least not specifically) to ensure the data has normal distribution.

My questions are:

Can we actually use these methods parametric methods for non-normally distributed data?
Other than statistical tests, is there a computational model/algorithm to detect multicollinearities in data, parametric or non-parametric? I've read that decision trees algorithms like Random Forests and XGBoost disregards multicollinearities and can also give feature importance information.


texts: Multilabel classification for a learning to rank application

texts:
I am looking for some suggestions on Learning to Rank method for search engines. I created a dataset with the following data:

query_dependent_score, independent_score, (query_dependent_score*independent_score), classification_label

query_dependent_score is the TF-IDF score i.e. similarity b/w query and a document.

independent_score is the viewing time of the document.

There are going to be 3 classes:

0 (not relevant),
1 (kind of relevant),
2 (most relevant)

I have a total of 750 queries and I collected top 10 results of each, so I have a total of 7500 data points.

I have been thinking of estimating a relevance function like:

w0 + w1*query_dependent_score + w2*independent_score + w3*(query_dependent_score*independent_score)

I can clearly see this is like a classification problem but I wanted some info on whether this is right way to approach this problem.

I referred to Machine learning technique to calculate weighted average weights? for some ideas.

Following is the code that I have written:

from sklearn.linear_model import LogisticRegression
import numpy as np

DATASET_PATH = "..."

search_data = np.genfromtxt(DATASET_PATH, delimiter=',', skip_header=1, usecols=(1, 2, 3, 4))
document_grades = search_data[:, 3:4]
document_signals = search_data[:, :3]  # This has 3 features.

total_rows = np.shape(search_data)[0]
split_point = int(total_rows * 0.8)

training_data_X, test_data_X = document_signals[:split_point, :], document_signals[split_point:, :]
training_data_y, test_data_y = document_grades[:split_point, :], document_grades[split_point:, :]

clf = LogisticRegression(multi_class="multinomial", solver="lbfgs")

clf.fit(X=training_data_X, y=training_data_y.ravel())

print(clf.classes_)  # [0, 1, 2]
print(clf.coef_)  # This is a 3 x 3 matrix?
print(clf.intercept_)  # An array of 3 elements?

Based on the sklearn's documentation coef_ should give me the values of w1, w2 and w3, and intercept_ should give me the value of w0.

But I have a matrix and an array for those weights. I am not sure how to get the values of the weights for the relevance function?

I am looking into learning to rank for the first time, so any suggestions are welcome.


texts: Trying to figure out which the training set is

texts:
Can someone help me on this one? As can be seen in the screenshot, it says the loss is 1/2. Where's that 1/2 coming from? How can I replace the values in the h(s) function?

Source PDF


texts: When is precision more important over recall?

texts: Can anyone give me some examples where precision is important and some examples where recall is important?


texts:
I have been reading about different methods of anomaly detection, their structure and the way they work.

Recently I have been trying to find some scholar articles, writings or books where I can learn more about the history of anomaly detection.

Until now I was not able to find anything, would appreciate if any of you would give me and help or suggestions regarding this.


texts: On what threshold we should resample the data?

texts:
When working with churn datasets, we usually find imbalanced datasets. My question is how to decide on what basis we should resample the data.
For example: while splitting the data before training we split in train and test on the threshold (70-30 or 70-25), likewise if I have 62% of 0 class, 38% of 1 class in this case do we need to resample the data?


texts: Loss function for an RNN used for binary classification

texts:
I'm using an RNN consisting of GRU cells to compare two bounding box trajectories and determine whether they belong to the same agent or not. In other words, I am only interested in a single final probability score at the final time step.

What I'm unsure about is how to formulate the loss function in this case. I see two options:

1) force the network to output the correct label at every time step i.e. if I am providing a positive training sample whose output should be 1, then my loss function would be a vector of ones subtracted by the network's output at each time step

2) only check the output at the final time step and use only that in my loss function.

Intuitively, the second option makes more sense, but I'm sure there are other factors that come into play too.


texts: Sentiment Analysis: Train Separate Models or Use One for All

texts:
We have been tasked with building an in-house sentiment tool and we are going to use it on a multitude of data sources; survey responses, reviews, social listening etc..

This may be obvious to some but I would like to hear some thoughts. So my question is, would you train separate models for each data source since the documents from each source would be quite different? We would then apply the appropriate model depending on the data source.

Or is it best practice to train one model using a sample of all the data sources and apply that model to everything?


texts: Proper derivation of dz[1] expression for backpropagation algorithm

texts:
For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]}) $ ? Could anyone show me why $w^{[2]T}$ instead of just $w$ ?

I have read other articles, but the transposed dimension of $w$ is ignored when chain derivation is applied.

Screenshot from Andrew Ng deeplearning coursera course video:


texts: How to create embeddedings for a column that is a list of categorical values

texts:
I am having some trouble deciding how to create embeddings for a categorical feature for my DNN model. The feature consists of a non fixed set of tags.

The feature is like:

column = [['Adventure','Animation','Comedy'],
          ['Adventure','Comedy'],
          ['Adventure','Children','Comedy']

I would like to do this with tensorflow so I know the tf.feature_column module should work, I just don't know which version to use.

Thanks!


texts: What is difference between "cv2.filter2D" vs Keras "Conv2D" function

texts:
When I have to sharpen an image using opencv, I use:

#Create our shapening kernel
kernel_sharpening = np.array([[0,-1,0],
                              [-1, 5,-1],
                              [0,-1,0]])# applying the sharpening kernel to the input image & displaying it.

sharpened = cv2.filter2D(image, -1, kernel_sharpening)

In above code sharpened is our resultant image. As you can see in above code I used opencv function named filter2D to perform convolution of input image with the kernel, and as a result I got sharpened image.

Recently I went through this link regarding image Super-Resolution (link)

And found out Keras has something similar to filter2D and Keras calls it  Conv2D.

Its syntax is as follows:

dis2 = Conv2D(filters=64, kernel_size=3, strides=2, padding='same')(dis1)

My question is what is the difference between opencv filter2D, and Keras Conv2D ?

(I assume both do the same role of convolution of image with a kernel, I may be wrong pls correct)


texts: Is there a maximum limit to the number of features in a Neural Network?

texts: I have created a dataset which has rather large number of features for example-100,000. Is it too large for a decent computer to handle ( I have a 1080ti )?


texts: Machine learning problem with only original data without test and validation data

texts:
I am new to machine learning and I am trying to solve a problem where I have to predict if a customer will buy a home insurance product or not.

I have got a dataset which tells me that which of the bank's customer bought a mortgage from the bank.
I have got another data of the customers who bought the mortgage first, then the bank ran a campaign to provide them with home insurance randomly and this dataset tells me that which of the mortgage customers actually bought home insurance from the bank.

Now my job is to predict which customers should I pick for the bank that will have the highest possibility to subscribe to the home insurance product.
I do not have a separate train/test/validation dataset, but just one dataset. How do I  approach this problem? Should I create the validation and test data from my original dataset that I have been given? how should I approach this problem to predict correctly?


texts: Efficient way to create matrix that shows if data exits per day

texts:
So I have a dataset containing different ID's and the time the data was created.
    ID              Date
0   123123          2021-03-24 12:43:13.494000+00:00
1   123412          2021-03-24 12:43:13.494000+00:00
2   123123          2021-03-24 12:43:15.935000+00:00
3   234234          2021-03-24 12:43:15.935000+00:00
4   432424          2021-03-24 12:43:13.494000+00:00

The goal should be to validate that there is at least one data row for every id for every given day.
What I did so far is converting the timestamps to dates like this:
0         2021-03-24
1         2021-03-24
2         2021-03-24
3         2021-03-24
4         2021-03-24

Now I missing a solution to create a matrix that tells me if for the given ID on the given date a data row of this data exists or not:
The data frame creation looks like this:
df = pd.DataFrame(index=df['id'].unique(),columns=df['date'].sort_values().unique())

which creates this matrix:
    2021-03-19  2021-03-20  2021-03-21  2021-03-22  2021-03-23  2021-03-24  2021-03-25
12341   NaN NaN NaN NaN NaN NaN NaN
12312   NaN NaN NaN NaN NaN NaN NaN
12324   NaN NaN NaN NaN NaN NaN NaN
12345   NaN NaN NaN NaN NaN NaN NaN
12345   NaN NaN NaN NaN NaN NaN NaN
... ... ... ... ... ... ... ...
12399   NaN NaN NaN NaN NaN NaN NaN
12394   NaN NaN NaN NaN NaN NaN NaN
34567   NaN NaN NaN NaN NaN NaN NaN
98764   NaN NaN NaN NaN NaN NaN NaN
10023   NaN NaN NaN NaN NaN NaN NaN

sure I could just use loops to now fill the values, however this would be a a really inefficient way to do it.. I think there is a better way and I am super sure somebody can tell me how so I can learn it for the future.
Thank you very much!


texts: Classification model accuracy with ensemble methods

texts:
I came through this statement in a Machine Learning text book based on law of large numbers:
Suppose you build an ensemble containing 1,000 classifiers that are individually correct only 51% of the time (barely better than random guessing). If you predict the majority voted class, you can hope for up to 75% accuracy!

I understand the analogy if we consider average over 1000 predictions but how majority votes lead to 75% accuracy from 51% (individual)?


texts:
While i am using the confusionMatrix in R, I am getting the following error:

Error in binom.test(sum(diag(x)), sum(x)) :
  'n' must be a positive integer >= 'x'

The code i use as follows:

rf.pred <- predict(rf, testing, type = "class")
confusionMatrix(rf.pred, testing$target)


texts:
I have two models, $m_1$ and $m_2$, and I want to ensemble them into a final model. I want to be able weight one or the other more according to a grid search. There are two main ideas that come to my mind when doing so:

Define a family of models $m_1 \cdot a + m_2 \cdot (1 - a)$, where $0 < a < 1$, find the $a$ that gives the best score.
Define a family of models $m_1^a \cdot m_2^{1 - a}$, where $0 < a < 1$, find the $a$ that gives the best score.

However, in certain cases, I've seen top models in Kaggle competitions doing fairly different things, like having a final model of the form $m_1^a + m_2^b$.
My question is, what are the advantages and disadvantages of every solution? When do they work better and when do they work worse? When is the third kind of ensemble suitable, and is there any heuristic to tune $a$ and $b$?


texts:
I have written this code

fig, (axis1, axis2,axis3, axis4)=plt.subplots(2,2,figsize=(10,4))

and I am getting this error

ValueError: not enough values to unpack (expected 4, got 2)

I tried many ways to remove this error but all was in vain.
Can you explain to me why I am getting this error?


texts: What is the difference between word-based and char-based text generation RNNs?

texts:
While reading about text generation with Recurrent Neural Networks I noticed that some examples were implemented to generate text word by word and others character by character without actually stating why.

So, what is the difference between RNN models that predict text per-word basis and the ones that predict text per-char basis? Do word-based RNN require a bigger corpus size? Do char-based RNN generalize better? Maybe the only difference is input representation (one-hot encoding, word embeddings)? Which ones to choose for text generation?


texts: Which Deep Learning architecture is best for classifying short videos of variable length? I would like to classify videos that last from 1 up to 3 seconds.


texts: Where can I find datasets with labeled duplicate text documents?

texts:
I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate documents from the corpus.
Where can I find such data?


texts: How to write formula inside the loop to run this code in every hour continously in every day in panda python

texts:
I have a csv file with five column data value.

five columns are ['A','B','C','D','time']

Here my 'D' column is the output column and 'A'is my first column.

Here I upload my coda and csv file.

data = pd.read_csv('data20.csv')
data = pd.DataFrame(data,columns=['time', 'A','B','C','D'])
for i in range(1, len(data)):
   data.loc[i+1,'A'] = data.loc[i, 'D'] + data.loc[i, 'B'] - data.loc[i, 'C']

my csv file ;

In my csv first row in A column 63 is represented and my output is 63 in D column. D column is my desired outputs. then in second row in A column its represented as 0. but I want to apply previous output value(63) into second row as A column input. Then that value will calculate by column B and reduce by C column then my desired output value is 60 represent in D column. So like this process will continue. So in A column 0 values replace by previous output value in D column.

Here I want to call that every hour this process continue. In my csv file till to 2 p.m output value is 72.

So then this 72 is will be the input of A column at 3p.m.then B and C column values will calculate or reduce using this formula will give the output for at 3.00 p.m.

So like this I want to find the value of D column at 4p.m, 5 p.m, 6 p.m till to 12 midnight and it will continous to next day. So I wrote the code and I ddn't have any idea to give time for this process. Can anyone help me to solve this problem?


texts: What might explain a dramatic vertical jump in sensitivity

texts:
Context: I'm training an RNN with LSTM layers using the keras api. I have sequences of 20 timesteps, but just a binary response. For example: X=[[[1,2,3], [2,3,1]], [[4,1,2], [2,1,3]]], and y = [[1.0], [0.0]]. The binary variable y explains the action of the next timestep, because that is all I'm interested in, even though each vector in the sequence has an associated response to it. When training the model with one timestep ahead, I get specifications that look like the following.

Everything looks normal there. However, since I also want to predict two timesteps ahead, I train the same model, but cutoff the last vector in the sequence, but keeping the same response, thereby forcing to look ahead by two timesteps. After plotting the same curve, something strange happens:

A sharp vertical jump occurs around the cutoff of .43, in such a way that I've never seen before. When I test it with 3 timesteps ahead, the exact same thing happens, just at a different cutoff (around .2). What I curious about, is this common, and is it indicative of something greater? Why would such a drastic change in just the sensitivity occur?

I'm not sure if it's relevant, but this is the layout of the model:

model = Sequential()
model.add(Masking(mask_value=0, input_shape=(seq_length, 66)))
model.add(LSTM(64, return_sequences=False, W_regularizer=regularizers.l2(.001)))
model.add(Dropout(.2))
model.add(Dense(1, activation='sigmoid'))
rms = RMSprop(lr=.001)
model.compile(loss='binary_crossentropy', optimizer=rms, metrics=['acc', 'fmeasure'])


texts: Matching Data Text of Two Place with Exception

texts:
I have data of two places name and it's address in a row and i have to match it. Data is text type, I have read, it have to convert to numeric type that generated by the text. I extracted the numeric from the name place and address using fuzzywuzzy score matching and counting the length of it and make it the feature for classifying to.
This is the structure of data (this is just example but represents the data)
id     place_name    address                                         places_name_candidate_match   address_candidate_match                            match
1      Liberty Bell  143 S. 3rd Street, Philadelphia, Pennsylvania   Liberty Bell                  S. 3rd Street, Pennsylvania                        True
2      Liberty Bell  143 S. 3rd Street, Philadelphia, Pennsylvania   Liberty Bell                  S. 3rd Street, Pennsylvania                        True
3      Hershey Park  100 Hersheypark Drive, Hershey, Pennsylvania    Hershey Park                  S. 3rd Street, Pennsylvania                        False
4      Hershey Park  100 Hersheypark Drive, Hershey, Pennsylvania    Hershey Park                  100 Hersheypark Drive, Hershey, Pennsylvania       True

I would like to make an exception for a place which is a park that if it isn't park has different addresses but same place name, the result is still true, but if it is specifically park, the place name is the same but the address is different it should return false. But sometimes there are place names that don't contain the word "park", how do I create a feature to teach machines to make those exceptions? or is there another approach?


texts: How to Bootstrap dataset for 10000 AUC scores?

texts:
I am new to ML and trying to learn the nuances. I work on a binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3554.

What I currently do is

1) split the data into train(70%) and test(30%)

2) initiate a model -->  logreg=LogisticRegression(random_state=41)

3) run 10 fold cv -->   logreg_cv=GridSearchCV(logreg,op_param_grid,cv=10,scoring='f1')

4) fit the model -->  logreg_cv.fit(X_train_std,y_train)

5) Do prediction  -->  y_pred = logreg_cv.predict(X_test_std)

Now my question is, how to generate 10000 AUC scores.

I read that people usually do this get a confidence interval of their train and test performance AUC scores.

So, I would like to know how to do this?

I know that bootstrap means generate random samples with replacement from same dataset. But do we still have to split them into train and test? But this looks no different than CV. How do we generate 10000 AUC's and get a confidence interval?

Can you help?


texts: What is the "matrix trick" in recommendation systems?

texts:
I just found slides from Matt Gormley (CMU) about recommendation systems. Under the heading "Unconstrained Matrix Factorization" he mentions:

Optimization problem
SGD
SGD with Regularization
Alternating Least Squares
User/item bias terms (matrix trick)

What does "User/item bias terms (matrix trick)" mean?


texts:
I'm trying to understand the concept of SVM.
Consider linearly separable data $\{(x_i , y_i )\}_{i=1}^n , x_i \in \mathbb R^d , y_i \in \{−1, 1\}. \text{Let}\ \ \{x | w^T x + b = 0\}$ be the margin-maximizing hyperplane that separates the data.
Intuitively I understand that since there is a classification problem with at least 2 classes, there should be at least two support vectors (one for each class). But is there any formal proof of that (the minimum number of support vectors is 2)? And could there be more than two support vectors? If so, can you give an example?


texts: Neural Network for classifying input data to maximize accuracy

texts:
Recently I've been working on some deep neural networks for binary classification, and in developing the appropriate features, I found the necessity to classify the input data. For example, if the input feature is consisted of information five different objects listed sequentially, I want to add the "class" each object, increasing the feature size by five.

The problem is that I'm not quite sure how the inputs would be classified, or would be optimally classified. Therefore I was wondering if there was a method to train a network to find the appropriate classification method for the inputs, in order to maximize the training result. I found that the method used for word2vec wouldn't work here because in a sense, this would be unsupervised learning, because there wouldn't be any label for the training.

Thank you


texts:
If I understand overfitting is simply when our training loss is lower than our test loss. Now let's say we have 2 situations

Where your test loss actually starts going up after a certain number of epochs

Where your test loss is bigger than the train loss, but both are constantly decreasing.

If I understand correctly both of these would be considered overfitting, however are there different names for such occurrences?


texts: Is too much or very few training sample of a specific feature hamper the neural network model?

texts: I am analysing a technique "Sherlock" - a semantic type of column detecting technique wherein training dataset too many samples of a specific type are limited up to 15K and too few occurring samples exist less than 1K per class also excluded. What is the reason behind this? What are the disadvantages having too much or very few samples of a specific type in the input of a neural network?


texts:
I have 2 dataframes:

df1

Id     CategoryId
1       A
1       B
2       A
2       E
2       F

df2:

Id   A    B   C   D   E   F
1
2

I want to do a lookup which will help me fill up the values in df2 based on the values of df1

If df1 has id = 1 and CategoryId = A then I want df2 row 1 Column A to say Y else N

Final df2 should look like:

 Id   A    B   C   D   E   F
 1    Y    Y   N   N   N   N
 2    Y    N   N   N   Y   Y

I am not sure how to do this in Python.
I would really appreciate any help on this!

Thanks


texts: How do I deal with additional input information other than images in a convolutional neural network?

texts:
I try to convert a game state of a board game into the input for a convolutional neural network. A convolutional neural network is useful because the players have to place items on the board, and the convolutional neural network can take advantage of that spatial structure. Therefore I can describe the board well with a binary feature plane for each player (1 if there is an item on the board and 0 if not).
The players can not only place objects but also collect cards. A player can own a maximum of 19 of five different card types. How many cards a player has and what type they are, is important information for the neural network, but I cannot describe this with another feature plane. This has nothing to do with spatial structure. So how do I give the convolutional neural network such additional information, for example that the player has 6 cards of type A, as input?
There are also "places" where a player can place his items. These places have a number (so to speak how good the place is) from 2 - 12. I can describe these places again with a feature plane. But I wonder if the network distinguishes well enough between two numbers like 6 and 7? I could imagine that it can distinguish much better between ones and zeros.


texts: Size of Output vector from AvgW2V Vectorizer is less than Size of Input data

texts:
Hi,
I have been seeing this problem for quite some time. Whenever I tried vectorizing input text data though avgw2v vectorization technique. The size of vectorized data is less than the size of the input data. Is there any statistical reason behind this? In my case 100K is the size of the input and it gives 999,98 sized output
I'm wondering what is causing this problem. Thanks in advance

Code:

listofsentences=[]
for sent in x_train:
    listofsentences.append(sent.split())

training_model = Word2Vec(sentences=listofsentences, workers=-1,min_count=5)
modelwords = list(training_model.wv.vocab)

std_avgw2v_x_train = []
for everysentence in tqdm(listofsentences):
    count = 0
    sentence = np.zeros(100)
    for everyword in everysentence:
        if everyword in modelwords:
            w2v = training_model.wv[everyword]
            count += 1
            sentence += w2v

    if count != 0:
        sentence/=count
        std_avgw2v_x_train.append(sentence)

len(std_avgw2v_x_train)
>99998

len(x_train)
>100000

EDIT1: I'd like to mention that I Just started learning ML, Its been 55 days since I started. Also, the same code gives our 100K output samples While I vectorize with TFIDFW2V

I have attached the image of the same. Kindly look into it


texts: Is it possible to pass dataframes between R and Python using Arrow with zero-copy?

texts:
Arrow is an in-memory format, so I can have a dataframe in Python backed by the arrow format. So the dataframe sits in memory, can I use that dataframe directly from R without making a copy of the dataframe? Imagine if the dataframe is 100G in size so copying is very inefficient.

I read in the documentation of Arrow that there is zero-copy streaming, but there isn't a way to make the whole dataframe available.


texts: Log-loss - but only if too far away from target?

texts:
I'm training a neural net on a "needle in haystack" problem - there is a huge amount of data, also with a huge class imbalance (~1 positive per 10k negatives).  If I train the net on examples drawn so as to balance the classes, it works, but either overfits on the positives (since it sees the few positive examples repeated so often), or doesn't see most of the negatives.  The result of not seeing most of the negatives is that there is a number of false positives that crop up, even on examples which were in the training set but which were never used in training.  Presumably if the network was trained on those that would fix the problem.
The partial solution: every once in a while during training, run inference on all training data, and pick the hard examples (either false positive or false negative); use a balanced sample of just those for the next stage.  This does work, but is quite slow, and it is unclear how many to pick, with what thresholds, etc.
The idea: I would like to train so that the log loss only works if too far from the target.  In other words, true=0 and predicted=0.01 should not create a gradient at all; that way the vast majority of true negative examples would just run through a mostly-trained network but basically not do anything; the few false positives would generate a gradient, counterbalanced by the negatives.
Has this been done before?  Does it work?  Is there a better approach in the same situation?


texts: ValueError: When feeding symbolic tensors to a model, we expect the tensors to have a static batch size. Got tensor with shape: (None, 32)

texts:
inp=Input(shape=(max_length,))
x = Embedding(nb_words, EMBEDDING_DIM, embeddings_initializer=Constant(embedding_matrix),input_length=max_length, trainable=False)(inp)
x = SpatialDropout1D(0.2)(x)
x = Bidirectional(CuDNNLSTM(EMBEDDING_DIM, return_sequences=True))(x)
x = Conv1D(64, 5, activation='relu')(x)
x = GlobalMaxPooling1D()(x)
inp1 = Input(shape=(2,))
concat = concatenate([x, inp1])
x = Dense(32, kernel_initializer='normal', activation='relu')(concat)
x = Dropout(0.2)(x)
output = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[inp, inp1], outputs=output)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

when i try to fit the above model it gives me the value error

history = model.fit([X_train_pad, x], y_train, batch_size=1024, epochs=2, validation_data=([X_test_pad, x1], y_test))

Here max_length is 256 and i pad the input data with max_length.


texts: How to set limits of Y-axes in countplot?

texts:
df in my program happens to be a dataframe with these columns :

df.columns
'''output : Index(['lat', 'lng', 'desc', 'zip', 'title', 'timeStamp', 'twp', 'addr', 'e',
'reason'],
 dtype='object')'''

When I execute this piece of code:

sns.countplot(x = df['reason'], data=df)
# output is the plot below

but if i slightly tweak my code like this :

p = df['reason'].value_counts()
k = pd.DataFrame({'causes':p.index,'freq':p.values})
sns.countplot(x = k['causes'], data = k)

So essentially I just stored the 'reasons' column values and its frequencies as a series in p and then converted them to another dataframe k but this new countplot doesn't have the right range of Y-axis for the given values.

My doubts happen to be :

Can we set of Y-axis in the second countplot in its appropriate limits
Why the does second countplot differ from the first one when i just separated the specific column i wanted to graph and plotted it separately ?


texts: Detecting synchronization cascades in time series

texts:
I am researching delayed synchronisation in a system of coupled oscillators. There is a one-way causal connection between the oscillators, which leads to the synchronisation events occurring in a rough sequence.
When I plot this data a very clear pattern emerges:

I am looking for a way of identifying and fitting a line to the highlighted areas of the graph. Do you have any ideas on how to find these patterns?


texts: Test set is representative of population. Is the evaluation of the ML on the test set represents absolute truth how model will behave in real world?

texts:
The question is more theoretic. Lets assume that the test set is perfect representation of the population.
If I will evaluate the machine learning predictive model on the test set, can we call the good results the absolute proof that the model will work in real word? By good result I mean result that we consider acceptable for us to call the model "correctly working and giving acceptable good results"?


texts: Effects of slipperiness in OpenAI FrozenLake Environment

texts:
I am trying to wrap my head around the effects of is_slippery in the open.ai FrozenLake-v0  environment.
From my results when is_slippery=True which is the default value it is much more difficult to solve the environment compared to when is_slippery=False. It takes roughly 10K iterations to solve when is_slippery=True compared to roughly 150 iterations when is_slippery=False.
I used the same cross-entropy method for both of them.
Now my issue is trying to understand the implementation from the repository and how they were able to model steps in the environment in such a way to mimic slipperiness.
This is the implementation from the repository with the different ways steps are taken based on is_slippery.
for row in range(nrow):
            for col in range(ncol):
                s = to_s(row, col)
                for a in range(4):
                    li = P[s][a]
                    letter = desc[row, col]
                    if letter in b'GH':
                        li.append((1.0, s, 0, True))
                    else:
                        if is_slippery:
                            for b in [(a - 1) % 4, a, (a + 1) % 4]:
                                li.append((
                                    1. / 3.,
                                    *update_probability_matrix(row, col, b)
                                ))
                        else:
                            li.append((
                                1., *update_probability_matrix(row, col, a)
                            ))


texts: Difference between neural architecture search (NAS) and Meta Learning

texts:
Is NAS a separate domain when compared to Meta Learning or does it fall under Meta learning?
To be more precise. Is 1 or 2 correct? Or is it something completely different?


texts:
I have a machine learning model that try to fingerprint the functions in a binary file with a corpus. Final output of upon inputing a binary file is a table with one to one mapping between the binary function and the corpus function as follows-:

As you could see from the names, some of the functions are correct while the others are incorrect. Is there a way to calculate precision and recall for the above result? I understand that precision and recall make sense if I am doing other ML tasks such as image classification. Using a confusion matrix will help to calculate both the metrics easily. However, I am confused and feel that I could not do such measures as it is just one to one mapping which is either true or false. If precision and recall does not make sense, is there any other metrics I could use to evaluate the model? Thank you!


texts: The definitions of machine learning I was able to find are very vague. What are the common properties of all machine learning algorithms? For example, keyword extraction algorithm (that use only statistical formulas) is ML? Language detection algorithm (that use dictionlaries for every language and statistic) is ML?


texts: Abbreviation Classification using machine learning

texts:
I would like to classify abbreviations using machine learning. For example:
I have watermel. and I ask for user what is watermel.(my application context is about food). Then He classify as watermelon.
In other time, If Other user insert waterme. Is It exist a way to infer that waterme is the same as watermelon using machine learning techniques?


texts: How to use multiple text features for NLP classifier?

texts:
I am trying to build text classifier, Usually, we have one text column and ground truth. But I am working on a problem where dataset contains many text features. I am exploring different ways how to utilize different text features.

For example, my dataset looks like this

Index_no                   domain  comment_by   comment       research_paper      books_name

01                         Science  Professor   Thesis needs  Evolution of         MOIRCS
                                                more work     Quiescent            Deep
                                                              Galaxies as a        Survey
                                                              Function of
                                                              Stellar Mass

02                         Math    Professor   Doesn't follow  Evolution of
                                               Latex format   Quiescent           nonlinear
                                                              Galaxies as a       dispersive
                                                              Function of         equations
                                                              Stellar Mass

This is just a dummy dataset, Here my ground truth (Y) is domain and features are comment_by, comment, research_paper, books_name

If I am using any NLP model (RNN-LSTM, Transformers etc), those models usually take one 3 dim vectors, for that if I am using one text column that works but How to many text features for text classifier?

What I've tried :

  1) Joining all column and making a long string

Professor Thesis needs more work Evolution of Quiescent Galaxies as a Function of Stellar Mass MOIRCS Deep Survey

  2) Using a token between columns

<CB> Professor <C> Thesis needs more work <R> Evolution of Quiescent Galaxies as a Function of Stellar Mass <B> MOIRCS Deep Survey

where <CB> comment_by , <C> comment, <R> research_paper, <B> books_name

Should I use <CB> at the beginning or use like this?

Professor <1> Thesis needs more work <2> Evolution of Quiescent Galaxies as a Function of Stellar Mass <3> MOIRCS Deep Survey

  3) Using different dense layers (or embedding) for each column and
  concatenate them.

I've tried all three approaches, Is there any other approach I can try to improve the model accuracy? or extract, combine, join the better features?

Thanks in advance!


texts: Columntransformer multiple columns with vector inputs

texts:
This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this.

My question is related to the sklearn's ColumnTransformer class.
Considering 'description' corresponds to a text/string data column with 9508 rows. The following works as expected:

In [61]: transformer = ColumnTransformer(transformers=[('text-features', CountVectorizer(), 'description')])

In [62]: X=transformer.fit_transform(df)

In [63]: X.shape
Out[63]: (9508, 5913)

However, the following (note the [] around 'description'):

In [64]: transformer = ColumnTransformer(transformers=[('text-features', CountVectorizer(), ['description'])])

In [65]: X=transformer.fit_transform(df)

does not work as expected:

In [66]: X.shape
Out[66]: (1, 3)

Note that there is no issue parsing the list of column names for other transformers such as OneHotEncoder(). This is making it difficult to programmatically configure transformers for dataframes containing heterogenous data type columns.

One possibility is to configure transformer for each column individually even if multiple columns require the same transformer but I was wondering if there is better way to deal with this?


texts: Is it possible to create a 3d to 2d U-net?

texts: I was curious if it is possible to create a U-net type architecture that takes in a 3d image and outputs a 2d image? Or, alternatively, would some other architecture be better suited for this problem? Has anyone tried stacking 2d conv layers in keras (or any other lib) on top of 3d conv layers? I am about to try this myself, but was just wondering if it makes sense, and if any deep learning experts cared to weigh in.


texts: Which data science framework should I start learning?

texts: I have learned Python but am confused about which data science framework should I start off with? Matplotlib or Pandas or Numpy? Also, can you recommend which tutorials I should look for each one of them?


texts: Equivariance vs Invariance in Convolutional Neural Networks

texts: Could someone please explain to me in details (possibly from mathematical point of view) what is the role of Equivariance and Invariance in Convolutional Neural Networks, and how are they actually achieved? I found a similar question here but I did not get the full picture.


texts: Are there any control-flow/conditional statements in AI/ML models?

texts:
I was recently asked this during an interview.
When we write a C program, it has a control-flow in the form of conditional statements like if, while, for that executes a condition, and that decides how the control flows through the application.
In AI/ML models, e.g. CNNs like ResNet, I don't see any control-flow (either static or dynamic). It's just a series of consecutive convolution layers. So,

where are these control-flow/conditional statements in AI domain? What do they look like?
Which models have them? Any examples?


texts: Best way to fuse clusters smaller than a certain size in a distributed way?

texts:
I have X clusters of size (n1,n2...,nX) and I want to fuse small clusters with less than T members with the nearest adjacent cluster i.e. the nearest neighboring cluster.

The problem is when I've got two small neighbor clusters, A->B and B->A which create an infinite loop.

I try to avoid that by removing one of those which have this property [(A,B),(B,A)].

But is there a better way to accomplish this task in sequential way ?

Is it possible to do the trick in distributed way ( Spark ) ?


texts: What to do when seed has a big impact on model performance?

texts:
I have a training procedure set up for an image recognition task. Each time I train a model, I record training loss, validation loss, validation precision and validation recall.
Recently I switched from EfficientNet to a ResNet-based model. Both models are pretrained, so weight initialization is deterministic. With the old model I ran 5 experiments (each on 5-folds) with exactly the same parameters, varying only the seed and got around 0.001 standard deviation for validation loss. With the new model I ran 3 experiments (also with 5 folds) with exactly the same parameters and got a standard deviation of 0.028 for validation loss.
This high variance makes it very hard to interpret the effects of changing non-seed parameters when running new experiments with the new model, since in many cases the differences in performance sit within one standard deviation. Experiments take multiple hours each, so it's not feasible to run multiple experiments for each condition.
What does one do in cases like this?


texts:
I am trying to build a LSTM model in keras where I have one question with 10 answers but only ONE among them is correct. So basically im tring to build a 10 class classification problem. As most of the research papers are using Mean average precision(MAP) and Mean reciprocal rank(MRR) to evaluate this type of problems, i want to use them as custom metrics.

So my first question:
Is the below code correct to calculate mean average precision?

def MAP(ground_label, predict_label):
mAp = 0
map_idx = 0
extracted = {}

for idx_, glab in enumerate(ground_label):
    if ground_label[idx_] != 0:
        extracted[idx_] = 1

key = np.argsort(predict_label)[::-1]
for i, idx_ in enumerate(key):
    if tuple(idx_.tolist()) in extracted:
        map_idx += 1
        mAp += map_idx / (i + 1)

assert (map_idx != 0)
mAp = mAp / map_idx
return mAp

And the second question is how can i modify this snippet of code to use this as a custom metric function because keras custom metric assumes that the inputs are tensors to the function.


texts: Eigenvalues of covariance matrix are negative

texts:
I'm working on the PCA of the mnist dataset, and I get a very strange result, I created a matrix whose rows are flattened mnist images, When I try to compute the eigenvalues of the covariance matrix, I get some negative values. But the covariance matrix is positive semi-definite.
np.linalg.eigvals(np.dot(mnistBis[:, 0:20].T, mnistBis[:, 0:20])) # mnistBis.shape=(60000, 784)

array([ 4.79599869e+02, -1.19628465e+02,  9.68398702e+01,  1.88726171e-01,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00])

Modification:
Here is the full code
import tensorflow.keras.datasets.mnist as mnist
import matplotlib.pyplot as plt
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()
mnistBis = np.reshape(x_train, (-1, 28*28))
np.linalg.eigvalsh(np.dot(mnistBis[:, 0:20].T, mnistBis[:, 0:20]))

```


texts: About different structures of neural network

texts:
https://www.mathworks.com/help/deeplearning/ref/fitnet.html is the tutorial that I am following to understand fitting data to a function. I have few doubts regarding structure and terminologies which are the following:

1. Model & number of hidden layers

By hidden layer we mean the layer that is inbetween the input and output. If number of layers = 1 with 10 hidden neurons (as shown in second figure) then is it essentially a neural network which is termed as an MLP. Is my understanding correct? In general,

if the number of hidden layers = 0, we call the NN as a perceptron.
If the number of hidden layers >=1 but less than 3, the NN becomes an MLP. Is the picture in the link that of an MLP since it contains 1 hidden layer of 10 neurons?
if the number of hidden layers >3, the NN is called as deep NN aka deep learning

Is that correct?

2. Linear vs nonlinear mapping function

The resulting model eventually learn to map the input to output data.

Do we call the machine learning model as linear or nonlinear ? Or is
this term associated to the mapping function ?
Which layer's mapping function determines this? Based on which layer's activation function do we say that the mapping function or the model is linear or nonlinear? For ex, In this picture, the last layer is the output layer and the activation function looks like an identity/linear. But the hidden layer has sigmoid activation function which is nonlinear. Therefore, is this model a nonlinear function?


texts: Look for previous date in dataframe that has certain column category in R

texts:
I have the following data frame:

      Date.POSIXct       Date      WeekDay DayCategory Hour Holidays   value
1 2018-05-01 00:00:00 2018-05-01      MA    MA-MI-JU    0        0      30
2 2018-05-01 01:00:00 2018-05-01      MA    MA-MI-JU    1        0      80
3 2018-05-01 02:00:00 2018-05-01      MA    MA-MI-JU    2        0      42
4 2018-05-01 03:00:00 2018-05-01      MA    MA-MI-JU    3        0      90
5 2018-05-01 04:00:00 2018-05-01      MA    MA-MI-JU    4        0      95
6 2018-05-01 05:00:00 2018-05-01      MA    MA-MI-JU    5        0       5

DayCategory groups days of the week in the following way: Mondays goes to LU DayCategory. Tuesday, Wednesday and Thursdays go to MA-MI-JU DayCategory.
Friday goes to VI, Saturdays to SA and Sundays to DO Categories respectively.

I would like to find the value for the same hour in the previous day (Date) with the same DayCategory, while Holidays remains within the same group (e.g. if one instance has holiday 0 but previous day with same DayCategory has 1, we should lookv for the previous one, etc.)

As an intermediate step and to understand the process I would like to add a column PreviousDaySameDayCategory with the Date of the previous day that has the same DayCategory that the corresponding row. Some times it will be just the same date minus seven days ("LU","VI","SA","DO") but other days it will be just one day.

Reproducible data:

library(lubridate)
Date.POSIXct <- seq(as.POSIXct("2018-05-01"), as.POSIXct("2018-05-31"), "hour")
mydf <- as.data.frame(Date.POSIXct)
mydf$Date <- as.Date(substr(as.character(mydf$Date.POSIXct),1,10))
mydf$WeekDay <- substr(toupper((weekdays(mydf$Date))),1,2)
mydf$DayCategory <-as.factor(ifelse(mydf$WeekDay == "MA" | mydf$WeekDay == "MI" | mydf$WeekDay == "JU",
                                    "MA-MI-JU", mydf$WeekDay))
mydf$Hour <- hour(mydf$Date.POSIXct)
mydf$Holidays <- c(rep(0, 24*7),rep(1, 24*7), rep(0, 24*16+1))
set.seed(123)
mydf$myvalue <- sample.int(101,size=nrow(mydf),replace=TRUE)

I have manually started the first days and craeted a vector of how the solution should look like:

  a <- rep(NA, 24)
  b <- mydf$value[1:24]
  c <- mydf$value[25:48]
  d <- rep(NA, 24)
  e <- rep(NA,24)
  f <- rep(NA,24)
  g <- rep(NA,24)
  h <- rep(NA,24)
  i <- mydf$value[169:192]
  solution <- c(a,b,c,d,e,f,g,h,i)
  solution

I would appreciate any hint in the thinking process to solve this kind of problems that I face with relative frequency.

I also published this question in Stack Over Flow, I guess this kind of question fits more in this community than in StackOverflow, is this correct? I would like to delete the one that less fit the commnunity.


texts: How to tune XGBClassifier's each and every hyper-parameter "efficiently"?

texts:
It's a generic question on tuning hyper-parameters for XGBClassifier()
I have used gridsearch but as my training set is around 2,00,000 it's taking huge time and heats up my laptop.

I need codes for efficiently tuning my classifier's parameters for best performance.


texts: Approaches to pre-processing the huge but organised text data, with & without the generators

texts:
I've a huge text file, hence I'm reading it line-by-line, applying some basic cleaning, and separately writing the X & Y to 2 different csv files. Further I'm preparing 3 directories for each csv - train, val & test and writing each line as a separate csv to appropriate directories - This aids in using the fit_generator() method conveniently, by reading these files 1-at-a-time and train the model.

The concern is, before training, I've pre-processing steps and performing those on these many files, 1 file at a time, doesn't seem to be a practical approach(it won't be time-efficient as the operations wouldn't be vectorized, besides there would be lot of read/write on disk since storing each processed file is also inevitable), are there any other approaches in dealing with such scenarios? What are the best practices? Are custom generator functions the only way? Appreciate any help.

Update: Also, What if my processed data-set is a coo-matrix? Is there a viable way other than converting it to dense before writing? Moreover, my concern is neither about optimal resource utilization nor about time efficiency, it is more about what are the different ways of handling such scenarios, an example might help.


texts: Fixed: Keras Neural Network on handwritten digits labeling: Model reads the surrounding space instead of digit

texts:
Our assignment consists of labeling handwritten numbers, ranging from 0 to 9.

We build a Convolutional Neural Network that achieves a good accuracy on our labeled testset. We now want to use this model to predict the numbers on our unlabeled dataset, called Z.

When printing out the predictions, it seems that the model predicts the number 0 way too often and that it scores a very low accuracy.

We have:

X_train of float64 and shape(42000, 28, 28, 1)
Y_train of float32 and shape(42000, 10)
X_test of float64 and shape(28000, 28, 28, 1)
Z_test(the unlabeled set) of floats64 and shape(15000, 28, 28, 1)

When printing out the digits I saw that our train and test datasets look like this:

Our unlabeled dataset, Z, looks like this:

Our code looks like this:

Y_train = train["label"]
X_train = train.drop(labels = ["label"],axis = 1)
X_train = X_train / 255.0
X_test = test / 255.0
X_train = X_train.values.reshape(-1,28,28,1)
X_test = X_test.values.reshape(-1,28,28,1)
Y_train = to_categorical(Y_train, num_classes = 10)
Z_test = unlabeleddata / 255.0
Z_test2 = Z_test.reshape(-1,28,28,1)

Is there a solution to fix this problem?

Thanks!

Edit: The problem was fixed with this code:

Z_test = unlabeleddata - 255.0
Z_test = unlabeleddata / 255.0
Z_test = Z_test - 1
Z_test = np.absolute(Z_test)
Z_test2 = Z_test.reshape(-1,28,28,1)


texts: Derivation of Ward's formula for agglomerative clustering

texts:
Figure describe formula of ward method for hierarchical or agglomarative clustering i.e. increment of total error after merging two cluster.

How did they get

$$\frac{n_A\cdot n_B}{n_A+n_B} ||\vec m_A-\vec m_B||^2 $$

from first line of formula as shown in figure?


texts: Detecting unique icon on video in real time

texts:
For a project I am working on I'd like to be able to detect unique icons/barcodes in video footage. Suppose you have 10 people in the frame where each person is wearing a t-shirt with a similar but unique identifier (something like a barcode) and the programme is to detect each unique identifier.

Here are three options I'm working with now and associated drawbacks:

Using brute force feature matching (SURF, SIFT or ORB) - not sure how accurate this would be in real-world scenarios as people move and rotate so to if the identifiers are very similar can this still achieve good accuracy?

Training tiny YOLO model. Now this seems like a good option but is not very scalable and it just seems like there's a simpler solution
Simple template matching - is not a very good option due to changing perspective, size and angles

Any advise would be highly appreciated!

EDIT

I'd like to add some images for context. The idea is that each person would have a logo with a unique binary patter around it. See images below:

As you can see the two are similar but not the same due to the stripes around it.

I'm applying an Adoptive Gaussian Blur to it and getting 'barcode' like images that are clearly different

Now, it doesn't seem to me that normal SURF matching would be able to recognise the subtle difference in two images at least not reliably


texts: What metrics must i use in my data(unstructured) preprocessing research?

texts:
i am currently working on preprocessing unstructured data (emails,logs,bug reports and irc chats).

i wish to prove preprocessing improves the content quality.

are there metrics available to prove this? that is to compare the raw data and the preprocessed data and show some quality improvement in numbers.

content quality - i mean to say that the preprocessed corpus is more suitable for mining with topic models than the raw one.

I am aware of perplexity. i dont want to use this metric as it only evaluates the topic model performance and not the corpus.Also i wish to evaluate the corpus quality even before the topic model is applied.

Can topic coherence be used as a metric to evaluate the quality of the corpus?
can more coherent topics be generated from preprocessed documents than the raw one?


texts: What model should I use for bounding box detection?

texts: I am working on building a cow detector for a local farm. I have a dataset of images with bounding boxes (not segmentation polygons) for every cow appearing in the images (different number of cows in every image). What is the best and easiest model to use to accomplish this task? Many cows stand behind others so there will be a lot of cases of occlusion. Is there a go-to model that I can get running quickly in Keras to perform well on this dataset? Any tips would be appreciated.


texts: How to approach this data set for linear regression?

texts:
I am new to data science and am currently working on a data science project and have to answer a few questions about the following data set with 18k data points:
https://www.kaggle.com/karangadiya/fifa19

The question I have to answer is as follows:
To which extent can you determine the value of a player using the players most important features?

Features in this case being the columns from 'Crossing' to 'GKReflexes'. I grouped up all the players as follows:
Att = ['RF', 'LF', 'RS', 'LS', 'RS', 'RW', 'LW', 'CF', 'ST'] # attack positions
Mid = ['LAM', 'CAM', 'RAM', 'LM', 'LCM', 'CM', 'RCM', 'RM', 'LDM', 'CDM', 'RDM'] # midfield positions
Def = ['LCB', 'CB', 'LB', 'RCB', 'RB', 'LWB', 'RWB'] # defense positions
GK

I already looked at the data and was able to determine that certain player features are much higher for certain positions.

I plan on using multiple linear regression to answer the research question, but I'm not sure on how to split the data into a train and test set.

I'm thinking of one of the following approaches:

First split the large data set into smaller data sets for each position (GK, DEF, ATT and MID) and then make a train and test set for each position.

or

Leave the large dataset as it is with all of the positions combined and split that into a training and test set.

I am currently leaning more towards the first approach because I think I'll be able to better determine the price of the player using the features that are important to the players position, but I am not sure if that's the correct way to go at it.


texts: Why divide by batch size when back-propagate from softmax + log loss

texts:
Question
In neural network mini batch training, at the back-propagation from the (Softmax + cross entropy log loss) layer, the gradient is divided by the batch size. Please explain why need to do so. This is from a book that implements DNN from scratch in Python.

layer.py
class SoftmaxWithLoss:
    def __init__(self):
        self.loss = None
        self.y = None # softmaxの出力 (output from softmax)
        self.t = None # 教師データ (label)

    def forward(self, x, t):
        self.t = t
        self.y = softmax(x)
        self.loss = cross_entropy_error(self.y, self.t)

        return self.loss

    def backward(self, dout=1):
        batch_size = self.t.shape[0]
        if self.t.size == self.y.size: # 教師データがone-hot-vectorの場合 (when label is one-hot-vector)
            dx = (self.y - self.t) / batch_size
        else:
            dx = self.y.copy()
            dx[np.arange(batch_size), self.t] -= 1
            dx = dx / batch_size                      # <----- Why divide by batch size?

        return dx

common/functions.py
def softmax(x):
    x = x - np.max(x, axis=-1, keepdims=True)   # オーバーフロー対策 (To avoid overflow)
    return np.exp(x) / np.sum(np.exp(x), axis=-1, keepdims=True)

def cross_entropy_error(y, t):
    if y.ndim == 1:
        t = t.reshape(1, t.size)
        y = y.reshape(1, y.size)

    # 教師データがone-hot-vectorの場合、正解ラベルのインデックスに変換
    # When label is one hot vector, convert into index of the label
    if t.size == y.size:
        t = t.argmax(axis=1)

    batch_size = y.shape[0]
    return -np.sum(np.log(y[np.arange(batch_size), t] + 1e-7)) / batch_size

Update
Answered in Why is softmax classifier gradient divided by batch size (CS231n)?


texts: the probability distribution of dependent variables

texts:
There are three variables, X3 is a function of X1 and X2, X2 also depends on X1.The dependency relationship is shown as the following graph.

In specific, X3= f(X1, x2) and X2=g(X1). Therefore, X3=f(X1, g(X1)). If the probabilistic distribution of X1 is known, how to get the probabilistic distribution of X3.


texts:
Does there exist a fast and convenient way for handling such a problem:

class MyModule(nn.Module):
def __init__(self):
    super(MyModule, self).__init__()
    self.linears = nn.ModuleList([nn.Linear(10, 20) for _ in range(10)])

def forward(self, x, indices):
    x = self.linears[indices](x)
    return x

You see i want to access different layers in the network conditioned on an additional input, which is also a list. Further i want to process the whole batch at once and the output.shape != input.shape.


texts:
Question: Can Stata present or output the actual betas/coefficients in a datasheet when it performs the conditional logit model?

I understand that Stata has a "predict" capability, but I need to store the betas/coefficients and use them later, after new factor values have been calculated.


texts: When is it appropriate to split a dataset on a categorical value and generate $n$ models instead?

texts:
When doing regression or classification when faced with a categorical attribute with $n$ possible values there are two options:

Feed this attribute directly into your model.
Partition your data into $n$ pieces based on the categorical attribute and train a model for each separately. During inference choose the model appropriately based on the same attribute.

One of the advantages of approach #2 is that it allows you to do more specific feature engineering. E.g. if you are modeling property prices and you decided to make separate models for residential/industrial properties you can choose separate features that are relevant for each.
Another advantage of approach #2 I can think of is that it can linearize otherwise non-linear relations. E.g. for a residential property having a railroad track nearby almost always heavily reduces property value while for an industrial property it could be a massive value booster.
In general, what factors go into deciding between approach #1 and #2?


texts: I know that Keras is developed for quick deployment. Is it just for beginners or also useful in industry for professionals?


texts:
I am trying to recreate the ResNet50 from scratch, but I don't quite understand how to interpret the matrices for the layers.

For instance:
[[1x1,64]
[3x3, 64]
[1x1, 4]] x 3

I know it's supposed to be a convolution layer but what do each of the numbers represent?


texts:
I am trying to create a association using apriori algorithm.The data contains around 33000 records.Below is the sample of the data

id  code
1   19
1   58
1   111
2   19
2   111
2   167
3   12
3   79
3   85
4   96
5   19
6   58
7   12
7   18
7   40
7   48
7   85
7   86
7   135

In R this data is:

structure(list(id = c(1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5, 6, 7,
7, 7, 7, 7, 7, 7), code = c(19, 58, 111, 19, 111, 167, 12, 79,
85, 96, 19, 58, 12, 18, 40, 48, 85, 86, 135)), .Names = c("id",
"code"), row.names = c(NA, -19L), class = "data.frame")

Using the following code I tried to build the association

# creating a string of codes based on the id
library(dplyr)
Asso2 = Asso %>%
  group_by(patient_id) %>%
  summarise(hi = toString(hcc_id))
# converting into transactions
library(arules)
fact <- data.frame(lapply(Asso2,as.factor))
trans <- as(fact, 'transactions')
# Applying aprior
rules = apriori(trans, parameter = list(supp = 0.001, conf = 0.001,target = "rules"))
rules
inspect(rules)

I am getting totally 96 rules like below with empty lhs and I trying to understand whether we cannot make any rules from this data or am I missing anything here. Since I am novice in this, I would like to get some help.

#     lhs    rhs               support     confidence  lift
#  1  {}  => {hi=19, 96, 108}  0.001021696 0.001021696 1
#  2  {}  => {hi=176}          0.001021696 0.001021696 1
#  3  {}  => {hi=88, 108}      0.001021696 0.001021696 1
#  4  {}  => {hi=72}           0.001051746 0.001051746 1
#  5  {}  => {hi=88, 96}       0.001051746 0.001051746 1
#  6  {}  => {hi=108, 112}     0.001081796 0.001081796 1
#  7  {}  => {hi=84}           0.001081796 0.001081796 1
#  8  {}  => {hi=100, 103}     0.001111846 0.001111846 1
#  9  {}  => {hi=18, 108, 111} 0.001111846 0.001111846 1


texts: scipy fit for t distribution seems broken for bi-modal data

texts:
I am using the scipy.stats.t.fit function, and I am surprised by the results. If I fit on some bimodal data, say

data=[1,1,1,1,5,5]

I get

df=0.39126249808550329
loc=1.0
scale=5.7172845190830792e-21

That is, the scale is effectively zero, and I will never be able to sample anything near 5, just the more frequent data point 1.

I guess you really can't fit on data that is too different from a t-distribution - but is scipy really giving the best t dist fit to the data? I would think that if I compute a sample mean and variance myself, i.e,

df = 5
loc = 2.33
scale = 1.88

That I'd have a better fit, although I haven't computed the likelihood of sampling [1,1,1,5,5] from these two t distributinos.


texts: Machine learning image training set data as whole object but finding part of object

texts:
If the images to test against are only part of the object, does it help to have training set data of the whole object, or is it better for all the training data to be parts of the object? For instance, if test pictures are the legs of a chair or the top of a chair, does it help to have training set image data of the entire chair? Or should I focus on only getting training set data of parts of the chair? I'm planning on using TensorFlow.

I would be classifying the object as a chair, not part of a chair. But if it would give me a higher rate of recognizing chairs by having it recognize parts, it could recognize parts, and I could combine the information at the end. I only need to recognize at least one chair (or much more likely part of a chair) in the image.


texts: I manipulate the time series using the different structures of the neural networks in order to make a prediction, and I wonder if there is a way to choose the parameters of the networks intelligently? from the characteristics of the signal, namely (trend, seasonality ...) can we choose these parameters that will make learning better?


texts: Decoupling the performance impact of batch sizes from their impact on training speed

texts:
We are training a sequence-to-sequence-model to do something like Named Entity Recognition using PyTorch. It turns out, we get the best results with batch size 1. However, this slows down training significantly. Our GPU could support batch sizes of up to 64.
Now it really seems to me that the batch size parameter is doing two very different things:

It is a hyperparameter that clearly has an impact on model performance (e.g. measured in precision/recall).
It organizes how training data is shuttled to the GPU and back to do fast computations there.

Can we not decouple these two things? Like train with a batch size of 64, but then compute the loss function and backpropagate for each training example individually?


texts: Predict compatibility of 2 people as boolean classification problem

texts:
How can I predict the compatibility of 2 people as a boolean classification problem?

I want to know if below is an appropriate approach to modelling compatibility, or if I should be using "market basket analysis" or some other approach instead?

I'm less interested in the specific result below, and more interested in if this is a realistic way to frame this data science problem.

Background:

Assume people only have 3 attributes: compassion, extroversion and humor. These are also boolean and can be modelled as 1s and 0s in a list ([compassion, extroversion, humor]).

So someone with all 3 characteristics would be [1,1,1] and someone with only humor would be [0,0,1].

We have pairs of people who match and do not match, specified by 1 or 0, where 1=match and 0=no_match.

How to solve this?

I don't consider this a simple distance problem (ie: euclidean distance) because its very possible that generally an introvert and extrovert get along, but 2 extroverts don't.

Data:

person1   person2     match?
--------  --------    ------
([1,1,0], [1,0,1]) => 1
([0,0,0], [1,1,1]) => 1
([1,0,1], [1,0,0]) => 0
([1,1,1], [0,1,0]) => 0
([0,0,0], [0,1,1]) => 1
([1,1,0], [1,1,1]) => 0
([1,0,0], [1,0,1]) => 0
([0,0,1], [0,0,0]) => 0
([0,0,0], [0,0,1]) => 0
([0,0,0], [0,1,1]) => 1
([0,1,0], [0,1,1]) => 0
([0,1,0], [0,1,1]) => 0
([0,1,0], [1,0,0]) => 1

What I've tried:

My first thought was to concatenate both individuals' data for each example. Then use that to fit the model.

Data structured as python code:

X_train = [
    [1,1,0,1,0,1],
    [0,0,0,1,1,1],
    [1,0,1,1,0,0],
    [1,1,1,0,1,0],
    [0,0,0,0,1,1],
    [1,1,0,1,1,1],
    [1,0,0,1,0,1],
    [0,0,1,0,0,0],
    [0,0,0,0,0,1],
    [0,0,0,0,1,1],
    [0,1,0,0,1,1],
    [0,1,0,0,1,1],
    [0,1,0,1,0,0],
]

y_train = [1,1,0,0,1,0,0,0,0,1,0,0,1]

X_test = [
    [0,1,1,0,0,0],
    [1,1,0,1,0,1],
    [1,0,0,1,0,0],
    [0,0,0,1,0,0],
    [0,1,0,0,0,0],
    [0,0,0,0,0,0],
]

y_test = [1,1,0,0,1,0]

Computing the match:

from sklearn.metrics import classification_report

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier()
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)

print(classification_report(y_test,y_pred))

             precision    recall  f1-score   support

          0       0.60      1.00      0.75         3
          1       1.00      0.33      0.50         3

avg / total       0.80      0.67      0.62         6

I'm less concerned about the results here and more interested in if this is a proper way to frame this problem.

Can you offer a suggestion?


texts:
I have plotted the above precision recall curve for a binary classification problem where the classes are imbalanced. Classes are either negative or positive There are 3018 observations and 550 of them are positive. This means that only about 18% of the data are positive results. I read from here that the baseline for the precision recall curve is the ratio of the positives to the number of observations (in a balanced dataset the baseline would be 0.5). This is differently from the AUC baseline which is always fixed.
The Precision Recall AUC score is only .44, but the ROC AUC score is .72. What exactly does this mean? I know the ROC AUC is misleading for imbalanced datasets, but a precision recall AUC score of only .44 seems very low. How do I interpret this? Is it actually a bad score? How can I understand this curve in reference to the baseline?


texts: What's the best method to merge N categorical features into one and keep it as categorical

texts:
I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels.
I have more than one sentence that needs to be considered by the model, so I need to find the best method to merge them into one sequence and keep it categorical so the fine-tuned model, BERT, RoBERTa, XLNet, etc; can consider the attention or positional relation between the tokens through the NN.
A resume of the Transformer idea:

Input embedding -> positional embedding -> multi-headed attention (self-attention)  -> feed-forward -> normalization layer | output embedding -> positional encoding -> multi-headed attention (look-ahead mask) -> normalization layer -> feed-forward  -> linear && probability distribution; besides its components encoder and decoder that uses some of the items described before

I have considered the obvious approach of merge the N featured into one string, but it doesn't give me a good result in terms of F1, I thought about polynomial features but I need to keep it as categorical and make relations between the features and them inverse it to categorical looks a lot of work for a lot of redundancy and word embedding, but these methods create a feature space where the categorical features have a meaning by its positions, which are normally discrete atomic symbols. Any tips on how to apply one of these methods with a different approach or a method that I don't know?


texts: Finding a kernel with feature transformation

texts:
Suppose we have feature transformation $\Phi(x) = [1, x_1, x_2, x_1x_2]$. Now we want to find the kernel corresponding to $\Phi$.

What I have done is using kernel decomposition, we have:
$$
K(x, y) = \Phi(x) .\Phi(y)\\
K(x, y) = 1.1 + x_1y_1 + x_2y_2 + x_1x_2y_1y_2 \\
K(x, y) = 1 + \sum_{i=1}^{N}x_iy_i + \prod_{i=1}^{N}x_iy_i
$$

$x, y$ are in $\Bbb{R}^2$

Is that kernel valid?


texts:
I recently tried to run Regularized Greedy Forest algorithm (rgf Classifier) from this package https://pypi.python.org/pypi/rgf_python.

I did run pip install rgf_python, but when I tried to run the example this error pops out in spyder:

  File "C:\Users\me\Anaconda3\lib\site-packages\rgf\sklearn.py", line 111, in <module>
    "config flag 'exe_location' to RGF execution file.".format(_EXE_PATH))

Exception: C:\Users\me\rgf.exe is not executable file. Please set config flag 'exe_location' to RGF execution file.

What should I change in order for config the flag 'exe_location'? May sound beginner but can't find it anywhere.


texts: Are there different types of confidence values?

texts:
Query related to apriori association in ML. I am learning apriori association from a teacher. I have a simple dataset of five transactions as follows.
T1: {Milk, bread}
T2: {Milk, bread}
T3: {Milk, orange}
T4: {Milk, apple}
T5: {apple, orange}
If bread is purchased then how much confidence we have that milk will also be purchased with it?
I think the answer should be, (2÷5)/(2÷5) = 1.0 means 100%. As bread and milk purchased together is 2 times divided by only bread is purchased 2 times. So 1 is the answer.
But my teacher says it should be, (2÷4)/(2÷5) = 1.25 means 125%.
Can someone share their views on the same?


texts: ML that learns to predict and play a simple wagering game

texts:
I have a simple game I'm building for fun, just to see how well ML can work with simple data sets.

Basically it's just a game where it has turns that go like this:

Computer generates a random number $x$, and does not show the player.
Player wagers that they can guess a number lower than $x$. Call the wager amount $w$
Player tries to guess a low number $g$.
If $g \lt x$, then player gains $wg$ points.
If $g \ge x$, then player loses their wager, $w$ points.

If player has funds $f_t$ at start of turn $t$ then another way to put this is:

$$f_{t+1} = \begin{cases}
    f_t + wg,& \text{if } g\le x\\
    f_t - w,              & \text{otherwise}
\end{cases}$$

Here's an example of play:

Start with $f=50$
Turn 1, $x=11$, player wagers $w=8$ and guesses $g=9$. Player gains $+72$ points, so $f=122$ at the end of the turn.
Turn 2, $x=10.5, w=4, g=7$, points change by $+28$, $f=150$.
Turn 3, $x=20, w=1, g=6$, points change by $+6$, $f=156$.
Turn 4, $x=2, w=10, g=15$, points change by $-10$, $f=146$.

I wanted to use ML to try and predict this by feeding it pre-generated turns to see if it can find any patterns. There are only a few variables involved, so I figured it shouldn't be overly complicated. Ideally I would like for the ML to learn how to play the game.

I am wondering what type of ML is applicable to this kind of problem? It is not clear to me where to start, even though I have made the game simple. I have more complex games that I made in the past that I would like to try to apply ML to as well.


texts: GRU learns small-scale features, but misses large scales

texts:
Playing around with weather data, I have set up a simple RNN with one layer of GRUs. It is trained to recover the temperature of the next day, given weather data of the last 5 days, each with 1-hour intervals.

What I find peculiar is that after training several epochs, the result is something that has a lot of the small scale features of the data, but it lacks the large-scale structure. Frequently, there seems to be just an offset between prediction and data, e.g. at x=900 below. On the other hand, the very steep peaks are also not well fitted.

Here is the code of the model:

model = keras.Sequential()
model.add(keras.layers.GRU(units=120,
                         activation='relu',
                         dropout=0.1,
                         recurrent_dropout=0.4,
                         return_sequences=False,
                         input_shape=(120, 14)))
model.add(keras.layers.Dense(1))
model.compile(optimizer=keras.optimizers.RMSprop(),loss='mse')

The training set has 120 (=5 days) of weather data, with 15 variables at each point in time.
For example, these are the first 2 of 120 vectors in the first training set:

print (training_x[0,0:2,:])
[[ 0.87422976 -2.0740129  -2.12744145 -2.05861548  1.04950092 -1.32397418
  -1.53525603 -0.78058659 -1.53697269 -1.53946235  2.29360559 -0.01027133
  -0.01893096 -0.25892163 -1.72366227]
 [ 0.87183698 -2.02652589 -2.07923146 -1.96947126  1.14054157 -1.30976048
  -1.49940654 -0.78875511 -1.49932401 -1.50404649  2.24182343 -0.02325901
  -0.03515888 -0.09510368 -1.72366227]]

They were normalized beforehand.

(I should note it is based on this)

I am trying to figure out whether this is a well-known phenomenon and/or in which direction I should change my model in order to compensate. I am using 120 GRU 'units' followed by a single unit dense layer.


texts: How R performs the operations like Inf + Inf, Inf * Inf, Inf - Inf and Inf / Inf?

texts: In R When we do some operation like Inf + Inf, Inf * Inf gives Inf back, whereas similar operation  Inf - Inf and Inf / Infgives results as NaN, how R actually handles these type of concepts


texts: Methods of building machine learning models

texts:
I have seen pages where they mention 5 methods of building models.

1) All-in
2) Backward Elimination
3) Forward Selection
4) Bidirectional Elimination
5) Score Comparision

I usually implement a linear regression or any algorithm using sklearn

from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X, y)
y_pred = lr.predict(X_test)

How can one implement those 5 methods of building models?

Can anyone explain the importance of this and What is most commonly used one out of these?


texts: Clustering efficiency in a discrete time-series

texts:
Is it possible to identify the point in time where the cluster separation is at its most in a discrete time series clustering?

Say I have 4 clusters of discrete time series and I want to pick a point in time where I can tell with the least bias which cluster it belongs to after a kmeans clustering, what other criteria than classification success can U use to identify my cluster separation performance?


texts: is it acceptable if the reward of test of DQN is lower than reward of training of DQN in minimization problem?

texts:
if we train a DQN over 40000-60000 episodes for 500 time steps. The mean of reward during last 100 training steps is about 1.1 times of reward during the test process.

Environment is stochastic!

The more time steps should be more stable?? What is the effect of time steps?

For instance, the mean of reward during the last 100 episodes of training is 6.1 however the answer during the test is 5.2 (My problem is minimization).
Could we say our DQN is OK? One test random is 120! however another is 400!

Or vice versa, is it acceptable if the reward of test in minimization is more than for instance 2 times of mean of last 100 training set?
Could we say our DQN is OK?

I did not run as a human! but if it is important I will play this game.


texts:
I have the following dataset where each row represents a customer, and each column is a product. So for example, the first customer (row 0) buys product 2 and product 7. The second customer (row 1) buys products 3,6 and 7. And so on...

Basically I want to recommend customers who are already buying certain products, some other products. Here is an example:

Let's say I choose all customers who are buying products 1 and 4. How would I recommend them the next best...let's say 3... products based on what other 1 and 4 customers are buying?

I'd like to use keras neural networks or random forests for this kind of task. Thanks :)

product 1.......product 2.........product 3....... product 4...... product 5........ product 6..... product 7


texts: Is the maximum BDeu Bayesian Network always the empty network?

texts:
I'm recently reading a paper about Scoring Mechanisms for Bayesian Networks.

For the BDeu score, it appears that the maximum possible score of BDeu for Bayesian Network structure learning is zero.

Does it mean that the best network is always the empty network?


texts: Extracting information from table within invoices

texts:
I am trying to find specific information from supplier invoices using ML. Now, I have a lot of invoices I can use for test (+100k). I figure the first step is to label them, so I can train a model to extract the information I need.

Below is a sample invoice (I have anonymized it by removing the shipper, logo etc.).

As you can see in the above image, I am trying to extract three entities from the invoice:

table
amount (for each line in the table)
tariff (for each line in the table)

Now I actually don't need the table, but what seems to be common across all my supplier invoices is that the amount and tariff for line imtes, is always within a table.

My idea of a pipeline is:

Get all invoices (PDF files) and convert each page to a PNG image.
Label all the individual pages, using a label annotation tool, to label table, amount and tariff - including the bbox information for each label.
Train a model using the above dataset.
???

My questions are:

Does this seem like a good approach to such a problem?
Do I have to split up the model into two? With this I mean, does it make more sense to first identify the table within an image with one model, then serve the table image to another model, and extract the tariff and amount with the 2nd model?


texts: How to tune the hyper-parameters of an estimator in Orange Tool

texts:
Hi I want to tune/search hyper-parameters of SVM in Orange tool. How can I do?

I want to apply Exhaustive Grid Search to SVM

http://scikit-learn.org/stable/modules/grid_search.html


texts: Is there any possibility that cost function might end up in local minima rather than global minima while implementing the Neural network using Tensorflow ?


texts: How to deal with time features/seasonality in NYC taxi dataset

texts:
I'm using Catboost (and will also construct an LSTM) to predict pickup demand for Queens NYC taxi zones 1 step ahead using data spanning a few months (using resamples of an hour per taxi zone). I currently use the following features:
demand (target), taxi_zone (53 total), demand t-1 ... t-6, hour_sin, hour_cos, month_sin, month_cos, year_cos, year_sin, day_of_week, and some other exogenous vars. And I will be using taxi_zone/dow as a categorical variable. Is this approach correct, to transform time signals into cos and sin variables? Papers discussing the same topic/or e.g. energy demand don't mention time signals at all. And github repositories tackling this problem sometimes just send the raw time signals to the models, which seems to be very wrong. Should I use another method instead? What about detrending?
Also, the data after resampling for 1h is sometimes quite sparse (0 demand). Is it still correct to use standard regressor methods for Catboost/LSTM?


texts: Classification algorithm with multiple output for a set of features

texts:
I want to build a classification algorithm that will predict multiple values for a set of features. For instance, lets say I have a customer demographic data like Income, age, sex, city and I want to predict 5 top products they will buy: like if a customer is male, has income less than 30k, lives in Tier1 city and has age 20-35 years--then the output is: 1. Mobile 2. Car 3. Furniture i.e., there are three outputs which means they would most probably buy Mobile, Car and Furniture with decreasing value of probability.
Which algorithm will be best suited for this problem?


texts: Technology stack for linear regression on (not so) large dataset

texts:
While attending to the Coursera's Machine Learning Course, I figured out that I could use a database from the company I work for (~50MM records) to do some linear regression experiments.

But one of the steps involved on proposing this experiment, is to define the technology stack required for this task.

From my understanding the following tasks should be covered:

Read raw data and store it on an non-production database
Transform data to a "regression friendly" format
Store the transformed data on an intermediate database
Compute the actual regression

For #1 I can take some paths, like doing a custom .NET or Java program, or even use an ETL process (this is more to copy data to somewhere else and don't mess with production database).

On #2 the funny part begins: should I consider a specialized tool for a <100MM records database? If so, what would you suggest for transforming this data into a matrix-like representation?

I believe #3 is dependant on the #4: I see lots of samples (eg.: in R, or Matlab/Octave) based on text or csv files. Are these the standard formats for these computations? Or should I read from a database

For #4, from what I could understanding using R is the way to go, right?

Finally, should I consider a multi-gig multi-processors server, or considering it's an experiment in which spending some hours of computation is not a big issue, a 4GB machine will do the job?

I am aware that this question may be considered too broad, but I really would like to hear from you about what should I consider for it, and even if I am missing something (or going to a totally wrong path).

Regarding the data, you can consider it like the house pricing in Boston: it's a 30 features (columns) dataset, used to predict the value for one of these columns.

(question originally posted on Stack Overflow)


texts: How backpropagation through gradient descent represents the error after each forward pass

texts:
In Neural NEtwork Multilayer Perceptron, I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how many samples are chosen while training. That is, SGD iteratively chooses one sample to perform forward pass followed by backpropagation to adjust the weights, as oppose to GD where the backpropagation starts only after the entire samples have been calculated in the forward pass).

My question is: When the Gradient Descent (or even mini-batch Gradient Descent) is the chosen approach, how do we represent the error from a single forward pass? Assuming that my network has only a single output neuron, is the error represented by averaging all the individual errors from each sample or by summing all of them? To me this sounds like an implementation-dependent manner but I want to know if there is such a conventional way of that.


texts: How to create a df that gets sum of columns based on a groupby column?

texts:
country   year    gender    measure               value0 ... value12
 A         2000    1         vaccinated_at_month   2      ... 1
 B         2000    1         vaccinated_at_month   13     ... 12
 A         2000    0         vaccinated_at_month   4      ... 3
 A         2000    9         vaccinated_at_month   5      ... 4
 B         2000    0         walked_at_month       3      ... 13
 C         2001    1         vaccinated_at_month   6      ... 5
 C         2001    0         vaccinated_at_month   3      ... 2

I want to be able to remove the gender column and collapse all values into sums based on the previous categorical columns.

I have tried

df_new = df.groupby(['country', 'year', 'gender', 'measure'])['value0', ... 'value12'].apply(lambda x : x.astype(float).sum())

However, df_new still preserves the gender column. I am trying to get this outcome:

country   year       measure               value0      ... value12
 A         2000       vaccinated_at_month   11 (=2+4+5) ... 8 (=1+3+4)
 B         2000       vaccinated_at_month   13          ... 12
 B         2000       walked_at_month       3           ... 13
 C         2001       vaccinated_at_month   9 (=6+3)    ... 7 (=5+2)
 C         2001       vaccinated_at_month   3           ... 2


texts: Memory efficient structure for membership checking without false positive

texts:
The initial task can be described like this:
I have a requirement to deduplicate HUGE list(potentially billions of items) without storing the original items - it's simply unaffordable

All I need to know is answer to the question "Has my system ever seen this element before?"

The most close data structure I was able to find so far is a bloom filter, but it has false positives which better to avoid in my task as it results in data loss

For example providing I account to store at least 2^32 items, with positive error rate of just 1%(which means 1% of all urls won't be visited) I would need at least

n = 4,294,967,296, p = 0.01 (1 in 100) → m = 41,167,512,262 (4.79GB), k = 7

4.79GB of memory...

The task itself is a high scale web crawler, so I need to keep track of already visited urls(or sha1 hashes of this urls)

Any help is welcome

Thanks!


texts:
Why the A/B Testing, the Correlation and the Regression are considered methods of Statistical Analysis, while the method of "Association Rules" belong to the Data Mining field ?

Which are the difference between Statistical analysis and Data mining


texts: What mu and sigma vector really mean in VAE?

texts:
In standard autoencoder, we encode data to bottleneck, then decode with using initial input as output to compute loss. We do activate matrix multiplication all over the network and if we are good, initial input should be close to output. I completely understand this.

But within a Vae, I have few problems, especially in understanding latent space vector. I think I know how to create it, but I am not really sure what is its purpose. Here I will write a few ideas, please let me know if I am correct.

We first create mu and sigma matrices, which are just matrix multiplication of previously hidden layer and random weights.
To create Z (latent matrix), we use parameterization trick. mu+log(0.5*sigma)*epsilon, which is a random matrix with 0 mean and 1 std. I have seen this Z (latent matrix) always produces Gaussian distribution, no matter what is the distribution of mu and sigma vectors. And here is my question. Why, why do we want to feed forward latent matrix with Gaussian distribution?
When we come to the end of decoding. We compute loss function and we penalize the network with KL divergence of Mu and Sigma Matrix. I don't understand why is it important in the first place to compare a distribution of Mu and Sigma matrices? I assume, that when we do back-propagation, mu and sigma become more close in term of distribution, but my question is why is this important, why those two matrixes must be close to each other in term of distribution?

I would really appreciate very simplistic answers with simple examples if it is possible.


texts: What's Joint Training in Neural Networks?

texts: I'm having a hard time trying to find a good explanation of the process of Joint Training in Neural Networks. I already understand the concepts of Fine Tuning and Feature Extraction, and i know it has to do with the practice of taking a network model that has already been trained for a given task, and make it perform a second similar task.