texts: Grid search prediction results with best parameters on training set

texts:
i want to get prediction results on svm with best parameters but i didn't find way to get it. How to get prediction results on K fold?

from __future__ import print_function

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
from time import *
from sklearn import metrics
X=datascaled.iloc[:,0:13]
y=datascaled['num']

np.random.seed(1)
# Split the dataset in two equal parts
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0)

# Set the parameters by cross-validation
tuned_parameters =  [{'kernel': ['rbf'], 'gamma': [1e-2, 1e-3, 1e-4, 1e-5],
                     'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000]},
                    {'kernel': ['sigmoid'], 'gamma': [1e-2, 1e-3, 1e-4, 1e-5],
                     'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000] },{'kernel': ['linear'], 'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000]}]

print()

clf = GridSearchCV(SVC(), tuned_parameters, cv=10,
                       scoring='accuracy')
t0 = time()

svmclf=clf.fit(X_train, y_train)
t = time() - t0
print("Best parameters set found on development set:")
print()
print(clf.best_params_)
print()
print('Training accuracy')
print(clf.best_score_)


texts: clustering people according to answers on survey

texts:
Hi I am finding it hard to find online the best clustering algorithm for clustering people according to answers they gave on 20 question survey. There are four categories which each of these answers can fall into. I want to cluster the respondents according to their category answers, assuming it is multiple choice questions on the survey, with questions only having two options. If person A chose the same or similar answers to person B in a similar category then they should be clustered together.

What is the best algorithm? Sorry, I found it hard to do google searches for this question.


texts: Algorithms for aggregating duplicate identities based on non-numerical data?

texts:
I have a large dataset (2M entries) of people, but many people have multiple entries in the database with slightly (or significantly) different identifying information. For example, I may have J. Doe and John Doe, or I may have John Doe with an accompanying email address and John Doe without an accompanying email address.

I've been looking at different clustering algorithms but nothing seems well-suited to what I'm doing, which is to aggregate the entries based on rules like the following:

guess first and last names based on whether one of the names is written in all capitals
aggregate "J. Doe" and "J. Doe" if email addresses match
aggregate "J. Doe" into "John Doe" if no other people have first name starting with "J" and last name "Doe"

With a smaller dataset this would be a relatively straightforward task to do just with some simple rules, but with the number of entries I have the aggregation tasks can get really slow and the logic gets pretty convoluted. My current solution (based around using the fulltext search function within my database to find similar entries, adding hashes based on those results, and then aggregating based on a mix of hashes and types of ambiguity) works, but every time I try to run it or update it it just screams that it's the kind of problem somebody else has already solved. But I haven't been able to find a solution.

Are there algorithms that will do what I want based on rules like this? Or specific packages or software that might be helpful? Or am I approaching this problem completely wrong?

Thanks!

(Please note though that I am well aware that there are many different ways to incorrectly aggregate identities (e.g. that J. Doe could mean John Doe or James Doe), so I don't need warnings against trying to aggregate things.)


texts: Does anyone could help me to understand what is the autoencoders?

texts:
Does anyone could help me to understand what the autoencoders means?

What we expect is that the outputs are equal to the inputs, then why we need to do that? It doesn't make any sense to me.

I find some interpretation that it learned how to reconstruct the input data, does that mean that we could just pick some of the pixels from the origin picture and then reconstruct the whole original picture? If so, it still makes no sense to me, because the reconstructing part of the model is from the hidden layer to the output layer, we cannot just put the selected data into the hidden layer, cuz the input data of the hidden layer are combinations of the whole raw data from the input layer.

Thanks in advance.


texts: Dropout in other machine learning models

texts: Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any similar technique in linear regression, SVMs or tree-based methods?


texts: Suppose I have a dataset with some attributes, if the values of attributes is the same for two case but the label is different, how can topological data analysis can be helpful to carry out such kind of dataset.


texts: Custom DataFrame format for exporting to excel sheets

texts:
I have the following DataFrame and I want it exported in an excel file with different sheets having different words as shown in the image
0   2020-06-14  coronavirus es  -0.021277   coronavirus is  -0.006024
1   2020-06-13  coronavirus es  -0.021277   coronavirus is  0.000000
2   2020-06-12  coronavirus es  -0.021277   coronavirus is  0.000000
3   2020-06-11  coronavirus es  -0.011111   coronavirus is  0.000000
4   2020-06-14  COVID-19 es 0.000000    COVID-19 is 0.000000
5   2020-06-13  COVID-19 es 0.000000    COVID-19 is 0.000000
6   2020-06-12  COVID-19 es 0.000000    COVID-19 is 0.000000
7   2020-06-11  COVID-19 es 0.000000    COVID-19 is 0.000000

I want to be able to export it to different sheets in an excel file in the following format
with the 1st sheet being as follows and the next having (COVID-19 es, COVID-19) and so on..
Date        Coronavirus is  coronavirus es
6/11/2020   0              -0.5
6/12/2020   0.05            0
6/13/2020   0               -0.5
6/14/2020   -0.05           -0.66

so far I have tried using pivot wr.pivot_table(columns=['query_x', 'query_y']) but that does not give memy desired result and also trying to export it in different sheets is a bit confusing for me.
I have tried the following.
writer = pd.ExcelWriter(path, engine = 'xlsxwriter')
df1.to_excel(writer, sheet_name = 'b')
writer.save()
writer.close()


texts: Importance of Variance in Machine Learning

texts:
Selecting a column from a Dataframe, plotting its Histogram using matplotlib and then finding the variance is the steps that I have to take for this part of a project.

The final goal of the project is detecting anomalies in data.

I understand the part of plotting the data in form of a graph to understand it better.
On the other hand I’m not sure why finding the variance of this column is so important.

Would really appreciate if I get any information regarding this.

I know the definition of variance but just can’t seem to understand the usage and importance of it in this project.


texts:
I am trying to find all linear regimes in a certain plot, given certain data points.
If you are given two vectors of data $y$ and $x$ and $y$ is some possible non-differentiable function of $x$, how would you go about finding all the linear regions in $y$? Of course, after the data has been denoised and is sufficiently clean.
The one approach I am thinking about is taking the second derivative using finite difference methods. So essentially, evaluate as frequently as possible $\frac{d^2y}{dx^2} \approx \frac{y(n+1)-2y(n)+y(n-1)}{h^2}$ and see how close it is to $0$ upto a certain tolerance, taking into account the scale of the data. Yes, this method does not seem great, as the number of samples I have to take is rather large, and the larger the data vectors get, the longer it will take.
Could anyone suggest any papers, or alternative algorithms to determine how to find the linear regimes in a dataset?
I appreciate any advice you may have.


texts: Algorithm to find Unique users from their transactions

texts:
Scenario
An eCommerce site allow users to purchase items without creating an Account. However, it captures following attributes from the Purchases based on the type of payment and type of delivery:

Name
Address
Email Address
Card/Account Number

Note: It is possible one type of transaction only get one subset of attributes (e.g. Name and Address) and another type of transaction get another subset of attributes (e.g. Name and Card Number)
Expectation
To offer customized products or to prevent Fraud, it want to find individual users.
E.g. It should identify following three transactions are done by single user:

name: Alice, email: alice@wonder.land
email: alice@wonder.land, Card Number: 123456
name: Bob, Card Number: 123456 (Alice made purchase for Bob).

As you can see car-number/email is a stronger link than Name. So different attributes have different weights.
I have looked at couple classification algorithms (k-mean, k-mode..), but none of them is addressing this use case as I don't want to classify users but identify individual users.
Is this a standard data science problem and which algorithm will best fit here? Thank you!


texts: categorizing articles using deep learning

texts: Is it possible to write a program using deep learning that categorizes an article (news, entertainment,..., etc. ) into a positive category or negative category based on a specific theme?


texts: How to find the filename associated with a prediction in Keras?

texts:
My question is really simple, how to find the filename associated with a prediction in Keras? That is, if I have a set of 100 test samples named  and I get a numpy array which contains the estimated class probabilities, how do I map the filenames to the probabilities?

import cv2
import os
import glob

def load_test():
    X_test = []
    y_test = []
    os.chdir(testing_path)
    file_list = glob.glob('*.png')
    for test_image in file_list:
        img = cv2.imread(test_image,1)
        X_test.append(img)
        y_test.append(1)
   return X_test,y_test

if __name__ == '__main__':
   X_test = np.array(X_test, dtype = np.uint8)
   X_test = X_test.reshape(X_test.shape[0],3,100,100)
   X_test = X_test.astype('float32')
   X_test /= 255


texts:
I'm trying to create a new column called Class which is a calculated column.
Basically when the Math class is NA then the class will be the CS class, otherwise it will be vice versa.

How can I do this in python? Table name is Classes

MathClass |  CSClass |  Class
------------------------------
Trig      |  NA      |  Trig
NA        |  Java    |  Java
NA        |  Python  |  Python


texts: How to do backward features elimination when considering interactions between them

texts:
I have a multi linear regression problem,

$Y$ is my target and $X_1, X_2, X_3$ are my features.

In my regression, I consider the interaction between $X_1, X_2, X_3$ and I add a bias.

So my problem is given by :
$Y  \sim X_1 + X_2 + X_3 + X_1X_2 + X_1X_3+ X_2X_3+ bias$

Now, I fit my model with statsmodels.api.sm and I want to eliminate the feature the highest p value recursively.

My first question is : for example, if the highest p value is for the $X_1X_2$ feature, is it okay to eliminate this feature even when $X_1$ and $X_2$ can be statistically significant ?
My second question : in the case when all the interaction of some feature have a p value greater than 0.05 in the first iteration, Could I eliminate this feature and all the interactions ?

Thank you for your help


texts: Mismatch between optimum features and grid scores using RFECV?

texts:
I have a dataset with 5K columns focused on binary classification. I have more than 60 columns. I am trying to find the best features through RFECV approach. Though it produces 30 optimum features, when I plot in graph, I only see 12 features. Please see my code and plot below

model  = RandomForestClassifier(n_estimators=100, random_state=0)
model_b = LinearSVC(class_weight='balanced',max_iter=1000)
# create the RFE model and select 15 attributes
rfe = RFECV(model,step=5,cv=5)
rfe = rfe.fit(X_train_std, y_train)
# summarize the selection of the attributes
feat = rfe.support_
fret = rfe.ranking_
features = X.columns
print(rfe.n_features_) # this returns 30 as output

print(rfe.grid_scores_) this produces the below output

I was expecting to see the grid scores for 30 features and in the plot, I was expecting to see the x-axis to have 30 features as well. But it shows only for 12 features. Similarly if I had only 19 features in my dataset, RFECV returns all 19 as optimal feature which is fine. But again in grid score it only shows 4

q1) Does this mean beyond 12 features, there is no increase in model accuracy?

q2) I assume grid_scores are nothing but weightage/rank which indicates the influence that a feature has on outcome. But how do I get the name of this 12 features?

q3) Wwhy does it show optimum no of features as 30 but grid scores is shown only for 12.

Can you help me with these question please?


texts: The larger an encoding dimension in NLP the better?

texts: Is the longer encoding array (word2vec or any other kind) always more precise than the shorter (not regarding the computational cost, only the performance)?


texts: time series forecasting - sliding window method

texts:
I have been trying to understand this sliding window technique but to no avail and really unsure as to how I would implement it.

My dataset: I have hourly values for the electric load for a year (over 8700 data points) - image below. I'm going to split the dataset into a training set  (1 jan to 30 sept) and a test set (1 oct to 31 Dec).

Im going to use supervised learning techniques such as Regression trees and random forests (basically anything that is available to me in scikit learn), train them on the training set then make predictions on test set.

I understand that I need to use historical known values  as input features to input into a model. As a result i created "Load_lagN".

Am I correct in saying that because I have created 10 lagged variables (Xt-1 to Xt-10) , this is the equivalent of using a sliding window of size 10?

How would I then  just simply train the model on the training set and make predictions on the test set without using the sliding model?

With the sliding window model, does that assume only the past n values (10 in my case) are relevant?

If I have split my dataset into a training and test set, how does the regression tree become trained and then make predictions in the test set (I'll stick to one step ahead forecasting for now) whilst implementing this sliding window technique?

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html#sklearn.model_selection.TimeSeriesSplit
Ive come across this method - TimeSeriesSplit - is this what I need to use for the sliding window technique or is it only a cross validator?

My understanding of the sliding window method:
As  you can see in the image below, I use values of 10:00 to 19:00 to predict the value at 20:00, move the window so that this new value is now included, then predict the value for 21:00. This keeps happening until I have exhausted the training set. I then make predictions.

What are your thoughts?


texts: How to interpret my logistic regression result?

texts:
I'm having a hard time to interpret my result of the logistic regression.
I have a few question.
Firstly, how can I check if a feature is more important to the others, like that there is a real significane by it.
I have an accuray of 0.58 which is pretty bad. Anyway, why does the RFECV feature selector of sklearn tell me to use feature1 to get the best result but by training my model where I use hte statsmodels library, the model will all 5 variables is slightly better?
Also, why does my two models when I train it with sklearn and statsmodels is different in result?

It really confuses me everything. It would be nice if someone could tell me an easy way to interpret my results and do it in one library all.
My codesnippets
import statsmodels.api as sm
X = df_n_4[cols]
y = df_n_4['Survival']

# use train/test split with different random_state values
# we can change the random_state values that changes the accuracy scores
# the scores change a lot, this is why testing scores is a high-variance estimate
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)
logit_model = sm.Logit(y_train, X_train).fit()
y_pred = logit_model.predict(X_test)
cf_matrix = confusion_matrix(y_test, y_pred.round())
sns.heatmap(cf_matrix, annot=True)
plt.title('Accuracy:{}'.format(accuracy_score(y_test, y_pred.round())))
plt.ylabel('Actual Szenario');
plt.xlabel('Predicted Szenario');
plt.show()

Part 2
X = df_n_4[cols]
y = df_n_4['Srvival']

# use train/test split with different random_state values
# we can change the random_state values that changes the accuracy scores
# the scores change a lot, this is why testing scores is a high-variance estimate
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)
print(len(y_train)," Testdata")

# check classification scores of logistic regression
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)
print('Train/Test split results:')
print(logreg.__class__.__name__+" accuracy is %2.3f" % accuracy_score(y_test, y_pred))
plt.title('Accuracy Score: {0}, Variablen: feature1'.format(round(accuracy_score(y_test, y_pred),2), size = 15))
cf_matrix = confusion_matrix(y_test, y_pred)
sns.heatmap(cf_matrix, annot=True)
plt.ylabel('Actual Szenario');
plt.xlabel('Predicted Szenario');
plt.show()

I don't have too much statistical background so please take it easy on me! I will provide any further information you need. Thanks so much I'm stuck since a week, as all I read and further do, confuses me even more.


texts: Classify samples based on other sample probabilities

texts:
I was wondering if there's a way to train a classifier or set up a way of classifying after that can classify certain samples as some relationship between the previous two.

I notice that, for example, when I use the predict_proba from scikit-learn's RandomForest, I can see the probability that a class was predicted like this: [0.3, 0.43, 0.27].  I want to do something like: if | p(class1) - p(class2) | < ε then class 3 should be boosted. Maybe this means using a binary classifier at first and then checking the relationship between the binary classes.

To describe the situation further, class 1 and class 2 are distinct and class 3 has a mixture of both features.


texts: Why Generator error increases after some epoch

texts:
I'm training Realtivistic GAN on dog images. I don't understand why my generator loss increased after 100 epoch.

This is the generator loss graph.


texts:
I am currently experimenting with the idea of including the class of a feature vector as a separate feature. My work is about preposition selection in learner language use. I want to train a classifier that, given a certain position of a sentence, uses relevant context features around that position to build vectors. The classifier detects if there should be a preposition or if it should remain empty. Some features include:

Neighbouring N-grams, where N={2,3}, at Part-Of-Speech and token-level
Neighbouring nouns and verbs

The class expresses a binary distribution: 1 for preposition presence and 0 for preposition absence.

Consider that if a language learner opted to select a preposition in a given sentence, but the classifier thinks there should NOT be one, it can be interpreted as the classifier marking a language error: The learner inserted a preposition in a prohibited context. Conversely, if a learner did NOT use a preposition at a given position in a sentence, but the classifier does expect one, it is also an error.

The model is trained on correct negative (no preposition) and positive ("yes" preposition) events (context features around a certain position: a whitespace or a preposition). This means that the assigned labels are the gold standard, similarly for the test data, because they are trained on *native *language use which is assumed to be error-free. The same assumption cannot be made when running the classifier on learner's language - after all, learners make mistakes, which is what I am trying to detect.

I want to include the decision (yes/no) of the learner into the classifier rather than just as the assigned label. Because in practice, the chance on either error is very small, so the classifier should be very confident before marking it as such. But when I do this, the classifier always selects the value of that decision feature as the class. After all, it is trained so that there is a 100% correlation between said feature and the assigned class.

How do I turn this learner's decision - which is NOT an absolute indication of the actual answer - into an indication instead?

I'm experimenting with Linear Support Vector Classifier and Logistic/Linear Regression.

Thanks a bunch, if more details are needed, please ask.


texts:
I'm working on a project about predicting kickstarter project success(classification) and my dataset has many columns that could be used as features such as :
state_changed_at,
launched_at,
created_at.

Now the dataset has these features on unix timestamps.

Do I need to convert dates to some other format?

Can date data be used as a feature ?

If so how do I handle them ?

Do I keep them as a unix timestamp and try to scale/normalize them ?


texts: Definition of the Q* function in reinforcement learning

texts:
I'm making my way through Sutton's Introduction to Reinforcement Learning. He gives the definition of the $q_*$ function as follows
$$
q_*(a) = \mathbf{E}[R_t | A_t = a]
$$
where $A_t$ is the action taken at time t and $R_t$ is the reward associated with taking $A_t$. From my understanding, $q_*$ represents the true value of taking action $a$, which is the mean reward when $a$ is selected.
But I'm confused about why $t$ is included in this equation at all. Should $q_*(a)$ really be $q_*(a, t)$? Or are we to understand $q_*$ as taking the expected reward across all $t$?


texts: Factor Analysis with Mixed Data gives many components

texts: I performed Factor Analysis with Mixed Data using PCAmixdata package from R. My dataset consists of 115000 records with 40 features of both categorical and continuous data. I checked the eigenvalues of the resultant 760 components out of which 260 has eigenvalues above 1.1 and the proportions of each of the component less than 1 except for the first component. So I am not sure if it is ideal to proceed with FAMD given the huge number of resultant components.


texts: Synthetic data generation for Churn prediction

texts:
I am trying to predict which customers will churn in next three months for a Telecom service provider, and I don't have any data!
I have to generate synthetic data and train some models.
One thing is , I can't use any open source dataset, because they are mostly denormalized and my requirement is a bit different. I have to start from individual tables of customer accounts and subscription details and join everything to create the final model input.
I have two questions:

How to generate some credible data for such Churn problem?
Most of the churn dataset consist of sone features and a label (0/1). But my doubt is, does it really kake sense? Because churning is not a point of time incident. It happens over the time. So I think the dataset should consist of features engineered at different snapshots of time and whether the corresponding account churned when this feature was observed. So in this way, each customer will have multiple entries in the final dataset. Suppose I was to predict churn for next 3 months. So for Jan-Mar I will have some features and label will the the churn status from June. Is it a suitable way to Model the problem? Am I thinking in correct direction?


texts: Neural Networks : Can I use both sigmoid and tanh as activation functions?

texts: In a neural network architecture  can I use the sigmoid function in some layers and the tanh function in the others? Is it a good choice?


texts: How do I change the location of Google BigQuery

texts:
So, I'm trying to use Google BigQuery for the first time for a project of mine, and I'm a bit confused. The documentation isn't helping much, and it looks like all the Google employees are gone thanks to the current epidemic, judging by the blogpost by Google.

I've got several .csv files containing tables of unlabeled numerical data uploaded to Google Cloud storage, in their Australian servers. What I want to do is to use BigQuery to perform k-means clustering and analysis of correlations between variables on them. However, I'm having issues trying to get the data transferred into BigQuery.

I tried using the "Data Labeling" tab to create a data set like their documentation suggests, but it only reads one CSV at a time (I want to combine all the CSVs into one dataset), and none of the data types seem to fit at first glance, with the only one that sort of matches being "text". However, once it finishes its process, it looks like the resulting dataset is empty, when the original CSVs have hundreds of rows and dozens of columns. I've also tried using the "Tables" tab to create a data set, but it is giving me an error message about needing to put my models in the same region as my data, and I can't see how to fix this; the only options it seems to be giving me seem to be "Global" and "European Union", so I'm not certain how to set it to "Australia".

Does anyone have any ideas about how to fix this?

An anonymized version of the data sort of looks like this (but the actual CSV is much larger, with many more rows and columns):

  Item_ID,From_Position,To_Position,Data_One,Data_Two,Data_Three
  AA22DDA002,103,104.0,182.35643356,8.12345678,-2.32453256,0.03234213
  AA22DDA002,104.0,105.0,483.23424678,3.34627319,-4.37600000


texts: predicting loan default based on transaction history

texts:
i have a training dataset with 25,000 different customers, each with a transaction history of 50-500 bank transactions (both deposits and withdrawals, the exact # varies across customers). each customer was given a loan (from 500-2000$), and they either defaulted or did not default on the loan (this information is given in the training set). the test set has around 15,000 different customers (not the same people as above), and i need to predict whether or not they will default based on their transaction history. there is also a string associated to each transaction giving the type of transaction (ie, were they buying beer, coffee, etc).

i'm wondering what is the best way to approach this problem from a machine learning perspective? ie, what model would be best to predict the probability of a customer defaulting on a loan based on their transaction history?

i'm thinking a neural network in keras would be the easiest approach, but i'm not sure how to train it due to the variable input size (different # of transactions per customer). also, i'm wondering if i should make use of the transaction type string, because that adds an entirely new element of difficulty (parsing the string and classifying it into high/low risk type transactions, etc)...unless there are libraries that can do this automatically..

thanks


texts: How to load label data presented in raster format into Keras/Tensorflow

texts:
I would like to use a CNN network to segment 2 objects (binary: "0: object not present 1: object present") into shapes, but I have an issue with data. The training data is 150 images and in "jpg" format and the ground truth (label data) is also 150 images of "png" rasters of 0 and 1
(resulting in black white images...)
Now, the question is how to load this hybrid of train images and label images in keras/tensorflow and if there`s a dummy example and/or demonstration on how to do that in python, I would be grateful.


texts: Is there any package in python that can identify similarity between alphanumeric alias names of a parameter?

texts:
For example: for a parameter like input voltage,
     Alias names : V_INPUT, VIN etc.

Now, I want the software to recognize each of the alias names as same. Is there any package/method by which I can achieve this?
Nltk is only allowing for dictionary words.


texts: How are the Convolution kernels learned?

texts:
As I went through the basics of machine learning, I failed to understand how do the Convolutional layers in a CNN learn the convolution kernels. After looking at first few tutorials, I thought the kernels were fixed: for example, there was a 2D kernel which extracts vertical lines, another one extracted horizontal lines etc. Later on I realized that kernels are learned by the network and not fixed.

Is there a good explanation on how the learning goes, how does the backpropagation work (compared to fully connected layers), what is the optimization function, etc.?

p.s.
Let this be an example function to explain: https://keras.io/layers/convolutional/.


texts: Why CNN doesn't give higher accuracy over simple MLP network? [From Keras examples]

texts:
I'm still new to machine learning and just came across powerful deep learning library, Keras.

I've read Keras document and tried few Keras examples on Github here. I've also studied some basic knowledge and concepts of deep learning from several sources but still haven't really had solid understanding in CNN and RNN which look to be very powerful networks.

So, to prove my assumption, I downloaded reuters_mlp.py example from Keras Github which originally uses simple MLP networks as a model. I combined the idea of CNN which I got from imdb_cnn.py example to reuters_mlp.py example and then observed the result.

Surprisingly, the result didn't come out like I expected. CNN performed worst than simple MLP networks. Can someone please explain why the accuracy of CNN is lower than the simple MLP networks?

Here are the outputs (Tensorflow as backend)

8982 train sequences, 2246 test sequences, 46 classes, num_words=1000

MLP (sequences_to_matrix, mode=bianry):

Epoch 1/5
8982/8982 [==============================] - 3s - loss: 1.3236 - acc: 0.6984
Epoch 2/5
8982/8982 [==============================] - 2s - loss: 0.7182 - acc: 0.8250
Epoch 3/5
8982/8982 [==============================] - 2s - loss: 0.4544 - acc: 0.8864
Epoch 4/5
8982/8982 [==============================] - 2s - loss: 0.3197 - acc: 0.9192
Epoch 5/5
8982/8982 [==============================] - 2s - loss: 0.2511 - acc: 0.9356
1920/2246 [========================>.....] - ETA: 0s

Test loss: 1.05213204963 Test accuracy: 0.785396260071

CNN (pad_sequences):

Epoch 1/5
8982/8982 [==============================] - 81s - loss: 1.9794 - acc: 0.5181
Epoch 2/5
8982/8982 [==============================] - 78s - loss: 1.4289 - acc: 0.6591
Epoch 3/5
8982/8982 [==============================] - 79s - loss: 1.1546 - acc: 0.7175
Epoch 4/5
8982/8982 [==============================] - 78s - loss: 0.9639 - acc: 0.7663
Epoch 5/5
8982/8982 [==============================] - 77s - loss: 0.8378 - acc: 0.7935
2240/2246 [============================>.] - ETA: 0s

Test loss: 0.960687935512, Test accuracy: 0.764470169243


texts: How do I predict a class for each time step using the information from previous timsteps

texts:
I have a classification problem but  different than usual. I have to provide 3 outputs (each of them either 0 or 1) for every input of 3 timesteps and 10 features.
What model architecture or approach is used in this type of problem?

Do I predict for each time step by separating the observations?
Do I predict for each time step and for next time step also use the previous time step?
Do I set output units to be 3 and train model to predict three values for 3 timesteps?


texts: Support vector machines in R: Finding the equation of a hyper plane (in 6 dimensions) and showing it's correct

texts:
lin.6d<-read.csv("D:/Documents/Linear-6d.csv", header=TRUE)

View(lin.6d)

plot(lin.6d,col = ifelse(lin.6d$Class == 1, "red","blue"))

library(e1071) # Package for Support Vector Machines

svm.lin.6d <- svm(Class ~ ., data = lin.6d, type = "C-classification", kernel = "linear", cost = 10, scale = FALSE)
print(svm.lin.6d)

tuned <- tune(svm, Class ~., data = lin.6d, kernel = "linear", ranges = list(cost=c(0.001,0.01,.1,1,10,100))) # Will show the optimal cost parameter
summary(tuned) # Optimal cost parameter is 10

# Find parameters of hyperplane (wx+c=0)
w <- t(svm.lin.6d$coefs) %*% svm.lin.6d$SV # In essence this finds the hyper plane that separates our points
c <- -svm.lin.6d$rho

The above code takes the data from an excel file "lin.6d" which contains vector points "X1,X2,X3,X4,X5,X6" and "Class:-1,1". So each vector point has a class 1 or -1.
I have plotted the data and colour coded each point depending on it's class (Blue if the point if it's class is -1, red otherwise).
I then used the svm function to obtain the coefficients of the hyperplane where:
w is the set of coefficients (W1,W2,...,W6) and c is the intercept.
hence I got an answer:
w=(0.02433109, 0.02803587, -0.02332039, -0.000118514, -0.009224677, -0.02626795)
c= 0.1712391
where wx+c=0 is the equation of the hyperplane separating the classes of the points.
My question is:
How do I know if this is the correct answer? How do I create a visual plot to show this can be represented as a line separating the classes?


texts: I need help outlining the workflow for predicting winner in book prize

texts:
I would like to predict the winning book in a yearly book prize that has been going on since the 1950s. As to now, I have data about candidates and winning books (title, editor, text, etc.). The workflow I thought of is to build a dataframe with: book title, book author (name), gender of the author, winner (yes/no), text of the book. Such a text would of course be vectorized. Then use a classifier with the winner as a target category.
Does this make sense? Are there potential mistakes I did not think of? There is no related research on the internet.


texts: Evaluating machine learning model with missing features

texts:
I am working on a credit risk binary classification problem. The classes are GoodPayers and BadPayers.  The training set has variables/features that contains:

DemoGraphics Data  such as - Age, Education, Loan Amount, Interest Rate
Behavioral Data such as - Payment in Month1, Payment in Month2, Payment in Month3, Payment delay in Month1, Payment delay in Month2.

The 10-fold cross-validation has 0.82 AUC on this set.

However, the unseen data just contains the 'Demographics Data' and does not have 'Behavioral Data' of Payment.  How do we deploy/test the model based on DemoGraphics Dataset only?


texts: Trying to understand encoder-decoder sequential models in Keras?

texts:
My understanding is that for some types of seq2seq models, you train an encoder and a decoder, and then you set aside the encoder and use only the decoder for the prediction step. For example this seq2seq time series prediction model from Uber:

Now I am trying to implement a to version of this in Keras.

This is the Keras code for a vanilla LSTM:

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=200, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

This is the Keras code for a stacked LSTM model:

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=200, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

And this is the Keras code for an encoder-decoder model:

# define model
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(RepeatVector(n_steps_out))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=100, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)

The problem is, I don't see much difference between the encoder-decoder code and the vanilla and stacked LSTM. In particular, I don't see how we are using only the decoder in the predict step, and which variable or method in Keras corresponds to the embedding that we would be using as an input for predicting new time series?

How can I implement the code for a model similar to the one in the illustration using Keras?


texts: Neural network (multi-layer perceptron) in R with count data as response

texts:
The goal is to run Poisson regression for neural networks (multi-layer perceptron) in R.
I am currently using the neuralnet package in R.
I have read Is there an R package which uses neural networks to explicitly model count data? and Nonlinear Poisson regression using neural networks: a simulation study. I have figured out that I should change the error function to err.fct = function(x, y) { -(-x+y*log(x))} (for Poisson).
Code:
fit <- neuralnet(nclaims ~ age_ph + gender + tar_region + weight + age_car + ptw,  data = df,  hidden = 1,  err.fct = function(x, y) { -(-x+y*log(x))},  act.fct = "logistic").
I know that act.fct = "logistic" specifies the sigmoid function in the hidden layer, but how can I change to the exponential function in the output layer?


texts: How to split a large set of coordinates into smaller sets using something like nearby selection?

texts:
I have a dataset filled with 3817 coordinates (latitude, longitude). What I want to do is create groups of ~500 coordinates so that smaller group can be used to solve Vehicle Routing Problem. The reason I want to do this is because the original dataset is large the solving the VRP takes too long.

I've tried splitting the map into grids and simply grouping the coordinates based on which grid they are in. But this is not the most efficient method. I've read one of OptaPlanner's blog posts where they write about Nearby Selection and would like to do this with my dataset.

Any ideas on how to accomplish this?


texts:
I'm trying to run a deep model using GPU and seems Keras running the validation against the whole validation data set in one batch instead of validating in many batches and that's causing out of memory problem
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[160000,64,64,1] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:GatherV2]
I did not have this problem when I was running on CPU, it's just happening when I'm running on GPU, my fit code looks like this
history =model.fit(patches_imgs_train,patches_masks_train, batch_size=8, epochs=10,
  shuffle=True, verbose=1, validation_split=0.2)

When I delete the validation parameter from the fit method the code works, but I need the validation.


texts: Are there real world applications where deep fully connected networks are better suited than ConvNets

texts:
I would like to give some brief background for my question to avoid answers that explain the difference between fully connected nets and ConvNets.

I completed the first 3 courses in the deep learning specialization from deeplearning.ai (Coursera).  Simultaneously I've worked through the first two home work assignments from Stanford's CS321n course for the Spring of 2017, as well as Michael Nielson's book on deep learning.

I believe I have a reasonable understanding of the architecture of both types of networks, and how to write them in python/tensoflow. I also implemented my own very simple fully connected feedfoward net in C where I was able to get 98% on MNIST.

I've also done some reading about this question in terms of which type of data is better suited for a ConvNet, and the answer I've come up with is similar to:

  If the order of the input features matters to their meaning, then a
  ConvNet will work best, but if you replace the position of one column
  with another and the meaning is still intact, then a ConvNet will not
  work.

This means that spreadsheet data (if I've understood correctly) is not suited for a ConvNet.

Unfortunately I don't know any deep learning practitioners to ask this question: are there fully connected deep neural networks in the wild that are currently in use?  The best example I can think of is financial data.  If that's true, what are some others?


texts: Loss Nan: How can I properly implement a LSTM Time-Series model with a lot of parameters?

texts:
The Problem:
I am very new to TF and Keras. I am attempting to train a time-series LSTM. When using only a few parameters as a test, the model seems to work fine. Once I increase the parameters to the full 17010, I almost instantly receive a loss: nan.
Network Specs:
My inputs have been normalized into a range between 0 and 1 and almost all look fairly clean with no significant outliers. When graphing each of the parameters, they look similar to this:

Some specs on my network:

Training examples (Time steps): 9835
Input parameters: 17010
Input sequence length: 30
Output parameters: 1
Output sequence length: 1

In summary, the entire sequence of inputs coming from the Timeseries generator end up with shape (30, 17010, 9835) I have tried batch sizes between 32 and 256.
Note: My inputs have a lot of missing values. Since my model is attempting to predict the increase or decrease as percent change from the previous time step (1 day). ie. y[i] > 0.5 --> increase and y[i] < 0.5 --> decrease. To fill these missing values I have assigned them a value of 0.5
I Have Tried:

Decreasing dropout rate
Clipping gradients (don't think this is necessary for me since everything is between 0 and 1)
Checked all inputs for NaN or Inf values
Increasing batch size
Decreasing my learning rate
Reducing number of input parameters to 3406

I have found conflicting information on the number of layers and units that are appropriate but I have attempted to play with these values a little bit. I have tried between 32 and 2000 hidden units.
My Code:
I am using a keras.sequence.TimeseriesGenerator for both training and validation like this:
dataset_train = keras.preprocessing.sequence.TimeseriesGenerator(
    x_train,
    y_train,
    length=past,
    batch_size=batch_size,
)

model = Sequential()
model.add(LSTM(hidden_units, return_sequences=False, activation='relu', input_shape=(past, dataset_train.data.shape[1])))
model.add(Dropout(0.2))
model.add(Dense(hidden_units))
model.compile(optimizer=keras.optimizers.Adam(learning_rate=learning_rate), loss="mse")

path_checkpoint = "model_checkpoint.h5"
es_callback = keras.callbacks.EarlyStopping(monitor="val_loss", min_delta=0, patience=5)
model.summary()
modelckpt_callback = keras.callbacks.ModelCheckpoint(
    monitor="val_loss",
    filepath=path_checkpoint,
    verbose=1,
    save_weights_only=True,
    save_best_only=True,
)

history = model.fit(
    dataset_train,
    epochs=epochs,
    validation_data=dataset_val,
    callbacks=[es_callback, modelckpt_callback, PlotLossesKeras()],
)

I'm not sure if this many parameters are near impossible to train with since I have a smaller number of examples or if something else is causing this. I appreciate any advice you can give me.


texts: Dropout backpropagation implementation details

texts:
Just to summarize
Understanding dropout and gradient descent
and
https://stats.stackexchange.com/questions/207481/dropout-backpropagation-implementation

Suppose I need to implement inverted dropout in my CNN. All the neuron outputs in dropout layer during feedforward phase are multiplied by mask/p, where mask is 0 or 1, p is retain rate. But should I apply the same operation (include division by p) at the backpropagation phase? I suppose positive answer (see the second link above), but I need to know for sure.


texts:
I am new to reinforcement learning and experimenting with training of RL agents.

I have a doubt about reward formulation, from a given state if a agent takes a good action i give a positive reward, and if the action is bad, i give a negative reward. So if i give the agent very high positive rewards when it takes a good action, like 100 times positive value as compared to negative rewards, will it help agent during the training?

Intuitively I feel, it will help the agent training, but will there be any drawbacks of such skewed reward structure?


texts: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() using panda python

texts:
enter preformatted text hereHere I have data to import from a CSV file.I wrote an equation inside the class and to solve the equation data will import from the CSV file.
When I run my code I got an error like "tuple indices must be integers or slices, not str" using panda python.
Can anyone help me to solve this problem?
I upload my code and CSV file here.

def ph_convert(time,we,h,a,w):
while time <= 30:
    level = 1.1
    level = float(level)
    if w == 1:
        ph= ((((6*we)+(1*h))/level -(4*a)))/time
    else:
        ph= ((6+((1*we)+(3*h))/level -(6 *a)))/time
    break
while time <=60:
    level = 1.25
    level = float(level)
    if w == 1:
        ph= ((((6*we)+(1*h))/level -(4*a)))/time
    else:
        ph= ((6+((1*we)+(3*h))/level -(6 *a)))/time
    break
print(ph)

data = pd.read_csv('data1.csv')
data['time'] = data['time'].apply(time_convert)
we = data['we'].astype(float)
h = data['h'].astype(float)
a = data['a'].astype(float)
w = data['w'].astype(float)
time = data['time'].astype(float)
print(ph_convert(time,we,h,a,w))

Subset of my CSV file:

we  h   a   w   time
48.1    150 53  1   6:15:00
48.1    150 53  1   9:00:00
48.1    150 53  1   9:25:00
48.1    150 53  1   9:30:00
48.1    150 53  1   11:00:00

error:

[![enter image description here][2]][2]


texts:
I have questions regarding this Normalized weights and Initial inputs video on Udacity course Deep Learning

In this video the lecturer talks about variables that go into Big-loss function should have zero mean and equal variances. I can’t figure about which variables the lecturer is talking about. Are they weights and biases or are they soft-max function and labels in the Big-loss function? And how  would zero mean and equal variance help in optimization?

The other question I have is, in the video the lecturer talks about weight initialization randomly using Gaussian distribution, I cannot understand how can we initialize weights using Gaussian distribution with zero mean and standard deviation sigma? And where optimizer will move the point (initialized weight), up or down to find the local minima?

So can you recommend me a book that I should study before taking this course, so that problems like these don't occur?


texts: Which parameters are hyper parameters in a linear regression?

texts: Can the number of features used in a linear regression be regarded as a hyperparameter? Perhaps the choice of features?


texts: Changing a list of tensors to a tensor of lists!

texts:
I have a list of tensors created by
[some_function(x) for x in something]

and this list becomes like this
[<tf.Tensor: shape=(), dtype=string, numpy=b'I have gone and you will go'>, <tf.Tensor: shape=(), dtype=string, numpy=b'we will go'>]

But I want to have an object like this
<tf.Tensor: shape=(3,), dtype=string, numpy=array([b'I have gone and you will go', b'we will go'], dtype=object)>

Which is a tensor of list of strings (Opposite to the previous one). Is there any way to this, assuming that I have access to all these tensors, i.e. some_function(x) for x in something separately.


texts: Tool for visualizing long time series data

texts:
I am searching for a software / tool / R package (any of these) that enables easy visualization of a time series.

My problem is the following:

I have a multiple time series with 7 attributes
I want to have all 7 attributes plotted beneath each other with a own y-axis
I want to be able to set y-axis resolutions for the plot
The time series is very long (1 year) but I want to look at each day individually and then basically scroll right to see the next day


texts: How could have a loss/cost/error function that cannot be written as an average?

texts:
Check this phrase in this online book:
https://hyp.is/GsHrloyvEeaYQ093X053Wg/neuralnetworksanddeeplearning.com/chap2.html

It refers that the quadratic cost function (RSS) can be written as an average for each individual training example.

so if the cost of a single training example is Cx = (1/2) * ||y-a||^2

you can have C = (1/n) * Σ(Cx)

Could you give an example of an error / cost function that cannot be written as an average?


texts:
I'm going to describe the procedure for gradient descent using the language of English:

1) For all of the files (e.g. documents) in our training corpus.

--> commence, now we are considering our first document

2) For all the features in the feature vector associated with that training example (document)

3) Take the dot product of the vector of weight values * and the vector of features specific to the document under consideration

* vector of weight values is applicable to all features, though all features will not be present in all documents, indeed only some features will be present in each individual document

4) Take the value resulting from step 3 and apply this to our loss function. I'm using the Hinge Loss function, i.e. the formula

Is that equation saying that the hinge loss objective function should be the sum of the misclassification error for every file in our training set?

So is that equivalent to the java code:

Math.max(0, 1 - y * value_from_step_three)

With y = {-1, 1}

^ THIS IS MY FIRST POINT OF CONFUSION

I guess that is inaccurate but I'm not sure, according to my mental model it should be right.

5) For every weight, update it as:

new weight valuej = old weight valuej - learning rate * value from step 4 * value of featurej.

I'm a bit confused about what value of featurej should be, since I guess this would correspond to a vector, that has values for every weight under consideration, i.e.

Example:

Document 1 = ["I", "am", "awesome"]
Document 2 = ["I", "am", "great", "great"]

Dictionary is:

["I", "am", "awesome", "great"]

So the documents as a vector would look like:

Document 1 = [1, 1, 1, 0]
Document 2 = [1, 1, 0, 2]

There will be weights:

[Weight"I", Weight"am", Weight"awesome", Weight"great"]


texts: How does KNN work if there are duplicates?

texts:
I am currently debating with my friend about how KNN handles duplicates. Suppose K = 2, and we have a 1-dimensional set of data points to illustrate my dilemma

I = {1, 2, 2, 2, 2, 2, 6}

Thus is it correct to say that the K=2 nearest neighbours of data point 1 is simply {2, 2}? Also, same reasoning if we did the 2 nearest neighbours of data point 2 it would be {2, 2} as well not including itself?


texts: Regression - predict n numbers based on another n numbers

texts:
I'd like to get a recommendation how to attack a problem of predicting multiple numbers.
Training data contains 4 columns, each says a probability of record being in the bucket. So, for example:
X = [0.25, 0.5, 0.25, 0.0]
and
corresponding output should be e.g. Y = [0.8, 0.1, 0.0, 0.1].
Each row should sum to 1.0

What type of approach would you recommend?

I already tried simple neural network with 4 neurons in the last layer and softmax activation but wondering if there is a better solution.

Thanks!


texts: Using non-negative matrix factorization (NMF) for facial recognition

texts:
I'm interested to find out how to implement NMF for facial recognition. I understand that the NMF works by taking V, which is a matrix of face images (n resolution x m persons), and factorize V = WH, where we get r basis vectors. I presume that this step is done using the training set. But I don't know how to proceed from here.

With PCA, after building the eigenfaces from the training set, we project the test image onto the face-space, then classify the image by comparing the weight coefficients between the test image and training images. But what about NMF? Do I use the basis vectors? What do I do with them?


texts:
i am trying to train a deep neural network to figure out that if there is a 1 and 0 present in the first two columns of X that the output is 1 otherwise its 0. but im only getting a 75% accuracy on the model!

import numpy as np
import tflearn

X = [[0, 0, 1],
    [0, 1, 1],
    [1, 0, 1],
    [1, 1, 1]]

Y = [[0, 1],
    [1, 1],
    [1, 0],
    [0, 1]]

Xtest = np.array([[1, 1, 1],
                  [0, 1, 1],
                  [1, 0, 1],
                  [0, 1, 1]])

# Build neural network
net = tflearn.input_data(shape=[None, 3])
net = tflearn.fully_connected(net, 32, activation='sigmoid')
net = tflearn.fully_connected(net, 32, activation='sigmoid')
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net, optimizer='adam')

# Define model
model = tflearn.DNN(net)
# Start training (apply gradient descent algorithm)
model.fit(X, Y, n_epoch=5000, batch_size=16, show_metric=True)

pred = model.predict(Xtest)
for i in range(4):
    print(pred[i][0])

The output should be: [0, 1, 1, 1]

Training Step: 4999  | total loss: 0.50493 | time: 0.004s
| Adam | epoch: 4999 | loss: 0.50493 - acc: 0.7813 -- iter: 4/4
--
    0.01631585881114006
    0.4872587323188782
    0.9684665203094482
    0.019177177920937538


texts: How to incorporate the uncertainty of the model coefficients in the prediction interval of a multiple linear regression

texts:
I'm dealing with the modeling of small experimental data sets. As most experimental work does not generate thousands of samples, but rather a handful, I need to be inventive in how to deal with this small number of data sets (say 10-20). I've been building a nice framework to do just this, and at this point I am interested in generating error-bars with the predicted values.

In a rough outline, this is what the happens in the framework (e.g. when applying a multi-linear model):

Create an ensemble of N data sets.
On each data set a regression gives rise to a (linear) model as given in Eq.1 below. This gives rise to N values for each of the coefficients $\beta$.
The mean is calculated for each of the three sets of $\beta$'s. (The mean could also be another function, but for now assume it's the mean)
These three mean $\beta$'s are the coefficients of the model to be used (again Eq.1).
The goal: find the prediction interval (PI) for the model in Eq.1 taking into account the fact that the coefficients $\beta$ are calculated from numerical distributions.

So take for example the following multiple linear regression model:
$$
y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 \tag{1}
$$
and I'm looking for an algebraic equation to calculate (numerically) the prediction interval (PI) for a new prediction $y_0$ (Confidence interval would be OK as well as it is related to the PI).

So far, my searches have only been able to provide me with answers which deal with the statistical nature of the data set ($x_i$'s). These provide me with an error component:
$$
\hat{V}_f=s^2\cdot\mathbf{x_0}\cdot\mathbf{(X^TX)^{-1}}\cdot\mathbf{x_0^T} + s^2 \tag{2}
$$
which can be used to calculate the PI, via:
$$
y=y_0 \pm t_{\alpha/2,n-k}\cdot\sqrt{\hat{V}_f} \tag{3}
$$

In contrast to those examples, each of the model coefficients ($\beta_0, \beta_1$ and $\beta_2$) in this case have an error-bar (extracted via bootstrapping from a distribution, with the distributions being numerical in nature not analytic, and the distributions are specific for each of the three coefficients).
Is there a way to incorporate the uncertainty of the $\beta_i$'s (c.q. the "error-bars") in the calculation of the PI (and CI).

Note
I Know, one could create an ensemble of the various model instances with the $\beta_i$ drawn from their respective distributions, and based on the distribution of obtained $y_0$ calculate the CI of the $y_0$, but this is not really computationally efficient and brings a lot of other issues which I would like to avoid.


texts: How to implement accurate counts or sums when you have different numbers of days of the week?

texts:
I have a dataset of transaction data for a retail outlet. I am using pandas and want to analyse revenue by day of the week, but there are unequal days in the dataset (i.e. an extra weekend). I have used df.dt.dayofweek to create an integer value for the day of the week, and grouped the data by that integer value using df.groupby(["Day_int",]).sum()

So I have a 'total' column at the end that I am interested in, but I want to create a a new column, something like 'adj_total' that applies a division operation of /3 to the days Monday to Friday and /4 to the days Saturday and Sunday. Is the best way to loop through the dataframe or is it better

Here is the df that I am working with (most of the column values are nonsensical, only total is of interest). Day_int is the group_by variable and should appear lower than the other columns.

Day_int Section Prod_name Cashier Date Time Receipt Total
0 ..................................................91341
1 ..................................................82262
2 ..................................................84145
3 ..................................................90115
4 ..................................................115497
5 ..................................................151971
6 ..................................................109210


texts: Multiple solutions with same minima in MLP with same weights

texts:
I came across an excercise on deep learning from here.
It goes as follows:
Consider a simple MLP with a single hidden layer of $d$ dimensions in the hidden layer and a single output. Show that for any local minimum there are at least $d!$ equivalent solutions that behave identically.
As the network is a MLP with one hidden layer, the equation would be:
$O = W^{(2)}(W^{(1)}x + b_1) + b_2$
Assuming I am correct, where do I need to go from here to get to the solution?


texts: Document classification using convolutional neural network

texts:
I'm trying to use CNN (convolutional neural network) to classify documents. CNN for short text/sentences has been studied in many papers. However, it seems that no papers have used CNN for long text or document.
My problem is that there are too many features from a document. In my dataset, each document has more than 1000 tokens/words. To feed each example to a CNN, I convert each document into a matrix by using word2vec or glove resulting a big matrix. For each matrix, the height is the length of the document, and the width is the size of word embedding vector. My dataset has more than 9000 examples and it takes a lot of time to train the network (a whole week) which makes it difficult to fine-tune parameters.

Another feature extracting method is to use one-hot vector for each word, but this will create very sparse matrices. And of course, this method even takes more time to train than the previous method.

So is there a better method for extracting features without creating large input matrices?
 And how should we handle variable length of documents? Currently, I add special strings to make document have the same length, but I don't think it's a good solution.


texts: Can someone explain how 'One-Dimensional Convolutional Neural Network' works. I do understand the 2-D for image but for 1-D how is the filer created. is it fixed 1-D filter within a specific time interval or the operation is the same as we convolve a signal with a filter in signal processing y = f*x


texts:
There is a well known problem vanishing gradient in BackPropagation training of Feedforward Neural Network (FNN)(here we don't consider the vanishing gradient of Recurrent Neural Network).
I don't understand why vanishing gradient does not mean the zero gradient namely the optimal solution we want? I saw some answer said vanishing gradient is not exactly the zero gradient, just means the update of gradient is very slow. However, the stopping rule of gradient decent is just the unchange of parameter within $\epsilon.$
So can anyone give me a clear answer?


texts: Dot product as template matching with a linear classifier(cs231n)

texts:
I'm not sure I understand something from this article:
https://cs231n.github.io/linear-classify/
In the context of linear classification of images, it is written:
Interpretation of linear classifiers as template matching: Another interpretation for the weights W is that each row of W corresponds to a template (or sometimes also called a prototype) for one of the classes. The score of each class for an image is then obtained by comparing each template with the image using an inner product (or dot product) one by one to find the one that “fits” best. With this terminology, the linear classifier is doing template matching, where the templates are learned.
But how could we use this as template matching for images with pixel values of 0? Which might happen also for mean centered data, as the image will be grey.
Also, not just for 0 pixel values, but low pixel values in general, will the weights be very large to compensate for low pixel values?(and then W as an image will not be similar by color to the images we learned from)
Thanks


texts: Sequential pattern mining in SPMF: Giving words as input

texts:
My dataset is the result of multiple joins from large transactional database stored in a MySQL database. There are some columns with text value and double values.  I am trying to do a first level sequential pattern mining on this dataset using SPMF.

Is it possible to give words as input in SPMF? The documentation tells that only positive integers are allowed.
If I want to give words or decimal numbers, which software/package shall I use?

Thanks.


texts: Time series forecasting in Python with 2 categorical variables

texts:
What approach is the best for a time series forecasting where you want to include 2 categorical variables in python? Im not finding any useful information that can help guide me with this; mainly everything I've found is about numerical variables only (both independent and independent variables)
My data looks like this:
Basically I have 10 years (2010-910) of the medium salary data, per county (42) per field of activity (19). I am looking for guidance to get predictions for next years. As in - what would be the medium salary for a hotel agent in 2020 in X county?
Any ideas? I am a newby in ptyhon so I could use some code as well, thanks in advance


texts:
I have a bunch of scipy matrices (of the same #columns) loaded from disk.

I want to combine them into one scipy sparse matrix.

I am using scipy sparse vstack method.

I am able to load the matrices individually (with adequate memory left in RAM), but during the vstack operation, I am getting memory overflow error.

Can anyone suggest an alternative to vstack to stack sparse matrices?


texts: Why do we have to divide by 2 in the ML squared error cost function?

texts:
I'm not sure why you need to multiply by $\frac1{2m}$ in the beginning. I understand that you would have to divide the whole sum by $\frac1{m}$, but why do we have to multiply $m$ by two?

Is it because we have two $\theta$ here in the example?


texts: difference between XGBRegressor and XGBClassifier

texts:
I'm trying to understand the difference between xgboost.XGBRegressor and  xgboost.sklearn.XGBClassifier.

Can someone explain the difference in a concise manner?

Because when I fit both classifiers with the exact same data, I get pretty different performance.

This is how I fit the data.

clf = xgboost.XGBRegressor(alpha=c)
#clf = xgboost.sklearn.XGBClassifier(alpha=c)

clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print('Model roc auc score: %0.3f' % roc_auc_score(y_test, y_pred, average='macro', sample_weight=None))
print('Model pr auc score: %0.3f' % average_precision_score(y_test, y_pred))

when  clf = xgboost.sklearn.XGBClassifier(alpha=c)

Model roc auc score: 0.544

Model pr auc score: 0.303

when clf = xgboost.XGBRegressor(alpha=c)

Model roc auc score: 0.703

Model pr auc score: 0.453

What would cause this performance difference?


texts:
Is it possible to use a sequence of numbers as one feature?

For example, using libsvm data format:

<label> <index1>:<value1> <index2>:<value2>

+1 1:123.02 2:1.23 3:5.45,2.22,6.76
+1 1:120.12 2:2.23 3:4.98,2.55,4.45
-1 1:199.99 2:2.13 3:4.98,2.22,6.98
...

Is there any special machine learning algorithm for this kind of data?


texts: Should columns with close to zero variance be removed before or after one hot encoding?

texts:
I've been using R and the caret package since a while. The caret package provides a function to reduce the number of variables - nearZeroVar() wherein variables that have close to zero variance are returned and one can remove them from the dataframe

Caret also provides another function dummyVars that converts factors to dummy variables.

Now my question is - what should the order of using the two be? Should the features with near zero variance be thrown out and then converted to dummies or should it be the other way around? I have a feeling that the order matters because in some cases dummy variables might have a lot of zeros but still be important.


texts: Models to deal with huge possible outputs

texts:
My question is what kind of machine learning models could be used in the case we need to predict something from a large pool of possibilities. For example in a kaggle competition( Instacart challenge ) , they ask to predict what products will a user reorder , based on prior orders.

In this case the spectrum of possible outputs is huge ( there are +40000 possible products )

I have studied some models like CNN and usually the output for this models, is a logits tensor holding the probabilities, the size of the tensor usually corresponds to the possibilities in the classification problem. That are generally small ( maximum something like a hundred in cifar-100

But in this case, seems to me that this is not a good approach, because of the number of possible items.

So this CNN models are appropriated in this kind of tasks? what other approaches could be used?

Thanks.


texts: Counting the transition in a dataframe overtime

texts:
I am stuck at a problem and am thinking how to come out of it. I want to write a code in python with dataframe as below:

data = {'Id':['a', 'a', 'b', 'b', 'c', 'c'],
        'value':['Active', 'Notactive', 'Active', 'Superactive', 'Notactive', 'Superactive'],
        'date':['8-09-2019','15-09-2019','8-09-2019','15-09-2019','8-09-2019','15-09-2019']}
df = pd.DataFrame(data)
I want to reach to the result where i count the number of users who did transition in the 7 day difference:
Active to notactive: 1
Active to superactive: 1
Not active to superactive: 1
Active to Active: 0

Any help on how to proceed would be appreciated on python 3.7

Thanks


texts: What are good resources to learn the basics of graph theory

texts:
I just spent half a day "debugging" a script that tried to get the betweenness centrality of a weighted graph with negative weights. However the package I used for that julia's LightGraphs uses Dijkstra's shortest paths, which require non-negative weights as input. I didn't know this, and thus wasted a lot of my and other peoples time trying to find out why my script didn't work.
I the end I found it out. But I figured it would be good to learn some basic graph theory to avoid this kind of problems in the future. Can anybody recommend a good [book|course|other resource] on graph theory? I don't look for a source that rigorously proves all kind of properties, but more of an overview. If I need to know more about a specific graph/test/metric I can always look it up later.


texts: forecasting - likelihood of customers participating in next month sales

texts:
I have historical transaction information of customers for the last 2 years and other information about the customers like what type of card (gold/platinum) they used for transactions etc. is also there in the dataset. Using this dataset, I will have to forecast the likelihood of each customers transacting next month. What are the different approaches that I can analyze before I choose one?

EDIT

More info on the problem:

I have customer credit information and transaction history of the cards for an online ticket booking portal. From this portal customers can book flights, cruises and cars.
I have to predict the customers who are most likely to book something (flights, cruises and cars) in the next month. Now what possible approaches can I take for this?


texts: What regression to use to calculate the result of election in a multiparty system?

texts:
I want to make a prediction for the result of the parliamentary elections. My output will be the % each party receives. There is more than 2 parties so logistic regression is not a viable option. I could make a separate regression for each party but in that case the results would be in some manner independent from each other. It would not ensure that the sum of the results would be 100%.

What regression (or other method) should I use? Is it possible to use this method in R or Python via a specific library?


texts:
I'm trying to create a model that generates worlds for a game. The game is 2-dimensional and small worlds sizes. I've sampled about 15,000 worlds that are about 200x200 blocks. Each block has its own id ranging from 0 to around 1500, although not all ids are used.

Right now I'm training with 64x64 samples. Here's where I'm seeking advice. Right now I'm normalizing all the blocks into a sequential hash before normalizing them from -1 to 1. If I'm training with ids 0, 2, 3 I would first hash them into 0->0, 2->1, 3->2, then into 0->-1, 2->0, 3->1. I don't know if that's right, because my generator is getting completely weak after a few epochs and the discriminator loss is converging to 0 usually pretty quickly. I know I haven't given much information, but I'm just wondering if another approach might be easier to train. My other idea would be to have a generator that outputs a 64x64x300 where each 64x64 dimension corresponds to a list of probabilities for each x, y for that block (like a softmax?) Any sort of advice would be appreciated, as I'm very much an amateur with machine learning.

Thanks, and let me know if you'd like more information for context (I'd thought I would spare the potentially unnecessary details)


texts: How can one quickly look up people from a large database?

texts:
Vocabulary

Face detection: Finding all faces in an image.
Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely makes follow-up tasks hard. Face embeddings are one other representation. In this case a face is a point on the unit-sphere in $\mathbb{R}^{128}$, IIRC.
Face verification: Given two face representations, deciding if they are the same

Question

I was just wondering how identifying a person with many potential people can work. So finding a face in an image works quite well and fast enough for most applications. Face verification as well. But I'm not sure how to scale this if you don't compare 1 face against 1 other face, but 1 against millions / billions of faces.

Suppose you have a lot of examples of faces with the identity of the person. Think of Facebook, where many people tagged friends. Or of countries with biometric passports.

In the real applications, the face verification task is easy because you can just brute-force compare against all candidates:

Facebook: Only candidates are your friends, so ~200 candidates.
Airport EU fast entry: Your face is compared against the passport. So only one candidate.

But then think about some dystopian books / movies, where cameras identify anybody. While tracking helps to reduce that problem, finding a match from millions / billions of examples is computationally super heavy. Assuming a single face verification takes ~200ms, for a million candidates it would already take 60h. For a billion users it would already be 6 years. For all people on earth who currently live it is 48 years.

So with that many candidates, you don't want to compare against all candidates.

When you use the face-embedding, it becomes a nearest neighbor search in $\mathbb{R}^{128}$.

Calculating the euclidean distance of two vectors in $\mathbb{R}^{128}$ takes roughly 15μs (see "timing" below). This means a single check over $7.5 \cdot 10^9$ people would take 31h. Way better, but still pretty long.

While the face-embedding approach pre-computes a good face representation, going over all examples is still a pretty dumb approach. If it was only $\mathbb{R}^1$, one could make a simple binary tree. For few dimensions, I think something like a k-d-tree might work. But what about 128 dimensions?

Is there another approach to get the person quicker?

Timing

import numpy as np
durations = timeit.repeat('np.linalg.norm(a-b)',
                          setup='import numpy as np;a=np.random.random(128);b=np.random.random(128)',
                          repeat=1000,
                          number=3)
print('min: {min:5.1f}μs, mean: {mean:5.1f}μs, max: {max:6.1f}μs'
      .format(min=min(durations) * 10**6,
              mean=np.mean(durations) * 10**6,
              max=max(durations) * 10**6,
              ))


texts: Calculating % by Dividing Filtered Matrix Columns in MS Power BI

texts:
Given: A monthly percentage (%) metric has to be calculated from dividing a column ('Numerator') from one table by a column ('Denominator') from another table, both filtered by month, as given in an example below:

Table 1:

Date_1   Numerator
01-Jan-19   5
05-Feb-19   4
04-Apr-19   1
07-May-19   3
11-Jun-19   5
22-Jun-19   4
25-Jul-19   5
31-Aug-19   1
03-Sep-19   4
25-Oct-19   5

Table 2:

Date_2   Denominator
03-Jan-19   7
05-Jan-19   9
16-Feb-19   8
22-Feb-19   7
04-Mar-19   10
18-Mar-19   8
24-Apr-19   8
25-Apr-19   8
01-May-19   10
10-May-19   8
04-Jun-19   8
08-Jun-19   7
03-Jul-19   8
18-Jul-19   6
23-Aug-19   10
31-Aug-19   9
03-Sep-19   9
20-Sep-19   7
03-Oct-19   7
27-Oct-19   6

One way this task can be solved is by using pivot tables in MS Excel by dividing the rows from 'Numerator' by rows from 'Denominator' corresponding to the same months in filtered 'Date_1' and 'Date_2' columns, and then formatting the results as %, as shown in Table 3 below:

Table 3:

Month_1   Sum of Numerator     Month_2   Sum of Denominator   Month   %
Jan       5                Jan       16                       Jan   31.25%
Feb       4                Feb       15                       Feb   26.67%
Mar                        Mar       18                       Mar   0.00%
Apr       1                Apr       16                       Apr   6.25%
May       3                May       18                       May   16.67%
Jun       9                Jun       15                       Jun   60.00%
Jul       5                Jul       14                       Jul   35.71%
Aug       1                Aug       19                       Aug   5.26%
Sep       4                Sep       16                       Sep   25.00%
Oct       5                Oct       13                       Oct   38.46%
Nov                        Nov                                Nov
Dec                        Dec                                Dec
Grand Total 37             Grand Total  160             Grand Total 23.13%

MS Power BI is used for querying the data automatically from the database, but the problem arises with division of pivot tables (which are represented as matrices in Power BI).

I want to obtain an equivalent of Table 3 in Power BI, which is something like this (disregarding the months ordered in alphabetical instead of chronological order):

But when trying to do calculation on original 'Numerator' and 'Denominator' columns from Tables 1 and 2, respectively, I obtain the following:

Question: How can one obtain an equivalent output to 'Month' and '%' columns from Table 3 in Power BI using the equivalent of Tables 1 and 2? I am new to Power BI and and am not sure of what the problem is and how to correct it.


texts:
I have a matrix with sparse data. A small extract from it is seen below. The columns represent years and the rows represent different race tracks. The feature values are velocities on that specific track a specific year. Generally the velocity increases with the year but that is not necessarily true. As seen below the matrix is sparse and for some tracks I only have values for a single year. How can one most accurately predict the missing values? I suspect one can use some kind of algorithm that resembles a recommender system like the ones used for recommending movies. Those are also based on matrices with lots of missing values but I don't really know how to adapt this to my problem.
I'd be very grateful for any hints on how to solve this.
/Hanson


texts: Uneven dataset, how I should train the model if I can't load all pictures at once

texts:
I have a dataset of multiple classes (About 50). The dataset doesn't have the same number of pictures per class, some have 300, some have 1000, and some more, and I've seen that this ruined my accuracy on my model

First of all, the data is too big for me to store in the RAM, so I have to split it into parts (I take 300 pictures of each class at once), train the network on that data then repeat the process until I have no more photos left.

The question I have is, is it a problem if I do this (I guess it is from my results)? I have for example from 100 classes of pictures, only 10 left and I train the model only on those 10,then on only 5,then on 3,etc.? Because I did that and after I went over the smallest classes, the overall accuracy went up, but those smaller classes accuracy went down, and in the end from 80% in the beginning for each class I got 0-2% on 48 classes, and 99% on 2 classes.

How can I solve this 'unevenness' (I don't know the word) then so I won't have this problem anymore?


texts: How to implement a cascaded neural network in Keras where 1st NN's forward output is cloned twice to perform forward on the 2nd NN?

texts:
Using Keras, I am trying to reproduce a few basic results from a published paper.
In this task, there are two neural networks - A & B, that are connected in a cascade formation, i.e. the output of model A feeds the input of model B. Please see the figure below.

Here are the specifics steps that need to be implemented:

Perform one forward pass on model A with a batch size of 64, and 13155 rows.
Clone the output to create a batch of 128 rows i.e. use each training sample twice. (There are auxiliary inputs which distinguish the duplicate rows, but I have omitted that info for the sake of brevity).
Perform one forward pass on model B by feeding it with #2 above.
Now backward the two networks, calculate a loss for each one, sum the loss & send it to the optimizer with the gradients of both networks.

As I am quite new to the world of neural networks & Keras, I'd really appreciate any help from the community to implement the above steps in Keras. In particular, I am quite confused about implementing step 2 (please see the attached figure).
As a starting point, I had found this answer to be somewhat aligned to my question, but the details of customization to my specific problem is fuzzy, for example how to implement the forward & backward logic separately etc.


texts: Subgraph isomorphism and Anti-monotone property

texts: While finding frequent subgraphs in single large graph, subgraph isomorphism (test) is not considered because its not anti-monotone. How and why subgraph isomorphism is not anti-monotone ?


texts: What is the reason behind having low results using the data augmentation technique in NLP?

texts:
I used Data augmentation technique on my dataset, to have more data to train. My data is text so the data augmentation technique is based on random insertion of words, random swaps and synonyms replacement.

The algorithm I used performs well in other datasets, but in mine, it gives lower accuracy results comparing to the original experiment. Are there any logical interpretations?


texts: Could not convert string to float error on KDDCup99 dataset

texts:
I am trying to perform a comparison between 5 algorithms against the KDD Cup 99 dataset and the NSL-KDD datasets using Python and I am having an issue when trying to build and evaluate the models against the KDDCup99 dataset and the NSL-KDD dataset.

Whenever I try to run the algorithms on the datasets I get the following error 'could not convert string to float: S0'

This error is produced during the during the evaluation of the 5 models; Logistic Regression, Linear Discriminant Analysis, K-Nearest Neighbors, Classification and Regression Trees, Gaussian Naive Bayes and Support Vector Machines.

Here is the code that I am using to evaluate the datasets:

#Load KDD dataset

dataset = pandas.read_csv('Datasets/KDDCUP 99/kddcup.csv', names = ['duration','protocol_type','service','src_bytes','dst_bytes','flag','land','wrong_fragment','urgent',
'hot','num_failed_logins','logged_in','num_compromised','root_shell','su_attempted','num_root','num_file_creations',
'num_shells','num_access_files','num_outbound_cmds','is_host_login','is_guest_login','count','serror_rate',
'rerror_rate','same_srv_rate','diff_srv_rate','srv_count','srv_serror_rate','srv_rerror_rate','srv_diff_host_rate',
'dst_host_count','dst_host_srv_count','dst_host_same_srv_rate','dst_host_diff_srv_rate','dst_host_same_src_port_rate',
'dst_host_srv_diff_host_rate','dst_host_serror_rate','dst_host_srv_serror_rate','dst_host_rerror_rate','dst_host_srv_rerror_rate','class'])

# split data into X and y
array = dataset.values
X = array[:,0:41]
Y = array[:,41]

# Split-out validation dataset
validation_size = 0.20
seed = 7
X_train, X_validation, Y_train, Y_validation = cross_validation.train_test_split(X, Y, test_size=validation_size, random_state=seed)

# Test options and evaluation metric
num_folds = 7
num_instances = len(X_train)
seed = 7
scoring = 'accuracy'

#  Algorithms
models = []
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))

# evaluate each model in turn
results = []
names = []
for name, model in models:
    kfold = cross_validation.KFold(n=num_instances, n_folds=num_folds,

random_state=seed)

    #Here is where the error is spit out
{
            cv_results = cross_validation.cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring) # Could not convert string to float happens here. Scoring uses string.
            results.append(cv_results)
            names.append(name)
            msg = "%s: %f (%f)" % (name, cv_results.mean()*100, cv_results.std()*100)#multiplying by 100 to show percentage
            print(msg)
}

# Compare Algorithms
fig = plt.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(Y)
plt.show()

Here is a 3 line sample from the KDDcup99 datatset:

0   tcp http    SF  215 45076   0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   1   1   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   0   0   normal.
0   tcp http    SF  162 4528    0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   2   2   0   0   0   0   1   0   0   1   1   1   0   1   0   0   0   0   0   normal.
0   tcp http    SF  236 1228    0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   1   1   0   0   0   0   1   0   0   2   2   1   0   0.5 0   0   0   0   0   normal.

I have tried using label encoding and it still spits out the same error and when I was looking through the sklearn websites, I noticed that the scoring value was for the string type, is this the cause of the issue? and if not, is there a problem with the way I have loaded the dataset?

EDIT I tried removing scoring value from the code and still got the same error.


texts:
I am confused about random_state parameter in some algorithms like AdaboostClasifier, DecisionTree and so on

Here is an example

from sklearn.model_selection import *
from sklearn.ensemble import AdaBoostClassifier
param_grid = {
    'learning_rate':[0.001, 0.10, 0.1, 1],

             'n_estimators':range(50, 400, 50)
             }

abc = AdaBoostClassifier(random_state=123)

# run grid search
grid_abc=GridSearchCV(abc, param_grid, scoring = 'accuracy')
grid_abc.fit(X_train, y_train)
#The best hyper parameters set
print("Best Hyper Parameters:\n",grid_abc.best_params_)
print("training accuracy:\n",grid_abc.best_score_)
prediction=grid_abc.best_estimator_.predict(X_test)
#importing the metrics module
from sklearn import metrics
#evaluation(Accuracy)
print("Accuracy:",metrics.accuracy_score(prediction,y_test))
#evaluation(Confusion Matrix)
print("Confusion Matrix:\n",metrics.confusion_matrix(prediction,y_test))
print("\t\tclassification report")
print("-" * 52)
print(metrics.classification_report(prediction,y_test))

The accuracy results is 0.9420289855072463

But when i change the random_state value to 0
I got another accuracy results 0.8584070796460177

How can i fix the result and be sure of the final results.It is ricky by the way Do i have to set random_state value of the train_test split as the classifier or no?


texts: About data cleansing, to what extent should we do our work?

texts:
As far as I know, when we deal with 'big data'. It is common we deal with more than 10 years' customers data. Quality issue is always there.

What I am thinking about, to what extend we should deal with the data quality issues? A robust algorithm should tolerate with the data quality issues but if we do better data cleansing work we get outcome confidently.

So how to balance this?

Somebody please give some ideas!

Thanks.


texts: Suggest an order to read data science books

texts:
I'm an undergraduate student that is currently studing data science.
However, since we generally don't delve deeply onto things in class, I want to take some time to review the fundamentals and understand things deeply in order to advance to more complex notions. When I did some research I came across these books :

Data Mining: The Textbook, by Charu C. Aggarwal
The Elements of Statistical Learning, by Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie
Pattern Recognition And Machine Learning, by Christopher M. Bishop
Machine Learning: A Probabilistic Perspective, by Kevin P. Murphy
Computer Age Statistical Inference: Algorithms, Evidence, and Data Science, by Bradley Efron and Trevor Hastie

If you know or came across these books, can you please help me choose the proper order to read these books, or tell me if these is a redundant book, or even suggest a book to add to the list

Any help is appreciated, thanks.


texts:
I use EfficientNetB0 for performing image classification with one of the class as "stone countertop table", the other class is "not stone countertop table". I use the reddest part in gradCAM to put a point saying that this is the "stone countertop table". But in some cases, it is pointing to the basin (as most of the stone countertop accompany with a basin, and lead efficientNETB0 focus on the wrong thing). Is there a way to tell EfficientNetB0 not to use basin as a hint to look for "stone countertop table", and therefore the reddest part in gradCAM will always point to the surface of "stone countertop table"?
This is what is desired:

This is not:

This is not as well:


texts: Fewer observations & larger documents vs More observations & smaller documents

texts:
Let's suppose that I have a dataset of 1000 documents.

Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}.

Let's suppose that the dataset has 600 positive reviews, 200 indifferent reviews and 200 negative reviews.

I want to train a classifier to classify a review as Negative or Indifferent or Positive based on the text of the review.

I am not thinking about using any word embeddings for now so I will probably use a TF or TF-IDF model (even though this may be a bit off topic for current question).

Let's suppose that in my case I split (in a stratified way) my dataset into a training set of 800 observations and into a test set of 200 observations.

My question is the following:
Is it better to have 800 separate documents in my training set or to merge these documents based on its categories/labels and create 3 very big documents?

There 800 separate documents of any of the 3 labels or 3 big documents of each of the labels is the best way to go and why?

My question stems from the fact that in the latter case if for example I do TF-IDF then this will be applied based on different categories/labels since each document will be about a category/label.

On the other hand, if I do (as we usually do actually) like in the former case then the TF-IDF will be categories/labels-agnostic and I do not know this helps things.

Is the answer simply that this an interesting but pretty bad idea because in this way you simply massively decrease the number of the observations with which the model/algorithm is trained and so you make much harder for him to figure out how to successfully classify things?


texts:
R newbie here. I'm doing some text analysis using the package quanteda. Basically, what I'm trying to do is put all the words follow the regex pattern child|(care) basically to capture any text which includes any of the words "child" or "care".
To do this, I can create a list and then use the dictionary function:
childcare_list <- c("child","care")
word_dict <- dictionary(list(childcare = childcare_list)).
However, how could I incorporate regex and do this for other patterns which would be tedious to type up manually as in the first line? For example, I may want to capture something like
\bC\w?V\w?D\-19 which captures possible typos of "COVID-19" e.g. "CiVID-19", "CpVID-19".
I could of course do covid_list <- c("CiVID-19", "CpVID-19", ...) but that would be too manual. As well, it doesn't use the \b anchor.
Basically, asking if it's possible to make it so a list contains all possible combinations of a regex.


texts:
I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct.

My questions:

What are the best practices to understand a dataset (high dimensional with numerical and nominal attributes)?
Practices to make sure the dataset is clean?
Practices to make sure the dataset doesn't have wrong values or so?


texts: How is bias added in a convolutional layer

texts:
In a typical neural network , bias is usally added like this

 v = activation ( w1*x1+ ... + Wb*b)

However, I am not really sure how it is done in convolutional layer. My one thought is that it is added with each convoltional operation for a neuron. Is that correct?


texts: Q learning and Neural Network for Tic Tac Toe

texts:
I have been working on a tic-tac-toe assignment for my Robot Learning class. We were asked to program a tic-tac-toe game and assign; +1 if X wins, -1 if O wins and 0 it the game results with a draw. In part 1, we were told to use Q table and in part 2 we were told to replace the Q table with a Neural network as the functional approximator.

It is my understanding that both methods should be achieving the optimal policy, can you confirm or deny my understanding?


texts: Semantic segmentation of an image with multiple labels per pixel

texts:
I am building a model for a multiclass sematic segmentation of a skin disease. At a moment I am using U-Net for binary classifications.
In this multiclass problem I have the following cases. There are four types of skin damage. There are four degrees of damage for each skin damage type: healthy, mild, moderate, severe. Healthy skin is A0_B0_C0_D0, "mild b and severe c" corresponds to A0_B1_C3_D0. I would like to train a single multiclass model to predict a dictionary {A: scoreA, B: scoreB, C: scoreC, D: scoreD}. Given that
4(damage types)**4(damage degrees) = 256 combinations
Do I need to train a 256 class model? My concern is that I only have 200 images in my training set and some of the combinations will not appear at all in the training set. Is there a way to train a 12 class model returning four values corresponding to  the most likely damage type for each damage type?
Update. Consider an rgb image. You want to classify brightness of each color channel in categories
Intensity   class
 0-63 -> "0"
64-127 -> "1"
128-191 -> "2"
192-255 -> "3"

Then each pixel belongs to one of 64 classes (r0g0b0, r0g0b1, ..., r3g2b2, r3g3b3). The training set has pixels of colors r0, r1, r2, r3, g0, g1, g2, g3, b0, b1, b2, b3, but it has no pixels of color r0g1b2 or of color r2g3b0. Three separate models (one per channel) will easily learn to predict the channel category, but it will never output r0g1b2 and r2g3b0 classes in 64 class model because it have never seen those classes. How to overcome this problem?


texts: Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

texts:
I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further downstream task.
Bert's last layer looks like this :

Where we take the [CLS] token of each sentence :

Image source
I went through many discussion on this huggingface issue,  datascience forum question,  github issue Most of the data scientist gives this explanation :

BERT is bidirectional, the [CLS] is encoded including all
representative information of all tokens through the multi-layer
encoding procedure. The representation of [CLS] is individual in
different sentences.

My question is, Why the author ignored the other information ( each token's vector ) and taking the average, max_pool or other methods to make use of all information rather than using [CLS] token for classification?
How does this [CLS] token help compare to the average of all token vectors?


texts: How to write decider function for multiple models

texts:
I have trained two classifiers .. Text Classification and Image Classification.
So both models gives score for each class. For example there are 3 classes. Each model give array of 3 confidence score for each class.
I want to write some decider function to predict the final class based on these two model results.
Dataset is very less around 100 observations.
What possible solution i can try.


texts: Which Kind of Machine Learning should I use for an Optimization Problem?

texts:
I have a problem where I would like to apply machine learning. For this purpose I have been learning about the topic and have come across supervised, unsupervised, semi-supervised machine learning techniques.

I know the problem that I want to solve, so unsupervised machine learning doesn't seem to be a fit, because from how I understood, it is used when we don't know what we are looking for.

But I don't have a pre-labelled data set either, because we ourselves don't know if we are operating at the optimum level.

Basically we want to optimize profit, and we have the option of tweaking certain parameters. As an example let's say we want to the ML Algorithm to find the price of a product that would optimize the profit.

What kind of ML can I apply here? Do you have any recommend frameworks? Any book that I can read to understand this (the books that I have referred to so far have all given examples of classification)?

Adding more context:- The actual problem is to balance a logistics network. There are multiple warehouses and delivery service providers(DSP) (DHL, GLS etc). We have some parameters that define to which warehouse/DSP we want to send the orders(to be packed and delivered). We want to find the optimum parameters that will yield profit. We have already processed data where we can train and validate the algorithm. But we don't know if the current parameters are correct, so I would assume that training the algorithm with the data that we have would not give us an optimum solution (because the labelled targets, i.e. the achieved profit might not be optimum)

With this specific algorithm we are trying to figure out the best warehouse/DSP combination for a given order. While we want to send a parcel to a cheapest warehouse/DSP option, it might be the case that the cheapest warehouse/DSP is out of capacity (limited by manpower at the warehouse or limited by the number of ordered trucks of a given DSP), in which case we might choose to send the order to a non cheapest warehouse, or to send the order the next day. With this solution we don't plan to optimize anything else (that is optimizing how stock is stored/storage costs etc.). The goal is to utilize the warehouse/truck capacity the best way we can. I don't need a full blown solution, but a few keywords/concepts/places to which I can refer would be enough.

PS:- Sorry for the half blown example in the original question. I just wanted to give a simple/similar example so that I can get some keywords, frameworks, books to further study.


texts: How would you describe cluster 2 from this output of a run of the EM program?

texts:
My description:

Cluster 2 consists of 9511 instances, the age is around 42 (ranges between 29.7207 and 54.5257). Considering Age, Cluster 2 is very well separated from Cluster 1, with a distance of 18.9513. On the other hand, Cluster 2 and Cluster 0 are very close though, their centroids are withihn a distance of around 0.8248.

What else could be added?