texts: Improvements to video-game cover CNN classifier (keras)

texts:
As a personal project, I'm trying to build a classifier which attempts to predict the metacritic score of a game based purely on its cover. I figured it would be a fun project to learn Keras image classification with (2D-Convolution to be more specific).

I'm definitely new to this so if I've made any rookie mistakes, please tell me. Here are some notes before I post the CNN:

1) I've written a Metacritic Scrubber which takes PS2, PS3, PS4, Xbox360 and Xbox One games which have metacritic scores, downloads the artwork and labels it with the score. Duplicate games are removed (which may be a mistake since they sometimes have different scores based on platform). I'm pretty happy with this code.

2) I've rounded scores on a scale of 0-9 instead of 0-100 to make the number of classifications smaller.

3) images are 123x98 with 3 channels. All images have been stretched to this size. I wonder if this may be a source of problems because some covers have been stretched. Values are between 0-255 for each channel.

4) Games with square cover art (DLC, non-retail, etc) have been omitted.

This leaves me with a data set of 3816 game covers. I figured this would probably be enough for an initial investigation.

The model I've built has been based on the work by Iwana et al in this paper:
https://arxiv.org/abs/1610.09204

model = Sequential()
model.add(Conv2D(64, (2, 3),activation='relu', input_shape=(123,98,3)))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(128, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Flatten())

model.add(Dense(720, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(720, activation='relu'))
model.add(Dense(720, activation='relu'))
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.001,momentum=0.1,decay=0.0005,nesterov=True), metrics=['accuracy'])
model.summary()

#Save output to file
csv_logger = CSVLogger('training.log')
#Save best model when possible
checkpoint = ModelCheckpoint('training_model_best.hdf5', monitor='val_acc', verbose=1, save_best_only=True, mode='max')

model.fit(np.array(images), np.array(labels), epochs=225, batch_size=10,
    validation_split=0.50,verbose=2,shuffle=True,callbacks=[csv_logger,checkpoint])

I'm using quite a harsh validation split of 0.5 because I really wanted to avoid overfitting. But regardless of whether I use between 0.2 - 0.5, I typically reach a validation accuracy of approximately 35%, which is pretty disappointing. A lot of the hyper-parameters I have no idea what to set to however.

Sorry for the long post. Ultimately, my result is a little better than random guessing but I'm hoping through some pointers, I can bring this accuracy up a bit. I wonder if actually, the cover files I've downloaded are too small.

Thanks for reading.


texts: Instance segmentation performance depending on the number of classes

texts: I have a setting in which I have approximately 10 classes, one of which I would need to put an extra emphasis on. Whilst training a mask-rcnn instance segmentation network (detectron2) I found out that I can do a lot better if I separate this one class and train a completely different network for it and then afterwards combine the predictions. Is this true in general for these networks? It would seem reasonable that the network would perform better with fewer classes to predict, but I couldn't find any good writings about this.


texts: Using machine learning technique to predict commodity prices

texts: Has anyone here tried to predict a commodity's price by using other commodities prices as features in a machine learning algorithm? What techniques have been successful?


texts: Pandas - how to make further sub-groups after groupby( ) and get_group( )

texts:
Sample data in .csv format

| No.|   IP     |      Unix_time     |    # integer unix time
| 1  | 1.1.1.1  |     1563552000     |    # equivalent to 12:00:00 AM
| 2  | 1.1.1.1  |     1563552030     |    # equivalent to 12:00:30 AM
| 3  | 1.1.1.1  |     1563552100     |    # equivalent to 12:01:40 AM
| 4  | 1.1.1.1  |     1563552110     |    # equivalent to 12:01:50 AM
| 5  | 1.1.1.1  |     1563552150     |    # equivalent to 12:02:30 AM
| 6  | 1.2.3.10 |     1563552120     |

Here's the current working code using pandas groupby( ) and get_group( ) functions:

data = pd.read_csv(some_path, header=0)
root = data.groupby('IP')

for a in root.groups.keys():
    t = root.get_group(a)['Unix_time']
    print(a + 'has' + t.count() + 'record')

You will see the results below:

1.1.1.1 has 5 record
1.2.3.10 has 1 record

Now, I want some changes.

For the same IP value (e.g., 1.1.1.1), I want to make further sub-groups based on a maximum time interval (e.g., 60 seconds), and count the number of elements in each sub-group. For example, in above sample data:

Start from row 1: row 2 Unix_time value is within 60 seconds, but row 3 is beyond 60 seconds.

Thus, row 1-2 is a group, row 3-4 is a separate group, row 5 is a separate group. In other words, group '1.1.1.1' has 3 sub-groups now.

How to make it?


texts: Is there a gradient descent-based optimization algorithm that works with non-linear constraints?

texts:
I have a function to optimize with ca. 200 parameters + one constraint (sum of squares of the parameters must be equal one)

This problem can be solved using Lagrange Multipliers and my intuition tells me, that methods that do that must be readily available.

If I had a choice, I would prefer an algorithm existing on JuMP.jl


texts: How to push tasks to remote Hadoop/Spark cluster

texts:
I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin notebook online. So I write my Spark code in a browser window in the moment, which I find not very handy.

I wonder what options are avalable to code on my local computer and "push" tasks to the remote Spark/Hadoop cluster?

So instead of coding in a browser window, I would be able to use (for instance) RStudio, write SparkR code, and execute the code on the remote cluster.

Since I'm rather new to this architecture, I'm thankful for any hints.

Note: This question is not about retrieving data from the Hadoop cluster. I already have a OCDB connection, so that I can can make SQL like queries.


texts: What could be possible features of a textual word?

texts: I wish to construct feature vectors of words in a document and then calculate their linkage distance to detect anomalies. My question is how can I model these features ? If possible please give an example so I can understand better. Further how distance calculated will vary from the text that is anomalous and the one that is normal ? (Please refer to D. Guthrie, PhD. thesis for more clarification).


texts: Tree Based Classification (XGBoost, LightGBM, etc) - Features from embeddings for sparse features?

texts:
I'm wondering if there is a possibility from using embeddings as inputs for tree based classification models?
For example we have a field called type of food, and there are many types of food out there.
One might transform this feature by:

Hashing Trick + OHE: We reduce our feature sizes, so that when we perform OHE we won't have too many columns.
Label Encoding: Transform features into numeric values, best works if there is some type of ordinality, however if not, sometimes it works too.

However based on a lot of paper like: DeepFM, where people use OHE encoded values, and process them into an embedding layer that is basically an autoencoder and generate dense features for the next layers.
But I've never seen people use an autoencoder as an input to a tree, nor is there a library that can do this without utilizing using PyTorch/TensorFlow to build from scratch to keep things simple.
On the other hand, if an individual autoencoder is trained, the autoencoder does not converge based on the classification loss. So I'm wondering if this is even feasible? Even though one can just try it out and build an autoencoder using PyTorch and feed it to XGBoost, etc.


texts: I read that decision trees (I am using scikit-learn's classifier) are robust to outlier. Does that mean that I will not have any side-effect if I choose not to remove my outliers?


texts: The differences between SVM and Logistic Regression

texts:
I am reading about SVM and I've faced to the point that non-kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?

Apparently, SVM chooses the maximum margin classifier and logistic regression is the one that minimizes the cross-entropy loss. Are there situations where SVM  performs better than logistic regression or vice-versa?


texts: How to properly resize input images for transfer learning

texts:
I have to resize some images of different size to 224x224 before they can be passed as input for VGG19, and then apply transfer learning.

I have tried these methods: add patches, take a square of 224x224 from the images' center, automatically adjust height and width.

I would like to know, if there is one, which is the best method for resizing these images.


texts: When to use Random Forest over SVM and vice versa?

texts:
When would one use Random Forest over SVM and vice versa?

I understand that cross-validation and model comparison is an important aspect of choosing a model, but here I would like to learn more about rules of thumb and heuristics of the two methods.

Can someone please explain the subtleties, strengths, and weaknesses of the classifiers as well as problems, which are best suited to each of them?


texts: Summarize and visualize a CSV in Java/Scala?

texts:
I would like to summarize (as in R) the contents of a CSV (possibly after loading it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and max of the data in a CSV file for each numeric (integer or real numbers) dimension. The standard deviation would be cool as well.

I would also like to generate some plots to visualize the data, for example 3 plots for the 3 pairs of variables that are more correlated (correlation coefficient) and 3 plots for the 3 pairs of variables that are least correlated.

R requires only a few lines to implement this. Are there any libraries (or tools) that would allow a similarly simple (and efficient if possible) implementation in Java or Scala?

PD: This is a specific use case for a previous (too broad) question.


texts: What machine learning algorithm should I use for specific user configuration?

texts:
I have a data-set that contains thousands of employee data, including their role, department (Applications Developer, IT Support, Network Management etc.), and using one-hot encoding all of the hardware or software they have been given.

I want to use this data so that when I input a new user with the data - role, department etc. their account will be compared to the model accounts and a prediction of what software or hardware they will need will be output.

What machine learning algorithm should I use for this? I was thinking an unsupervised approach would be appropriate but I am new to machine learning and data science so I could be going about this completely the wrong way.

Thanks for your help :)


texts: How to build an acne (object) detection model?

texts:
I want to create an acne (pimple) detection model and I have a general idea of how to make one but I wanted some input whether there are better ways to go about this.
Here's what I have in mind:

Label my dataset (would I need to go through every image and select every pimple to classify it or is there a better more time efficient way?)
Use TensorFlow's Object Detection API and train a model using my dataset

I'm wondering if it would be easier to detect acne by finding the "outliers" in your skin (where you have different color) or if it's better to create a object detection model?


texts: Convolution layer dimensions in deeper layers?

texts:
I am trying to understand the CNN network dimensions:

layer                   activation shape    number of weights

Input                   (25, 64, 1)         0
Conv2D(7x7, 1, 100)     (25, 64, 100)       4.900 (7x7x100)
MaxPool2D               (13, 32, 100)       0
Conv2D(5x5, 1, 150)     (13, 32, 150)       375.000 (5x5x150x100)
...

The input dimension is 25x64, then first Conv layer applies 100 convolutions - so the output dimension is 25x64x100. Max pooling reduces this to 13x32x100.

Second Conv layer applies 150 convolutions. What's unclear to me is the dimension after the second Conv layer. Shouldn't it be 13x32x150x100 (instead of just 13x32x150)? How is the convolution layer applied on a 3D input?

Number of weights for the first Conv layer is 7x7x100, and for the second layer it is 5x5x150x100, meaning that weights are saved for each of the 100 input layers from the previous conv. This also leads me to think that output dimension should be 4D.


texts: Confusion with euclidean distance implementation

texts: I have to implement the k means algorithm from scratch using python on this data set that has 29 columns and 476 rows. With all these different data points I am confused on how I can calculate the distance between the centroids and the data in order to make sure the data is in the right cluster. I know I have to use the euclidean distance, but I am not sure how I can use it with this data set. All the examples I've seen online usually deal with something that is only 2 dimensional. I need someone to explain to me what kind of approach I should take for this. This is the dataset:


texts: What does the ratio of precision/recall to their ideal values mean, and why are they equal?

texts:
When evaluating rankings, Normalized Discounted Cumulative Gain (NDCG) normalizes the score to a [0,1] range by dividing by the ideal score.

What happens if we take the same idea to Precision (Pre) and Recall (Rec) (at k) by dividing the precision and recall by the "ideal" precision and recall?

Obviously, the ideal precision and recall is 1.0, but this only applies while k (the value determining the number of top rankings to consider) is smaller than the number of actual positives (P). Namely, the ideal precision must drop when k exceeds P, since there aren't any more positives to place.

If we consider the formulae:

Pre@k = TP / k

Ideal Pre@k = min(P,k) / k

(where TP is True Positives)

Rec@k = TP / P

Ideal Rec@k = min(k,P) / P

And when we divide the precision and recall by their ideal values, the denominators cancel out and we are left with the same formula: TP / min(P,k)

This behaves like precision for k <= P, and recall for k>=P.
Conceptually, they represent the fraction of the ideal precision/recall score, but why would they be the same?
Is there some extra meaning I'm unaware of?

Edit:

Would it make sense to average two of these scores if they were calculated from two rankings with differing numbers of positives P? While the average over scores with the same P can still be divided into average precision and average recall halves, using different P's introduces a region where precision and recall is mixed.


texts: Meaning of curved line shape distribution in t-SNE plot

texts:
I have blood test results with 20 features from 170 patients, and trying to predict a categorical disease outcome.
The input features are all continuous values except sex (0:male, 1:female)
There are no missing values (used imputation for the few missing values)

When I plotted a 2D t-SNE plot, the results is as the attached image.(Figure 1)
A single curved line shape.
I expected the plot to be more scattered, but even if I increase the perplexity value from 13 to 20 or 30 it still remains as a line shape.
(the initial value of 13 is from a recommendation that sqrt # of samples is a good starting point)

Reducing the perplexity value to 6 makes the datapoints form small chunks (Figure 2), and I am aware this is because small perplexity values make the datapoints conserve proximity with only a few neighbors so this results in small groups.

In a related question this curved line shape appeared in a time-series dataset. I undetstood that time-series data have datapoints that change in small increments across most of the features, so the t-SNE results form a line.

So this means in my dataset, patients that have similar values, the "direction" of similarity is not random due to human physiology. For instance if patient A & B are similar and B has a slightly higher RBC value, the WBC, hematocrit are also higher and BIL values are lower. Hence, each individual patient may look like data from a time-series dataset.

Does this explanation make sense? Or is could there be a different explanation of this line-shaped distribution of a t-SNE plot?


texts: Classifier design for website screenshots

texts:
I'm working on a project that requires determining if the page representing a hosted file on a third party platform (such as rapidgator or nitroflare) is still up or not.

For example, here is a file that is still up.

Here is a page representing a file that is no longer up.

I have identified several hundred file hosting services and each of them have their own way of displaying up vs down pages.

Is it possible to design/train a model that would classify screenshots of these hosted file pages as up vs down? If so what approach is recommended? Does using a method other than image classification make more sense here?


texts: Can xgboost (or any other algorithm) give bad results with some bad features?

texts:
till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data.

In one of my problems, there are around 150 features and with xgboost I am getting a logloss of around 1 if I use all features. But if I remove around 10 bad features (found using some technique) I am observing a logloss of .45. That is huge improvement.

My question is, can bad features really make such big differences?


texts: REINFORCE algorithm with discounted rewards – where does gamma^t in the update come from?

texts:
I'm looking at Sutton & Barto's rendition of the REINFORCE algorithm (from their book here, pg. 328).

I can't quite understand why there is $\gamma^t$ on the last line. They say:

  [..] in the boxed algorithms we are giving the algorithms for the general discounted [return] case. All of the ideas go through in the discounted case with appropriate adjustments [..] but involve additional complexity that distracts from the main ideas.

This doesn't quite make it clear for me.

I didn't find any other description of REINFORCE that would include this. Does anyone have an idea how the $\gamma^t$ gets there?


texts: Does PCA decrease the feature on my Data set or just decrease the dimension?

texts:
I'm new in AI and sorry if my question is simple. I have a data set and want to use PCA to decrease the feature but after some research on the internet I'm confused about decreasing dimensions and features.

As an example I have a data set with 50 rows and 10 columns, if I use PCA it will reduce a data set with 50x5 (as an example) or 50x10 and just removed some dimensions?

I want to do it in MATLAB and want to use PCA function and don't want to write PCA function by myself.

What is the PCA parameters in MATLAB to decrease the feature? It's a lot of parameters and confused me.


texts: Why is spam detection a classification problem and not a class modelling problem

texts:
Trying to get my feet wet with machine learning on text.

The most common dataset I've seen in this space is the sms dataset with classes ham and spam.

And the most common and successful approach seems to be to model this as a binary classification problem and to use a multinomial naïve Bayes to solve it.

However I'm trying to understand why this is a binary classification problem.

I understand that the spam category has some common features associated with it across the class - such as ads, offers , free discounts and so on.

But there's no definition for what is a ham class is there? The definition of ham is - everything other than spam.

So why is this a binary classification task?

For more context - I'm trying to solve the problem of whether a news article belongs to the politics class or to the non-political class.

Suppose I have a labelled dataset of around 3000 samples in each class.

The non political class is a mix of classes like sports , religion , science and technology and miscellaneous.

Will a binary classifier work better than an algorithm such as oneclassSVM where anything other than political news is an outlier ?

What are some of the other algorithms that I can use to solve this problem? I have heard about PU learning but I haven't seen any implementations of algorithms in any machine learning libraries ( I'm working with python)

If any of you have experience doing class modelling on text. Please share your comments and insights

Thank you!


texts: NLP problem : Choosing the optimal parser for extracting quantitative values from text

texts:
I have a clinical NLP problem for which I would need some help to establish a proper framework.

I am trying to extract different elements from echocardiography reports. Those elements are both quantitative and qualitative.

For example, quantitative elements are in the form :

  "LVEF : 40%", "LVIM : 2 mm/s" , "the tricuspid regurgitation is 4mm/s"

and qualitative elements are in the form :

  "The ventricular function is depressed", "We noticed reduced diastolic function"

Ultimately my objective is to obtain the following table for one note :

ID LVEF LVIM TR Diastolic Dysfunction
1  40   2    4  Reduced

My current framework to do so is to use the Quanteda package and Spacyr.

Tokenization of the text

  This is pretty straight forward.

Parsing

  This is where I am a little bit hesitant.
  I believe that it is key for this project to capture the sequence of the word as retrieving what number is associated with which parameter would be difficult otherwise (i.e. for example in a bag-of-word scenario, it would be impossible to know if 4 refers to the value of LVEF or LVIM for example).

  I do not know what would be the best way to parse the text to retrieve that information.

Dictionary use

  A lot of the echocardiogram parameters are registered in different ways. For example, 'LVEF' is documented as 'Left Ventricular Function', 'Heart Function', 'Left Function'.
  I constructed a dictionary mapping all the variations possible for a given concept.

  At what point of the NLP pipeline should the dictionary be used and how does Quanteda work with custom dictionaries ?

Thank you all!


texts: How does the distribution of "features" in the hidden layers of neural networks work?

texts:
I would love some additional help understanding the hidden layers of neural networks.

I’ve read that each of the neurons of the hidden layers will relate to various features from the given data set.

During and after training, activated neurons combine to form an appropriate output from a given input.

In the case of image recognition, these features might be a curve or an edge in a particular region of the image.

Source: But what is a Neural Network?
(3Blue1Brown)

What I can’t wrap my head is how those features are distributed – what exactly is the mechanism (high-level) deciding which features to extract?

Is it just arbitrarily breaking down the images of the data set randomly and spreading it across all the neurons of the hidden layers?

I would therefore suspect that the number of hidden layers is also therefore determined by this process.


texts: Using Pandas, how do I strip part of a cell from an Excel file?

texts:
Thanks in advance, I have been struggling with this for a couple of hours now. I have a column named 'CaseNumber'.  the data in that column is structured as 'Case ID Number - 00123456'.

I am trying to create a loop that will go through the entire sheet and strip 'Case ID Number - 00'. So the final output would just be the 6 digit number set.

So far I have found the attribute .strip('Case ID Number - 00') but I cannot find the logic to create the for loop.

I appreciate any insight.


texts: Type of regression with nominal, ordinal, interval and ratio data

texts:
Statement of problem: An ambulance is at the hospital dropping off a patient. The goal of the paramedic is to get released from the hospital as soon as possible. I am curious, what are the factors in how long an ambulance off loads a patient at the hospital? Can I predict how long an offload will take given certain variables. And how confident can I be in this model? The Dependent Variable is HospitalTime, it is a ratio type of data and is measured in seconds. The Independent Variables are:

Hospital, a nominal type of data recoded into integers, 1 would stand
for Lee Memorial.
Ambulance, a nominal type of data recoded into integers, 9 would
stand for ambulance #9
PatientPriority is an ordinal type of data recoded into integers. A 1
is a high priority, 2 is a medium priority and 3 is low acuity.
MonthOfCall is an interval type of data recoded into integers. A 6
would be June and 12 is December. A 12 (December) is not twice as
much as a 6 (June) in this case.
HourOfCall is an interval type of data recoded into integers. Once
again, an offload happening at 10:00 pm is not more than something
happening at 10:00 am.
Officer1 and Officer2 are nominal data and are integers representing
an EMT and a paramedic.

My question is this: Given this type of data and my goal to predict the off loading time at the hospital, what kind of regression model should I look into?

I have looked at my statistics books from university days and they are all using ratio data. My data is mixed with nominal, ordinal, interval and ratio.

I have as much data as you could ask for. I have at least 100,000 observations.

Can you please push me in the right direction? What kind of model should I use with this type of data?

Shown below are observations to give you a tiny peek at my data:

IncidentID,HospitalTime,Hospital,Ambulance,PatientPriority,MonthOfCall,HourOfCall,Officer1,Officer2
757620,1849,7,11,2,10,10,234,771,chr(10) 802611,2625,7,11,3,1,18,234,777,chr(10)
765597,1149,7,12,3,11,2,234,777,chr(10) 770926,1785,7,12,3,11,15,234,777,chr(10)
771689,3557,7,12,2,11,14,234,777,chr(10) 822758,1073,7,20,3,3,13,777,307,chr(10)
767249,2570,7,22,2,11,11,560,778,chr(10) 767326,1998,7,22,1,11,18,560,777,chr(10)
785903,1660,7,22,3,12,12,234,777,chr(10) 787644,2852,7,22,3,12,17,234,777,chr(10)
760294,1327,7,23,2,10,14,498,735,chr(10) 994677,3653,7,32,2,2,15,181,159,chr(10)
994677,3653,7,32,2,2,15,181,159,chr(10) 788471,2053,5,9,2,1,3,498,777,chr(10)
788471,2053,5,9,2,1,3,498,777,chr(10) 759983,1342,5,11,2,10,8,474,777,chr(10)
791243,1635,5,11,2,1,18,234,777,chr(10) 800796,1381,5,11,3,1,11,234,777,chr(10)

P.S. This question is cross-posted in Stack-Overflow under the same title and author.


texts: Naives Bayes Text Classifier Confidence Score

texts:
I am experimenting with building a text classifier using Naive Bayes which has been pretty successful on my test data. One thing i am looking to incorporate is handling text that does not fit into any predefined category that I trained the model on.
Does anyone have some thoughts on how to do this? I was thinking of trying to calculate the confidence score for each document, and if < 80 % confidence, for example, it should label the data as "N/A"
This is my code so far:
df_train = pd.read_csv(__________)

text_clf = Pipeline([('vect', CountVectorizer()),
                  ('tfidf', TfidfTransformer()),
                  ('clf', MultinomialNB()),])
text_clf = text_clf.fit(df_train.text, df_train.label)

df['predicted'] = predicted

Like I said, it works well for documents that do fit into one of the categories, but if I have something that clearly does not fit into anything, it will still try and assign it a label, my guess is based on some kind of confidence calculation but just not sure how that works


texts:
What is the difference between using CNN with handcrafted features and CNN without handcrafted features?

Thank you


texts: I am trying to figure out the stationarity of time series data

texts:
Here, this plot shows the number of customers served per day from 1 jan 2018 to 31 dec 2019.
I grouped the entire data by each month and calculated the average and variance per month.
This is the average

This is the variance

I also ran an augmented dickey-fuller test and got the following result
ADF Statistic: -26.031963

p-value: 0.000000

Since, the mean and variance do not vary too much, should I infer that the data is stationary?


texts: Why validation loss worsens while precision/recall continue to improve?

texts:
I'm training a neural network on 'easy' dataset with ~15k examples.
Network overfits pretty fast.

The thing I cannot understand that after 5th epoch validation loss is starting to worsen, while precision and recall are continue to improve for 10 more epochs.
(loss = binary cross-entropy)

Diving deeper:

I've checked if there is a lot of predictions around probability ~0.5, but it is not:

Also, there is a plot of percent of correct predictions based on prediction probability. There is some pattern here, but the number of elements is quite small to make conclusions.

So, my question is: why can it happen, and what to do about it?


texts: Shifted feature distribution across different datasets

texts:
I am trying to validate a classifier using two different training and testing datasets.

The feature I am considering is a feature constructed doing the fold-change between two original features, i.e. $log2(feat_A / feat_b)$

The problem is that, across the two different datasets, I see the same distribution of fold-change across groups but with different values. See, in the images attached, the values for each group (each box) in the different datasets (each plot).

What may be causing the differences in fold-change? In this case the fold-change is different also in sign, which means that while in one dataset $feat_a$ is greater than $feat_b$, the opposite is true for the other dataset. However, the pairwise relation between the two groups (the two boxes in each plot) is maintained.

I was thinking about different normalization procedures on each feature across different dataset, which may explain the shift of the distribution of the fold-change. What other problems may be in these datasets?


texts: In CNN, why do we increase the number of filters in deeper Convolution layers for complex images?

texts: I have been doing this online course Introduction to TensorFlow for AI, ML and DL. Here in one part, they were showing a CNN model for classifying human and horses. In this model, the first Conv2D layer had 16 filters, followed by two more Conv2D layers with 32 and 64 filters respectively. I am not sure how the number of filters is correlated with the deeper convolution layers.


texts:
I know that ARIMA can't detect multiple seasonality, but it is possible to use fourier functions to add a second seasonality.

I need to forecast gas consumption composed by a daily, weekly (week days-weekend), yearly seasonality. Does it make sense to apply three times the STL decomposition by LOESS? The reason is that I applied the fourier method and I have bad results but I don't know if it is only because I applied it wrong.

I'm interested in the theoretical explanation, but here you find also the code:

ARIMA + 2 STL:

b <- ts(drop(coredata(dat.ts)), deltat=1/12/30/24, start=1)
fit <- stl(b, s.window="periodic")
b <- seasadj(fit)
dat.ts <- xts(b, index(dat.ts))

# The weekdays are extracted
dat.weekdays <- dat.ts[.indexwday(dat.ts) %in% 1:5]
dat.weekdaysTS <- ts(drop(coredata(dat.weekdays)), frequency=24, start=1)
fit <- stl(dat.weekdaysTS, s.window="periodic")
dat.weekdaysTS <- seasadj(fit)

arima <- Arima(dat.weekdaysTS, order=c(3,0,5))

With fourier:

dat.weekdays <- dat.ts[.indexwday(dat.ts) %in% 1:5]
dat.weekdaysTS <- ts(drop(coredata(dat.weekdays)), frequency=24, start=1)
z <- fourier(ts(dat.weekdaysTS, frequency=365.25), K=5)
arima <- Arima(dat.weekdaysTS, order=c(3,0,5),xreg=z)


texts: Group neural networks outputs using Keras/Tensorflow

texts:
I am trying to group the outputs of my neural network, in order to have them perform a separated classification.

Let's take the example where the groups are constituted of two nodes and we previously have four output nodes, then the network should look like this :

How can I achieve this using Keras or Tensorflow ? I was thinking that I need to implement a custom layer and do the operation inside of it, but I wonder if there is an easier solution using the Keras Functional API (or lower level Tensorflow ?) :)


texts: Categorical cross-entropy works wrong with one-hot encoded features

texts:
I'n struggling with categorical_crossentropy problem with one-hot encoding data. The problem is in unchanged output of code presenting below:
            inputs = keras.Input(shape=(1190,), sparse=True)
            lay_1 = layers.Dense(1190, activation='relu')
            x = lay_1(inputs)
            x = layers.Dense(10, activation='relu')(x)
            out = layers.Dense(1, activation='sigmoid')(x)
            self.model = keras.Model(inputs, out, name='SimpleD2Dense')
            self.model.compile(
                optimizer=keras.optimizers.Adam(),
                loss=tf.losses.categorical_crossentropy,
                metrics=['accuracy']
            )

Epoch 1/3
1572/1572 - 6s - loss: 5.7709e-08 - accuracy: 0.5095 - val_loss: 7.0844e-08 - val_accuracy: 0.5543
Epoch 2/3
1572/1572 - 6s - loss: 5.7709e-08 - accuracy: 0.5095 - val_loss: 7.0844e-08 - val_accuracy: 0.5543
Epoch 3/3
1572/1572 - 7s - loss: 5.7709e-08 - accuracy: 0.5095 - val_loss: 7.0844e-08 - val_accuracy: 0.5543

Few words about data: 1190 features (10 actual features with 119 categories). The inputs are a dataframe rows with 1190 values per sample. Output is a binary value 0 or 1.
Attempts done before: binary_crossentropy used with satisfying results, however, number of samples is not enough to get good results on validation data. Tried to use different activations and layer sizes.
Main question is why categorical_crossentropy is not working and how to use it in right way.
Also, one concern appears about data representation is it right way to use in one rare row of straightforward one-hot encoded data?


texts: sort occurrence matrix to minimize its spatial variance

texts: Is there any algorithm or an approach that sort occurrence matrix to reduce its spatial variance. I mean by spatial variance moving window variance (n*n moving box).


texts:
I'm participating in a Kaggle contest "What's cooking". The goal is to know wich kind of cuisine we have, depending on some ingredients. So it's a multiclass classification problem. I have an existing model, and I have been trying to improve it for 2 weeks without result. I am using scikit learn, and my existing model is

pipelineLR = Pipeline([
('vect', CountVectorizer(max_df=0.75)),
('lr', LogisticRegression(C=0.4,solver='lbfgs',multi_class='ovr',fit_intercept=True,warm_start=True)),
])

It's really basic. But the problem of this dataset is that we have unbalanced data. Also we have a a huge sparse matrix because of the CountVectorizer. I tried a lot of different model, with and without TF-IDF and this model is the best I can get. I have also confusion matrix, classification report and learning curve for it.

What I can notice is that is have high-variance. I think I can do something by preprocessing the data but I am really stuck. I don't know if my model is good enough to be improved, or if I have to find a totally new approach to it. Also what I can get from the classification report is that the model is working well for classes where I have a lot of data, but it's making a lot of mistakes for classes with less data.

I am very new to data science and I am trying to learn by myself. But I think I really need to have another point of view with this problem, because I can't manage to improve my score.


texts: Inferring composition of a subset of classes of multiclass dataset

texts:
I've come across a problem in assessing the design of a Machine Learning solution I am involved in, which I will describe using a commonly known dataset, the Iris Dataset, and some drawings.
Suppose you have training information about the 3 flower classes contained in the Iris Dataset, which means you can train a multivariate classifier that can distinguish between these 3 flower classes.
However, my dataset 4 classes of flowers, i.e. one extra class that the classifier hasn't been trained on. This dataset I call the "Main" dataset.  The classification is guaranteed to pick one of the 3 Iris Dataset classes, meaning you 100% misclassify any instance's of the 4th class not included in training.

The objective I have is to figure out the Main dataset's composition in the 3 Iris flower classes. I am trying to do this with a second dataset, which I call the "Auxiliary" dataset, containing ONLY the 4th extra class, with a size equal to the expected number of the 4th class's instances in the 4-class dataset. Thus, I get a picture of the possible outcome of using the Iris Classifier on the 4th flower class.

Using the above, is it safe to infer the composition of the Main dataset in the 3 Iris classifier classes, by subtracting the composition of the instances based on the 4th class auxiliary dataset?

"Main" composition in Iris classes = "Main" output composition -
"Auxiliary" output composition

Is there perhaps any literature relative to this?


texts: Graphical Dyslexia -- building intuition for common data visualizations?

texts:
It took me years before I could understand what was going on in a histogram. I had trouble understanding the tails, skewness, and concepts such as "fat tails."

With a lot of practice, histograms have become more intuitive for me.

I believe I have some sort of "graphical dyslexia."

I want to build an inituitive understanding of basic, common types of data visualizations such as:

histograms
scatter plots
dotplots
stem and leaf plots
density plots

By intuitive, I mean that when I glance at one of these plots, I want to figure out the gist of it within a few seconds (rather than stare at it for minutes without anything registering in my mind, as is usually the case for me).

StackExchange: can you recommend to me books/resources that can help me with this?


texts:
I went through adaboost tutorial and below are my simplified understanding:

Sample weight of equal value is given to all sample in dataset.
Stumps are created which uses only one feature from data set.
Using total error and sample weight stump importance is calculated.
Samples weights are changed i.e. samples which were predicted wrongly by stumps with high importance get sample weight increased and rest of the sample weight are decreased in the same order.
The process is repeated till the number of iterations mentioned where sample weight provides path for training.

Does adaboost contain multiple stumps with different splitting value for a single feature?
As mentioned, are the stumps created as first step or is it a continuous process?


texts: Sampling with replacement, specify the probabilities

texts:
I am trying to do sampling with replacement in Scala/Spark, defining the probabilities for each class.

This is how I would do it in R.

# Vector to sample from
x <- c("User1","User2","User3","User4","User5")

# Occurenciens from which to obtain sampling probabilities
y <- c(2,4,4,3,2)

# Calculate sampling probabilities
p <- y / sum(y)

# Draw sample with replacement of size 10
s <- sample(x, 10, replace = TRUE, prom = p)

# Which yields (for example):
[1] "User5" "User1" "User1" "User5" "User2" "User4" "User4" "User2" "User1" "User3"

How can I do the same in Scala / Spark?


texts: Neural Network backpropagation taking forever

texts:
I am constructing a neural network using Sigmoid neurons. My current network is  3 layers of 39 nodes each, using an input of 39 input variables across 10,000 observations.

I am calculating loss by summing the loss in each observation. That being said, I have to numerically calculate $39^2*3*10,000$ weight deltas in the network for every iteration of the code using backpropagation, starting from the output neuron and working backwards. This takes a long time.

Is there a more efficient way to approach backpropagation?


texts: What Are Some Good Ways To Implement The Reptile(OpenAI) Algorithm or Similar Few Shot Classification Models?

texts: What libraries can I use to implement the Reptile algorithm or a similar Few-Shot algorithm in Python? I tried to take some examples and alter them to work for my case, but I couldn't figure out how to make them work for my case. The ML isn't the focus so I don't think writing the algorithm myself is smart.


texts:
I have an assignment where I am trying to find correlations between Lightning Strikes and Telecommunication damage. The two datasets consist of many columns (especially the human-recorded Telecommunication damage one), but let's assume it was something like this :
Telecommunication_Damage_df.columns = (timestamp, geolocation(lat, lon), type_of_damage) and
Lightning_Strikes.columns           = (timestamp, geolocation(lat, lon))
I have done some EDA and cleaned the data, also assigned each lightning strike and each telecommunication damage row to a particular location(cities/areas), but now I am confused as to how I should proceed. Every other Data Science / Machine Learning project I have been involved in was much more direct and usually had one training/testing dataset whereas with this one I am stuck as to how I should proceed, is there a model that could help? Is there a methodology I am unfamiliar with?
I tried following this crime/geolocation tutorial (https://www.kdnuggets.com/2020/02/introduction-geographical-time-series-crime-r-sql-tableau.html) but it's not exactly the same, because there is one dataset being used, crimes, if a crime occurred then that's it, whereas here if a lightning strike occurred that doesn't necessarily mean a telecommunication problem was found and vice versa.
I know it's a bit vague, but I've been stuck for quite a while, and I was hoping that someone could guide me to any direction, because at the moment I am idle.


texts: Linear Regression in Python using gradient descent

texts:
I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ and the model from the normal (closed-form) equation is able to produce a training RMSE of $~2.3$. I am looking for ways in which I can improve my implementation of the gradient descent algorithm. Below is my implementation:

My gradient descent method looks like this:
$\theta = \theta - [(\alpha/2N) * X (X\theta - Y)]$ where $\theta$ is the model parameter, $N$ is the number of training elements, $X$ is the input and $Y$ are the target elements. $\alpha$ is the step size.

def gradientDescent(self):
    for i in range(self.iters):
        # T = T - (\alpha/2N) * X*(XT - Y)
        self.theta = self.theta - (self.alpha/len(self.X)) * np.sum(self.X * (self.X @ self.theta.T - self.Y), axis=0)
    return errors

I had set the $\alpha$ as $0.1$ and number of iterations as 1000. The gradient descent reaches convergence at around 700-800 iterations (checked).

My error function is like:

def error_function(self):
        # Error function: (1/2N) * (XT - Y)^2 where T is theta
        error_values = np.power(((self.X @ self.theta.T) - self.Y), 2)
        return np.sum(error_values)/(2 * len(self.X))

I was expecting the training error from the gradient descent and the normal equations would turn out to be similar, but they have a bit of a huge difference. So, I wanted to know whether I am doing anything wrong or not.

PS I have not normalized the data, yet. Normalizing leads to a much lower RMSE (~$0.22$)


texts:
This question calls people to share their personal experiences with keras_self_attention module.
I also summarized the problems I encountered and the solutions I found or received from answers.

Background
I am building a classifier using time series data. The input is in shape of (batch, step, features).
The flawed codes are shown below.
import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()
model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1],X_train.shape[2])))
model.add(Bidirectional(LSTM(units, dropout=dropout, recurrent_dropout=recurrent_dropout)))
model.add(SeqSelfAttention(attention_activation='sigmoid'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

history = model.fit(X_train, y_train,
                        batch_size=batch_size,
                        epochs=epochs,
                        validation_data=(X_val, y_val),
                        verbose=1
                        )

yhat = model.predict_prob(X_test)

Problems
1. The model raised an IndexError at the SeqSelfAttention layer.
Solution: This problem was because the `return_sequences` in the last `LSTM` layer was not set to `True`

2. The model raised a ValueError before the Dense layer due to the shape incompatible problem.
Solution: The solution is to add a `Flatten()` layer before `Dense` layer.

*3. How do Muplicative attention and Multi-head work?
*4. Does stacking multiple layers of self_attention(SA) work for improving accuracy? Should it be like LSTM+LSTM+SA+SA or LSTM+SA+LSTM+SA?


texts: keras custom layer weights with variable input size

texts:
I am trying to create a custom layer in keras which gets 3D Tensors as input. However, the first dimension of the 3D tensor is variable eg - [12,128,157] , [20,128,157], [8,128,157]. The custom layer will have weights whose shape is dependent on this variable dimension -
self.kernel1 = self.add_weight(shape=(self.output_dim,input_dim),
                                   name = 'kernel1',
                                   initializer=self.kernel_initializer,
                                   regularizer=self.kernel_regularizer,
                                   constraint=self.kernel_constraint,
                                   trainable=True)

where input_dim is the variable shape ( 12,20,8 etc) in this case.
Sample Model:
inputs=Input(shape = (None,128,157))  # tensor of shape (?,?,128,157)
x= CustomLayer(output_dim =10)(x)

This throws an error:
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

Is there any workaround to initialize trainable weights with variable size?


texts: Force selecting samples in majority class with random forest

texts:
Context: I have some data to fit a random forest classifier (binary output) with 1 being a very rare event. In particular, in my training set, there are only 614 1's out of 29400 points. I am using sklearn RandomForestClassifier.

I am setting class_weight = balanced to prevent the model from simply predicting 0 to every case. And it is working great!

However, there is also a small group within the 0 class (maybe only 20 - 30 cases) (Edit: actually about 300) that I want my model to capture. I believe because of the sampling natures when building trees, these class are not chosen very often. Is there a known way to solve this problem?

My thought:

Add an additional filter after the RF. Trouble is, it's hard to find some easy categorization methods for these 20 -30 negative cases.
Force the RF to include these samples when building the trees. Hence this post...

Thanks.


texts: Machine learning to predict apps (recomendation)

texts:
I am currently collecting mobile device data about user location and the time at which an app is being used for a cohort of users and apps.  I am trying to predict which apps are likely to be used at a given time by a given user.

Which model should i use in order to predict the apps that are likely to be used by a particular user? For example, should I collect data for 5 days a week then use the 3 days for training and then the other 2 days for testing? Given location, time of day, and day, what model should I use to predict which app is likely to be used?


texts:
is there a way to take the average of two classification report?

so if I have this two reports

              precision-recall  f1-score   support

 downstairs    0.82      0.84      0.83       146
 jogging       0.99      1.00      1.00       146
 sitting       0.97      0.93      0.95       149
 standing      0.91      0.89      0.90       148
 upstairs      0.80      0.86      0.83       146
 walking       0.90      0.87      0.88       148

accuracy                            0.90      883
macro avg       0.90      0.90      0.90      883
weighted avg    0.90      0.90      0.90      883

AND

           precision-recall  f1-score   support

 downstairs    0.75      0.70      0.72       292
 jogging       0.99      0.97      0.98       291
 sitting       0.92      0.94      0.93       299
 standing      0.93      0.76      0.84       296
 upstairs      0.68      0.77      0.72       293
 walking       0.77      0.87      0.82       295

accuracy                           0.84      1766
macro avg       0.84      0.84     0.84      1766
weighted avg    0.84      0.84     0.84      1766

how to take the average values?


texts: How to determine feature importance in a neural network?

texts:
I have a neural network to solve a time series forecasting problem.  It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features.  The performance of the model is average and I would like to investigate whether adding or removing features will improve the performance.  I have constructed the neural network using keras.

The features I have included are:

The historical data
quarterly lagged series of the historical data (4 series)
A series of the change in value each week
Four time invariant features tiled to extend the length of the series. (another 4 series)

I am aware I could run the model many times changing the combination of features included each time. However, along with tuning the hyperparameters (for it might be that 8 features works really well with one set of hyperparameters but not with another set) this is really a lot of possible combinations.

Is there any separate way that I can use to guage if a feature is likely to add value to the model or not?

I am particuarly concerned that I have four time-invariant features being fed into the model which is designed to work with time varying data and I would like a way to measure their impact and if they add anything or not?


texts: Keyword Extraction from a text followed by a key value using tensorflow

texts:
I have a pdf file that contains information . I would like to extract few key terms/phrase along with a value for example (current balance : CHF (swiss francs) 1,000)

I can convert pdf file to text using pdfminer . But how i can extract the above keyword using tensorflow text classification or other methods.

Can anyone suggest how i can start with it? I haven't come across a single example with tensorflow. There is a question here (Keyword/phrase extraction from Text using Deep Learning libraries
). but this doesn't help me to understand much as I am a beginner.

I used NLTK to tokenize words and read no of words by using

import re
text = re.sub(r'[_"\-;%()|+&=*.!?:#$@\[\]/]', ' ', f)
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

tokens = word_tokenize(text)
stop_words = set(stopwords.words('english'))
words = [w for w in tokens if not w in stop_words]
freq = nltk.FreqDist(words)
for key,val in freq.items():
    frq = str(key) + ':' + str(val)

So i have this text that i could like to extract the key phrase current balance : CHF (swiss francs) 1,000 along with the value using Neural Network. How can i do that? I came across many posts but couldnt get what i need.

PS: every time i upload a new file, i want my algorithm to find out the same keyword and use the value next to it.


texts:
I have a csv file with around 130 columns and 6000 rows

what is the best way to import them into python, so that I can later use them in a classification algorithm(columns are the labels and rows are individual samples)


texts: Weighing each label in multi-label classification

texts:
If, in addition to predicting labels using a multi-label classifier, I'm interested in predicting the weight of each label, what approach should be taken? To give an example, let's say I'm trying to predict movie genres from their plot and for a movie like Terminator the classifier predicts ['Sci-Fi', 'Action'], then is it also possible to estimate the proportion of those genres in that movie, like it's 70% Action and 30% Sci-Fi?

Multi-label classifier does give the probability for each class; is it a good idea to just normalize those probabilities and use them as the weights?


texts: I'm new to data science field and interested in performing prediction using clickstream data. In Practical Lessons from Predicting Clicks on Ads at Facebook paper section 3.1, a method called Decision Tree Feature Transforms confused me a lot. After searching around, there is no website / book / paper describing about this method. Can anyone give me link references describing this method in detail?


texts:
Is there a recommended approach for storing processed data for testing new data products?

Basically, I'd like to have a system where a data scientist or an analyst could think of a new data product to present to users, do the data processing to create it, and then put it in a data store that our application can then access easily.

What I'm not sure about is what kind of data store would be good for this type of "testing" use case.  Since it would need to be flexible enough to handle different types of data products, like aggregates, windowed data, etc.  And ideally it wouldn't require a huge instrumentation process to try out new things.


texts: How to interpret classification report of scikit-learn?

texts:
As you can see, it is about a binary classification with linearSVC. The class 1 has a higher precision than class 0 (+7%), but class 0 has a higher recall than class 1 (+11%). How would you interpret this?
And two other questions: what does "support" stand for? The precision and recall scores in the classification report are different compared to the results of sklearn.metrics.precision_score or recall_score. Why is that so?


texts: What is the appropriate name for this dataset?

texts:
I'll assume most of us know about cross-validation and are accustomed to using Training and Test sets. However, I've also seen references to a third set that you use to test a completed model. It's the test to see how your model does against a dataset that it hasn't seen before; helps determine if your model has been overly-fit to the training data.

I have seen this dataset referred to everything from "the third dataset", "the wild dataset", "the real-world dataset". So what is the appropriate name for this third dataset?


texts: Is fitting two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set more performant than one with 1000?

texts:
If I fit two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set, would it have better results than fitting a RandomForestClassifier with 1000 trees and use it to get test set probabilities?
As these algorithms are random based I would say that their performance should be roughly the same?
I am okay with some math to prove it, or any other way that might prove it.


texts: Reconstructing an audio signal from its Mel-scale spectrogram using an autoencoder

texts:
I'm looking for some papers/references that attempted to reconstruct audio signals from their Mel-scale spectrograms using an autoencoder or other neural network.
I am thinking of training the autoencoder by computing the Mel-scale spectrogram of the input audio signal, passing the Mel-scale spectrogram through the autoencoder such that the dimensions of the output layer exactly matches the dimensions of the input audio signal, and then updating the parameters of the autoencoder based on the MSE loss between the output layer and the input audio signal.
I would also appreciate any suggestions.


texts: Is image sharpening a good idea for data augmentation?

texts:
I'm training segmentation networks and while the dataset is somehow decent (~5k images) I wanted to augment it, so far I'm trying:

RandomFlip
RandomRotate
RandomBrightness changes
RandomShadows

Due to constraints of the problem I can't do random crops or shifts. Other than those augmentations I was looking into image sharpening, and was wondering if it could be a good candidate for dataset augmentation. I could find it in some web articles and many augmentation projects on Github, but I can't find any solid papers that refer to it as a possible augmentation technique. Does anyone have some experience/tips on the matter?


texts: Can 1D-CNN method apply to real-time time series classification?

texts:
So I got an EEG dataset with shape (data points, 19), each row's shape (1,19) represent 1 second of EEG.
I read much research on EEG classification that used many Deep Learning method and 1D-CNN is one of that.
My question is as the input of the 1D-CNN must have multi-row data, ex (50,19) for my dataset so it can filter a input matrix. But I want to predict new data row by row ((1x19) shape), can 1D-CNN use this input for predict new data?


texts: How to deal with word length variability while using char-level one-hot encoding?

texts:
I am trying to run some experiments on internal word structure of morphologically rich language (Russian).

Each valid character is 1-hot encoded and then fixed-length vectors are concatenated to produce word vector:

 c             a             t

(0... 1... 0) (0... 1... 0) (0... 1... 0)

This approach produces vectors of different length for words of different length. To unify them I have applied zero-padding:

c a t 0 0 0 0 0 0 0 0

k i t t e n 0 0 0 0 0

t r a n q u i l i t y

h a p p i n e s s 0 0

Does this approach sounds valid? What are best practices on the described issue?


texts: Based on transformer, how to improve the text generation results?

texts:
If I do not pretrain the text generation model like BART, how to improve the result based on transformer like tensor2tensor?
What are the improvement ideas for transformer in text generation task?


texts: How to combine dictionary of word-forms into a single list without duplicates

texts:
https://github.com/gutfeeling/word_forms

From the URL, I find out how to convert word into all the forms(verb, noun, adjective). I need that after conversion, list format.

e.g:

from word_forms.word_forms import get_word_forms

a=get_word_forms("review")
print(a)

output:

{'n': {'reviewer', 'reviewers', 'reviews', 'review'}, 'a': set(), 'v': {'reviewed', 'reviews', 'reviewing', 'review'}, 'r': set()}

my expected output is:

['reviewer', 'reviewers', 'reviews', 'review','reviewed', 'reviews', 'reviewing', 'review']

how this is possible?


texts: Expected value based on periodical interval measurements

texts:
Suppose you want to measure a process that goes on/off every 100ms +- rand 3000ms.
Your tools can do probes ones in >=1500ms.
How to get expected values of:

how long the process is off
how often the process goes off


texts: Examples of Open source huge business intelligence datasets

texts:
I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that computationally efficient. However, someone pointed out it need not be the case. Many of the training datasets are smaller in size.
I want to ask whether or not using a huge dataset for training/inference on traditional ML and data science algorithms makes sense? Is it a proper use case? If yes, can someone please point me towards such a huge dataset available online?


texts: Mapping sequences of different lengths to fixed vector - Python

texts: I am trying to make a chatbot using a deep neural network in python using keras. The problem I am having is that for the deep neural network to work, the input dimension has to be fixed. So my question is, how can I map a sequence of words (sentence of different lengths) to a fixed vector so I am able to feed this through a deep neural network?


texts: I still don't know how deconvolution works after watching CS231 lecture, I need help

texts:
https://www.youtube.com/watch?v=ByjaPdWXKJ4 This is the time-stamped video of "deconvolution". I can understand normal convolution but not so much with upsampling convolution. In the video he explained that you plop down the filter and use each individual scalar as the weight to apply to each value in the filter. I am having a hard time understanding how he got an output shape of 4x4? Is there a special formula for calculating the output for "upconvolution"? I am also confused on the padding and how it affects upconvolution?

My crappy paint image


texts: Multiclass classification in a balanced dataset with one high-priority label

texts:
I have a balanced dataset for a multiclass classification problem with one high-priority label (this ought to be classified properly at all costs). How do I go about creating a workflow for this problem? What specific feature engineering/selection methods and classifiers should I be considering for this problem?

To be more specific, the data I'm dealing with (including labels) is completely anonymized, so I don't have a clue as to what it actually stands for.

Some approaches I am considering -

Creating synthetic data points for the priority label through oversampling.
Creating a highly non-linear model for prediction as accuracy is very important.

Any help is much appreciated!


texts: Do I have to standardize my new polynomial features?

texts:
I have a vector X with n features previously standardized.

If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new features after the computing ?

Because knowing that my original features are already between 0 and 1, the new polynomial features will also be between 0 and 1. But the higher the degree is, the higher the "ratio" between the original one and the polynomial one will be.

Also,  would that be better to do the square computing on the values of the non standardized features, and then standardize these new features ?

By the way, when I say "standardization", I m thinking about substrat each value by the mean and divide it by the std.

Thanks in advance,

Regards.


texts: can accuracy rise while precision and recall drop?

texts: I am working on a model and running some experiments, I see that under some configurations, The accuracy rises while the recall and precision are much lower, what is the mathematical explanation? is the TN rate dropping?


texts:
I am working on a data entry task with approximately 6000 entries to go over.

The source comes in the form of a string and can look something like this:

  Air Canada B737 FFS

From this I can extract the following information:

  Company: Air Canada

  Model: B737

  Technology: FFS

For my initial plan of attack I iterated over the source strings using Regular Expression to extract as many keywords as possible, the problem is there are so many different Companies, Models and Technologies in my source, and not every source is as clean as the one above so it would take forever to write out all the possible regular expressions. Also, there is usually text within the source that isn't important, a red herring if you will.

As a result I have manually filled out 2000 of the entries myself.

My question is, do you think I could train a ML model with my manually filled dataset to do the rest of this task for me? I would consider myself to have intermediate skills in python, but what type of ML algo should I use for this task? Any help to point me in the right direction would be very much appreciated!


texts: Are there studies which examine dropout vs other regularizations?

texts:
Are there any papers published which show differences of the regularization methods for neural networks, preferably on different domains (or at least different datasets)?

I am asking because I currently have the feeling that most people seem to use only dropout for regularization in computer vision. I would like to check if there would be a reason (not) to use different ways of regularization.


texts: Time series forecast for everyday for till a distant future

texts:
I have time series data for every single day from last 5 years with seasonal variation and a general increase in trend. This is what my data looks like:

And I am trying to predict for every single day for 4-5 years in future. Approaches I have used currently:

LSTM, GRU (but they are extremely prone to overfitting ad I am trying successive predictions which results in massive error accumulation and often flat over time for smaller lookbacks and sine wave driven patterns for long lookbacks.
SARIMAX and other seasonal forecast models but they tend to scale very bad for a large period of predictions. Mostly they eventually result in a curve that follows trend(increase/decrease) but values are almost linear or unrealistically close to each other.

What I am currently thinking of:
Separate same day of each year and fit a model to those values and then have successive predictions for the same day for subsequent years. Thus having 365 different models and then concatenating the resultant values to have the whole year and future years predictions. This way I'll perfectly preserve the seasonality and yet accumulation errors will be minimized. So my question is:
Is this a good idea (and if it is, is there a method to do it feasibly without assessing each of the 365 models) or I should try any other approach?


texts: Train LSTM model with multiple time series

texts:
I am predicting energy usage for a bedroom within a school residential building with date, temperature, and humidity as input features, using 7 time-steps and predicting for one-day (one-timestep).

I am able to predict for one bedroom. The building has about 100 bedrooms, all more or less the same layout and occupancy, so the data for all these 100 bedrooms have the same date range as the first bedroom and same humidity and temperature levels on the same day. Now I want to train my model with all the 100 bedroom's energy data so that the model captures all 100 rooms energy patterns.

Then, I want to predict what the energy usage is for any given bedroom (not a specific bedroom), using this model that has been trained with all 100 bedrooms. In other words, the business question is "What is the expected energy usage for a bedroom within this residential school building?"

I would like to know how I can train all these 100 rooms into one LSTM model, given that LSTM is a time series model, and 100 rooms mean I have 100 times series. I cannot simply concatenate the datasets of all rooms.

How would I structure my data and run it in the LSTM model?

I would appreciate if someone could show me an example of how it could be done.


texts:
Given a time series problem,

Should ACF and PACF be done before or after differencing that make the time series stationary?
If ACF and PACF has shown different results, should the number of orders of AR/MA follows ACF or PACF?


texts: How to train a Model which can check whether image is from existing classes or not

texts:
I have given with a image dataset of 1000 classes , each class has 100 images. Now My requirement is to train a model which will take a image as input, and it should answer whether the image is present or not, there is no need to specify which class it is.. It just has to say yes or no.

I thought of using a CNN with a sigmoid function at last layer, but not able to connect the dots.


texts: What are the best methods to analyze the results of a pairwise comparison survey?

texts:
I have recently conducted as survey where I was comparing a range of different images. Due to screen size constraints I decided to display these images pairwise and ask the participants to pick one of the images.
There were 4 images being compared and therefore 6 possible pair combinations. My data now looks something like this.

A vs. B - 6 chose A, 3 chose B
A vs. C - 2 chose A, 6 chose C
A vs. D - 8 chose A, 1 chose D
B vs. C - 2 chose B, 7 chose C
B vs. D - 6 chose B, 3 chose D
C vs. D - 8 chose C, 1 chose D

How do I determine which option is the favourite? Can I make claims such as A is preffered by 60% of participants etc?


texts: built-in cov in pandas DataFrame results ValueError array is too big

texts:
So, I have a dataset with 39.949 variables and 180 rows. dataset is successfully
saved in DataFrame but when I try to find cov() it result an error.
here is the code

  import pandas as pd
  cov_data=pd.DataFrame(dataset).cov()

Here is the error

File "/home/syahdeini/Desktop/FP/pca_2.py", line 44, in find_eagen
cov_data=pd.DataFrame(data_mat).cov()
File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 3716, in cov
baseCov = np.cov(mat.T)
File "/usr/lib/python2.7/dist-packages/numpy/lib/function_base.py", line 1766, in cov
return (dot(X, X.T.conj()) / fact).squeeze()
ValueError: array is too big.


texts: How to build a webscraper for indeed.com

texts:
How can I scrape a URL such as: https://www.indeed.com/jobs?q=data+scientist+%2420%2C000&l=New+York&start=10
where I want to get the following parameters: job title, company name, job description, city, location, and salary.
What are the basic steps I need to take to build a scraper?


texts: How to perform a running (moving) standardization for feature scaling of a growing dataset?

texts:
Let's say that there is a function $r$
$r_n = r(\tau_n)$,
where $n$ denotes a so-called time-step of a system with an evolving state. Both $\rho$ and $\tau$ should equally influence $r$, and should therefore be scaled. The problem is, the sequence $(\tau_1, \tau_2, \dots, \tau_n)$ grows in time because $n$ grows.
How to perform a running standardization of $(\tau_1, \tau_2, \dots, \tau_n)$. Running mean is relatively simple to express:
$\text{mean}(\tau)_{n+1} = \frac{1}{n+1}\left[\tau_{n+1} + N \text{mean}(\tau)_n\right]$
where $\text{mean}(\tau)_1 = \tau_1$.
The standardization requires
$\tilde{\tau}_n = \dfrac{\tau_n - \text{mean}(\tau)_n}{\sigma(\tau)_n}$
where
$\sigma(\tau)_n = \sqrt{\dfrac{1}{n-1}\sum_{i=1}^{n}[\tau_i - \text{mean}(\tau)_n]}$ (1)
is the standard deviation of $(\tau_1, \tau_2, \dots, \tau_n)$.
Question: is there an expression for a running standard deviation? Online I've only found links on stack overflow and Matlab functions, but I am not sure which algorithm is best suited for feature scaling. By running (moving) I mean not having to store $(\tau_1, \tau_2, \dots , \tau_n)$ to calculate (1), instead update it incrementally.


texts: Predict next month's loan balance from historical data

texts:
I work for a bank. Every month I get a list of 10,000 customers of the bank with the outstanding loan balance for each customer. I add them up and report the total outstanding balance every month. I have data for past 24 months.

I am asked to give a rough prediction of the next month's total outstanding balance. I am not a statistician, hence need some help. So far ,I have below ideas, can someone comment on pro/cons of each?

Compute the average monthly % growth over last 6 months on Total balance; Multiply current balance by % growth  to predict next month balance
Do the same at an individual customer level, that gives us 10,000 predicted balances for next month. Add them up to get total balance
Are there any time series techniques I can use here?
Can I find the answer by some simulation techniques?


texts: Regarding Keras mean_squared_error losses

texts:
I am trying to do a RandomizedSearchCV for a simple regression task. Essentially two inputs to one output.
Upon inspecting the resulting models, it appears that the 'best' model has a mean_squared_error of around -8.3, while there were other 'non-best' models which had mean_squared_error close to 0 (i.e. around 0.004).
In this case, may I know if the Keras optimized model selection is correct? Why does it select one with mean_squared_error not close to zero?
Shouldn't the best model give MSE close to zero? I understand that the optimization seeks to minimize loss, but how can I interpret this?


texts: What is the best way to integrate our classes with mscoco classes?

texts:
I am using object detection api with efficientdet-d0.
I am interested with one class in MSCOCO (say class C), and two classes outside MSCOCO (say  class A and  class B), and I have the training set for class A and class B. How should we structure the config file to allow the training of a model with (class A, class B and class C) - where classes A and B make use of the training data, and class C only makes use of pretrained weight with MSCOCO?


texts: How to deal with classification problem where labels are non uniformly distributed?

texts: I have a data set which has around 1000 samples and are divided in 4 groups - A, B ,C , D. The problem I am facing is that there are very high number of data sample which have B and C s output. They outweigh other two by factor of 3:1. Due to this most of the classification problems are producing very inefficient results with results from other two classes being mapped to A and B quite often . Is there any way to deal with it?


texts: Understanding the ResnetGenerator in the CycleGan Model

texts:
In their famous research paper on CycleGans, the authors implement - well, a CycleGan.
There are two discriminators and two generators for the CycleGan. Now, they also provide their neural networks code. I am particularly interested in the generator(s), which they implement here:
So they use a ResnetGenerator, but I'm afraid I do not really understand it yet (cf. lines 119-159 and 315-373).
For the generator, why do we have both downsampling (Conv2d) and upsampling (ConvTranpose2d) layers? I generally know it like this for the generator that the generator only uses ConvTranpose2d layers, where the input is noise sampled from a uniform or Gaussian distribution..
That's why I am confused..


texts: Regression and Classification, which is better in financial market price prediction?

texts:
I want to use a model to trade in finanical market.
which i have several features, like macd, rsi, or other common features.
and my target is to make a tradeable predict in every time point.
so my target can be:

yield in a fixed time laster, like, 30 min. yt = close(t+ws) - close(t)
futures price direction, which only can be 1(price up in the future) -1 (price down in the future)

these are difference between regression and classification.
which one you think is better, and, any suggestions about this problem?
Thanks


texts: How to determine which features matter the most?

texts:
I have a large dataset that consists of search results of loans. Someone would input their details like income etc and the results would include a bunch of loans from different companies and different loan types (so there can be more than 1 loan per company).
The dataset consists of every unique search and all the corresponding results. I also have a column that shows which loan has been selected at the end by the user per each search.
I am looking to find out which features of the loan were most important to users, i.e. try to predict what loan the user will select depending on his/her inputs.
What ML model could I use for this? I am unsure how to approach the problem.


texts: Classification of articles fetched from the web as describing stock (trending) increase or decrease

texts:
I'm building an AI that works with natural language processing integrated in my school project application (an app dedicated to traders). I'm wondering what set of word is used most to express negativity (or positivity) in stock market? If a certain stock is decreasing in value (or increasing) what words are most likely to be appear on the press?

Ultimate Goal: Classifying an article fetched from the web as describing stock increase or decrease.

Are there any public free data sets tackling this subject?

Else, is there a data set that classifies sentences as trending or not trending?

Any suggestion, help or even collaboration would be very appreciated!


texts: Random Forest where objective is not to replicate past classifications

texts:
Normally when I train a random forest to classify observations into multi-class buckets the objective is to correctly predict which bucket an observation will fall into based on historical (training) data.

Instead, I want to train the random forest to classify observations on some other criteria, such as profit maximization. Is this possible?

Here's an example:

Imagine that we have data on Dropbox subscriptions over the past 1 year. Some Dropbox leads (potential customers) had 1 of 3 possible coupons, other leads did not have a coupon.

In this scenario I want to determine which (if any) coupon I should should offer to a lead to maximize net revenue, considering their likelihood of purchasing a subscription, their predicted retention (# of months they will continue their subscription) and the effect of the coupon on purchase price.

Theoretically some leads who are likely to purchase a subscription and be retained will not need a coupon to do so. Others might produce higher net revenue from a coupon for "$5 off per month" or "first month free", etc.

I presume the dependent variable should still be coupon type, i.e.

Y = No Coupon, Coupon A, Coupon B, Coupon C, Coupon D

Is it possible to make the random forest work in this way? You can consider this question language-agnostic, though if I have a choice I will try to do this in R.

I know that in the case of eXtreme Gradient Boosting in either R or Python I can specify a custom objective function.


texts: Is there an API to extract questions and labels from yahoo answers?

texts: I want to work on a ML project which involves the family and relationship category from yahoo answers. I want to extract questions from yahoo answers (only the title) with the label as the category it is in on yahoo answers. I want to use it in csv form to train my naive bayes model. Is there an API or something similar to do so ?


texts: Not able to restore attention model properly

texts:
I am referring this article on building an attention model using tensorflow.

I am trying to train a similar model on my dataset using google colab. Due to the session limit of colab and my large dataset, I need to save the model state and restore it to resume training.

However, I am not able to restore the model upon saving the parameters. I have saved the input and target tokenizers, model checkpoint and even the input and output tensors. However, every time I use checkpoint.restore and resume training the model it resumes training with a high loss(equal to random weights).

I always test my model before saving using the translate function on some test data and it generates a one line summary. However, when I restore the model and run some sample data on the translate function, I only get a single  tag as output (as if it is a newly initialised model).

Here is my code

checkpoint_dir = './training_checkpoints'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt")
checkpoint = tf.train.Checkpoint(optimizer=optimizer,
                                 encoder=encoder,
                                 decoder=decoder)

manager = tf.train.CheckpointManager(checkpoint, 'checkpoint_dir', max_to_keep=1)

The training step is

EPOCHS = 50

for epoch in range(EPOCHS):
  start = time.time()

  enc_hidden = encoder.initialize_hidden_state()
  total_loss = 0

  for (batch, (inp, targ)) in tqdm(enumerate(dataset.take(steps_per_epoch))):
    batch_loss = train_step(inp, targ, enc_hidden)
    total_loss += batch_loss

    if batch % 100 == 0:
      print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,
                                                   batch,
                                                   batch_loss.numpy()))
  # saving (checkpoint) the model every 3 epochs
  if (epoch + 1) % 3 == 0:
    manager.save()

  print('Epoch {} Loss {:.4f}'.format(epoch + 1,
                                      total_loss / steps_per_epoch))
  print('Time taken for 1 epoch {} sec\n'.format(time.time() - start))

I restore by doing

checkpoint.restore('ckpt-ckptnumber.index')

I save the tokenizers (both input and output) using pickle

with open('inp_tokenizer.pickle', 'wb') as handle:
      pickle.dump(inp_lang_tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)

I save the tensors using numpy.save()

np.save('X.npy', input_tensor)


texts: TF-Ranking transform data to ELWC - ExampleListWithContext

texts:
I have read all the guides, videos, and everything, but I have no idea how to convert my feature set to an ELWC datasheet format for TF-Rank ListWise problem. There is no description of this structure.

For example, a students profile is:

Student ID  age  grade  math%   physics%   english%  art%  math_competit  language_competit Rank
  14588     16    k12     98      67         88      100   first_place        very_good       5

If I have 20 students in the same class, how can I transform this data to be able to make a listwise prediction for every grade ( theoretically in every grade has 3 class with 20 students)


texts: does xgboost's eval_metric changes the loss function being optimized?

texts:
I'm using xgboost with the reg:logistic objective. As far as I understand, that means that I'm trying to optimize the log-likelihood loss function of my model, and that each boosting step would be in the gradient direction of this function (w.r.t all the examples I've given).

Does changing the eval_parameter to auc affect any of the above? Or is it just affecting what's measured on my validation set examples every 10 steps of boosting?


texts: Problem reconstructing Keras's IMDB reviews

texts:
Keras's sample IMDB reviews are represented as a sequence of word indexes. However,  a dictionary is provided that matches the index to the word I'm trying to use this to reconstruct the text of a review, but I'm getting gibberish. Here's my code:

from keras.datasets import imdb

(x_train, y_train), (x_test, y_test) = imdb.load_data()
ind = imdb.get_word_index()

def find_word(val):
    for key, value in ind.items():
        if value == val:
            return key

rev = x_train[0] #which review to look at
rev_recon = [] #hold the reconstructed review

for i in range(len(rev)):
    word = find_word(rev[i])
    rev_recon.append(word)

print(' '.join(rev_recon))
>>>the as you with out themselves powerful lets loves their
becomes reaching had journalist of lot from anyone to have
after out atmosphere never more room titillate it so heart
shows to years of every never going villaronga help moments

I thought the sequence of word indexes was supposed to represent the sequence in which the words originally appeared in the review. Am I missing something? Here's my code:


texts: Statistical Significance in Pairwise Ranking Algorithm

texts:
Can anyone recommend an algorithm/toolkit to rank items that have been rated in a hot-or-not style that gives statistical significance?

For example, out of a set of N images, two images are shown to users and the user picks the one that appeals most to him/her. The final output is a ranking: worst to best: image1 image5 image3 image8 etc.

I've come across the Bradley-Terry model with maximum-likelihood inference but I haven't found a toolkit that shows statistical significance of the rating.

Any toolkits in python especially would be much appreciated (though R is fine too).


texts:
I have a table in R. It just has two columns and many rows. Each element is a string that contains some characters and some numbers. I need number part of the element. How can I have number part?
For example:

    INTERACTOR_A INTERACTOR_B
1          ce7380       ce6058
2          ce7380      ce13812
3          ce7382       ce7382
4          ce7382       ce5255
5          ce7382       ce1103
6          ce7388        ce523
7          ce7388       ce8534

Thanks


texts: JupyterNotebook exporting graphs as SVG unintentionally cropped

texts:
Within JupyterNotebook the graphs are perfectly rendered and look great, however I cannot figure out why it's (unintentially) cropping my graphs during the exporting, such that the axis descriptions are partly missing in the SVG file. Could anyone give me pointer why that might happen?
Example of a confusion matrix:
confusion_matrix = metrics.confusion_matrix(y_test, test_pred_decision_tree)
matrix_df = pd.DataFrame(confusion_matrix)
ax = plt.axes()
sns.set(font_scale=1.3)
plt.figure(figsize=(50,25))
sns.heatmap(matrix_df, annot=True, fmt="g", ax=ax, cmap="bone")
ax.set_title('Confusion Matrix')
ax.set_xlabel("Predicted label", fontsize =15)
ax.set_xticklabels(labels)
ax.set_ylabel("True label", fontsize=15)
ax.set_yticklabels(list(labels), rotation = 0)
fig = ax.figure
fig.savefig('confusion_matrix.svg', format="svg", facecolor=fig.get_facecolor()

Confusion Matrix exported result


texts: back propagation through time derivation issue

texts:
I read several posts about BPTT for RNN, but I am actually a bit confused about one step in the derivation.  Given

$$h_t=f(b+Wh_{t-1}+Ux_t)$$

when we compute $\frac{\partial h_t}{\partial W}$, does anyone know why is it simply

$$\frac{\partial h_t}{\partial W}=\frac{\partial h_{t}}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial W}$$

not

$$\frac{\partial h_t}{\partial W}=\frac{\partial h_{t}}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial W}+\frac{\partial h_t}{\partial f}h_{t-1}$$

?

What I mean is, since both $W$ and $h_{t-1}$ depends on $W$, why is the second term in the expression above missing?

Thank you!


texts:
In the dataset I have a continuous variable AGE and categorical variable AGE_CATEGORY as well. They are both highly correlated.

Which method should I use to identify the feature to be dropped AGE OR AGE_CATEGORY?


texts: I have a dataset of computer systems with system characteristics as parameters. I want to rank best systems based on these parameters. The system works on a recurrent neural network model with the inputs of all previous transactions considered while choosing the current output. To my understanding, LSTM is something that is commonly used for such usecases but I heard it has limitations when I want to dynamically add new systems.  Are there any better alternatives?


texts: What algorithms are good to predict next numbers?

texts:
Let's consider we have several hundreds of numbers like ( 1, 2, 5, 8, 7, 15, 19, 8, 4, 6, ...) those are closed numbers of a stock on consecutive days for example. I like to know what algorithms are good to predict next numbers?  Is it possible to find a relationship between this sequence of numbers or we need extra input data to make a prediction?

I am completely new on this and seeking some ideas and guidance for a new comer in data science field(although I have some experience in machine learning and image and voice recognition).


texts:
I have performed logistic regression. And I am getting an accuracy of 77% with my current model. I divided my training set into cross validation set and train set. And I plotted a learning curve (graph of training examples vs cost function for train set and cost function of cross validation set). My learning curve is shown below -

What does it indicate? Since both the curves have a very less difference one is (0.51 and other in 0.52), Is my model biased, or is it correct?


texts: Including identifier in machine learning model as feature vs separate model for every identifier

texts:
I am new to machine learning and i am building a model to predict number of customers for the model branch at specific  hour/season/other feature.

I know it will be bad idea to pit id(branch_id in my case) into model but customer count in this case hugely depend on which branch it is so i cannot exclude it.

I can think of two solutions, i am not sure which one is right and what is the best practice.

Create dummy variable(one hot encoding to avoid wieghing one id more than other) for all branch ids,but since i have 600 unique branch ids my features will go up-to 600+rest_of_features.
Learn a separate model for each of the branch(600 models), i am not sure if it is right approach and also i am not very familiar with this approach and it will be very time consuming.

Looking for the suggestion

Example of the data is below

    +-----------+------+-----------+-----------+-------------------+
    | branch_id | hour | feature_2 | feature_3 | Count of customer |
    +-----------+------+-----------+-----------+-------------------+
    |         1 |   12 |        .. |        .. |                19 |
    |         1 |   01 |        .. |        .. |                25 |
    |         2 |   23 |        .. |        .. |                14 |
    |         2 |   01 |        .. |        .. |                 5 |
    +-----------+------+-----------+-----------+-------------------+


texts: Periodical loss increase in the learning curve

texts:
I am training a transformers-based machine translation (NMT) model.
The size of the parallel corpus is 4.5 million sentence pairs in two languages. What I am observing in the learning curve is that model is converging; however, I have some strange sharp periodical increases in training loss! It seams it has something to do with my dataset. Does anyone know how I should diagnose this issue.
Here is my learning curve:
Note: The orange and blue curves are for train and validation, respectively.


texts: How to compute Hessian matrix for log-likelihood function for Logistic Regression

texts:
I am currently studying the Elements of Statistical Learning book. The following equation is in page 120.

It calculates the Hessian matrix for the log-likelihood function as follows

\begin{equation}
\dfrac{\partial^2 \ell(\beta)}{\partial\beta\partial\beta^T} = -\sum_{i=1}^{N}{x_ix_i^Tp(x_i;\beta)(1-p(x_i;\beta))}
\end{equation}

But is the following calculation it is only calculating $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i^2}$ terms. But Hessian matrix should also contain $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i\partial\beta_j}$ where $i\neq j$.

Please explain the reason for missing out these terms.


texts: I m using a data set with 41 features numerics and nominals the 42 one is the class (normal or not) first I changed all the nominals features to numeric since the autoencoder requires that the imput vector should be numeric. so the number of features incresed from 42 to 122. I removed the class colomn because AE use unlabelled data and I used it to reduce dimensionality from 121 to 10 ( 121> 50->10->50-121) now I want to build a MLP to classify the data I divided the data set into 3 parts: train, validate and test set, I want to put the 10 features selected by the AE instead of the 121 but I dont know how (code?). and how add the class colomn again to the data set to do a supervised classification with MLP?


texts: How to create pipeline for Neural Network models?

texts:
I have tried creating a pipeline like this but I am getting error

- AttributeError: 'Pipeline' object has no attribute 'compile'

scaler = StandardScaler()
model = Sequential()

model.add(Dense(120, input_dim=46,activation='relu'))
model.add(Dropout(0.1, noise_shape=None, seed=None))

model.add(Dense(80, activation='relu'))
model.add(Dropout(0.1, noise_shape=None, seed=None))

model.add(Dense(40, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

pipeline = make_pipeline(scaler,model)
pipeline.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

pipeline.fit(X_train,y_train, epochs=50, batch_size=20, validation_data = (X_test,y_test))

# evaluate the model
scores = pipeline.evaluate(X_test,y_test)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))


texts: Accuracy of word and sent tokenize versus custom tokenizers in nltk

texts:
The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and provides a method to perform sentence segmentation on unseen corpus.

NLTK provides word_tokenize and sent_tokenize. Creating our own tokenizers can help us understand how one works, but in a production environment why would we want a custom tokenizer?  And if I built a custom tokenizer, how could I measure if it was better that NLTK's tokenizer?


texts: How do I aggregate cross-validation results for per-sample insights?

texts:
I'm trying to do feature engineering for point cloud data with three classes, and after having finished implementing the most obvious and simplest features with somewhat good results I'd like to use insights on the classification results to find difficult samples to look at to think of features that might distinguish those.
I think a confusion matrix is really useful for this, especially because it also shows how the minority class is handled.
It would be easy to compute metrics like a confusion matrix if I only had a training and test set, but I chose k-fold cross-validation for a robust estimation of the generalization error and because a single split for this imbalanced dataset would skew training and evaluation (because there are highly correlated groups of samples). This gives $k$ estimators, and I'm wondering how I can aggregate their predictions.
I'm currently using two methods:
Combining estimators predictions on their respective test splits
Since k-fold CV puts each sample in exactly one test split, I can combine those prediction results to get a single prediction for each sample. With this I can easily construct a confusion matrix. This method of combining is equivalent to sklearn's cross_val_predict, where its documentation warns:

Passing these predictions into an evaluation metric may not be a valid way to measure generalization performance. Results can differ from cross_validate and cross_val_score unless all tests sets have equal size and the metric decomposes over samples.

As I understand it, the problem is that different estimators based on different test sets are used, but it's not really clear to me. Is it still OK to use it to find difficult samples or at least get an overview from the confusion matrix?
Combining estimators predictions on the whole dataset each
I can also use the $k$ estimators to predict each sample of the dataset, which gives me $k$ predicted classes per sample. It's not easy to construct a confusion matrix for this, however I can plot a histogram to show how often the estimators are in full agreement correct (5 correct predictions) or in full agreement wrong (0 correct predictions). Anything in between is at least partially caused by differing training splits, whereas the zeros indicate either wrong labeling or something the existing features haven't captured so far, which makes those samples really interesting to me.

I'm planning to come up with features for these difficult samples and evaluate the performance difference with the confusion matrix of approach #1 without fiddling to much with parameters afterwards, and instead use a nested CV with GridSearch to generate the final feature + model selection.
Is there something wrong with this approach? Can it cause overfitting or overly optimistic results?


texts: Can a linear regression model without polynomial features overfit?

texts: I've read in some articles on the internet that linear regression can overfit. However is that possible when we are not using polynomial features? We are just plotting a line trough the data points when we have one feature or a plane when we have two features.


texts: Relation between number of landmarks and learning accuracy

texts:
Is there any relation between number of landmarks selected on the subject and the accuracy of learning these landmarks?
For example for detecting nose tip and eye corners, can we say that adding some labels for eyebrows and mouth and lips will help or not generally?
Intuitively, I think that giving more landmark to the machine, will give it more information and it can use relations between these points to predict desired landmarks more accurately.


texts: Intuitive explanation for representing gradient in higher dimensions

texts:
I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters.
For example, a network with 2 parameters that correspond to the X and Y axis, respectively, and cost function that corresponds to the Z axis makes sense...but how can we have a network with 1000 dimensions being represented in a 3D space, on a planar cost surface (not sure if word planar is being used correctly). Am I thinking of dimensions in the network the wrong way?


texts: Reinforcement (Q) learning: does it learn while in production?

texts:
I have a question for which I could not find the answer to it:

While training reinforcement learning (using DQN), I get a model for the best reward for the next action. Now, if I deploy this model (i.e., use this model to make predictions), does it keep learning (i.e., updating the Q values)?


texts: How could I go about finding the weights or importance of inputs based on outputs?