texts: Visualising 2-step sequence data to spot the differences among two different user groups

texts:
I have the following data frame for a certain user group. Basically, it shows the current page, and the previous page visited across all users. For example, 81 users come from BLQ2_2 to visit BLQ2_1. I have the same data set for another user group. I am trying find an effective way of visualising these data sets to spot the differences among the user groups.

PAGE    PREV_PAGE COUNT
BLQ2_1  BLQ2_2    81
        BLQ2_4     8
        BLQ2_3     6
        BLQ2_1     3
BLQ2_2  BLQ2_3    91
        BLQ2_4    22
        BLQ2_1     9
        BLQ2_5     5
        BLQ2_2     1
BLQ2_3  BLQ2_4    87
        BLQ2_1    12
        BLQ2_5     6
        BLQ2_2     2
        BLQ2_3     1
BLQ2_4  BLQ2_5    45
        BLQ2_1    26
        BLQ2_2    17
        BLQ2_3     8
        BLQ2_4     3
BLQ2_5  BLQ2_1    19
        BLQ2_2    12
        BLQ2_4    12
        BLQ2_5     8
        BLQ2_3     6

A bar chart could help. But, I wonder if there are other options around there that I could not think of. Another option could be arranging the pages like this:

BLQ2_1       BLQ2_2      BLQ2_3       BLQ2_4      BLQ2_5

and connecting them based on the visits (e.g., drawing a thick line -considering the COUNT value is 81- from BLQ2_2 to BLQ2_1)

I appreciate some input.


texts: How can I check if a machine learning model is feasible on a given dataset? What techniques like EDA, correlation etc. can be used to judge if a model is possible i.e. data and predictor variables will give reasonably accurate forecasts or in other words there is good enough signal in predictors?


texts: Wavenet - how are the skip connections from the residual blocks utilized?

texts:
I've been attempting to implement the Wavenet paper: https://arxiv.org/pdf/1609.03499v2.pdf
In the paper, the main diagram they use to describe the architecture is this one:

The paper mentions the use of residual and skip connections in order to enable the training of deeper networks, which I understand. But what I do not understand is why they extract the skip values and sum them before passing them to the last portion of the network.
In the paper, they state that they are attempting to predict the value of the sequence at time T, given the values of X0 -> X(T-1). They quantize the values into the range [0,255] and output a probability distribution describing the likelihood that the next element belongs to one of the 256 quantized classes. Therefore, this last portion of the network should output a probability vector fitting the above description.
My questions are:

Does the description above have any misunderstandings about how the model functions?
Why do the authors extract the skip values? Why do they sum the skip values? What do these values "mean" in relation to the problem of audio generation?
If they are simply summing the skip vectors how do they control for dimensionality? Shouldn't each residual block have a different dimension for its output due to the varying dilation factors? If so, how do we end up with a probability vector with 256 entries describing the probability that the next sample is in one of those classes (how are the dimensions tailored to reach the desired output dimension)?


texts: PCA and FastICA in scikit-learn giving near identical results

texts:
So after importing my data, transforming it, and splitting into training and test sets I tried running this script for PCA:

pca = PCA(random_state=2, n_components=2).fit_transform(X_train)
pca = pd.DataFrame(pca, columns=["X1","X2"]).assign(y = np.array(y_train))
pca.plot.scatter(x='X1', y='X2', c='y', colormap='viridis')

Giving me this graph

I then ran this script for ICA:

ica = FastICA(random_state=34, n_components=2).fit_transform(X_train)
ica = pd.DataFrame(ica, columns=["X1","X2"]).assign(y = np.array(y_train))
ica.plot.scatter(x='X2', y='X1', c='y', colormap='viridis')

If you look closely, you'll see these graphs are exactly the same. Except flipped over the X1 axis. Any idea what could cause this? I was expecting ICA to produce something very different from PCA


texts: Objective function for multi-label classification

texts: The customary objective function for multi-label (e.g. M labels) classification is binary cross-entropy. The problem is, if we use binary cross-entropy, we are assuming that the output labels are independent of each other, turning the problem to M independent binary classification problems. Is there any suitable objective function that makes the output labels to be dependent on each other?


texts: What algorithm is best suited to derive the best match between two people in a data set?

texts:
Say i have a large data set that contains the following data;

username,age,sex,music_genre,sports_genre,art_genre,rating
DanTheMan,25,male,rock,football,cubsim,50
LoopyLucy,23,female,pop,null,popart,76
MadDog,33,null,pop,football,cubsim,57

I would like run through the whole data set and match two users based on ̶t̶a̶s̶t̶e̶s̶
genres and rating that are similar or closer then others like a best match. The gender does not need to be considered, What would be the best algorithm to acquire this result? There is an equal amount of users, Everyone will get a match.

I have taken a look into the stable marriage algorithm (Gale-Shapley) however I would appreciate someone else's suggestion and opinion.

Thanks!


texts:
This is to seek career advice for a data scientist. What pertains within the role of data scientist and what does not regarding ElasticSearch.

Does backend development for ElasticSearch using languages/frameworks such as Node.js, Django etc fall under the role of data scientist, or does it come under software engineering/hard code backend development? I believe there are many data scientists in this forum who work on ElasticSearch, so I thought of checking if anyone has also expanded their role as a backend developer for ElasticSearch.


texts: matterport mask-r-cnn transfer learning on own dataset using VGG annotator ver.1

texts:
The code I used was taken from shapes.py, balloon.py, inspect_balloon_data.ipynb but mostly from train_shapes.ipynb in the Matterport repo because I read from some posts on Stackoverflow that adapting balloon.py and the ipynb notebooks there are the fastest way to get up and running on your own datasets annotated using the VGG Annotation tool (version 1 as version 2 won't work with the code). Link to VGG Annotation tool: VGG annotation tool I made some changes to balloon.py so that I can use the circle bounding circle rather than a polygon:

At the bottom of def load_balloon():

self.add_image(
                "tennisball",
                image_id=a['filename'],  # use file name as a unique image id
                path=image_path,
                width=width, height=height, #come from right above
                circles=circles)

and in def load_mask():

#convert circles to a bitmap mask of circle
#[(x,y) of center, radius]
info = self.image_info[image_id] #jpg num
mask = np.zeros([info["height"], info["width"],
     len(info["circles"])],dtype=np.uint8)
for i, p in enumerate(info["circles"]):
    rr, cc = skimage.draw.circle(p["cx"],p["cy"], p["r"])
    mask[rr, cc] = 1

it looks like the tennis ball is detected but the mask is placed in the wrong location.

FYI my repo is here with the notebook and balloon.py my repo

Edit:   In def load_balloon() in balloon.py I made some other changes to go along with the other code pasted here:

 # Add images
 for a in annotations:
 # Get the x, y coordinaets of points of the polygons that make up
 # the outline of each object instance. These are stores in the
 # shape_attributes (see json format above)
 # The if condition is needed to support VIA versions 1.x and 2.x.
            if type(a['regions']) is dict:
                #polygons = [r['shape_attributes'] for r in a['regions'].values()]
                circles = [r['shape_attributes'] for r in a['regions'].values()]
            else:
                #polygons = [r['shape_attributes'] for r in a['regions']]
                circles = [r['shape_attributes'] for r in a['regions']]

In the meantime, this github repo changes rectangle annotations in XML to the format required by Matterport Mask R-CNN code (ie. all points x and y that make up the perimeter of polygons):  xml to json

Edit:  I was able to do some transfer learning on Colab and my result:

The mask and bounding box are in the bottom left near the score.  At least it's sort of trying to bound the yellow part of the score I'm hoping.  This was with a training set of 21 images and validation of 10 images which is way below the recommended 100 to 1000 for transfer learning.  I just wanted to see that the code executes on Colab with minor adjustments (must mount Google Drive).  Full ipynb notebook is here:  tennisball-tracker

Edit:  I think I'm coming to the conclusion that this task is impossible for mask-r-cnn for tennis balls on TV broadcasts without a high FPS camera because the ball is so blurry to the point that it's nearly invisible in some frames.  Ie. This frame right above shows Rafa finishing his serve motion so the ball is near its top speed in a tennis match and I cannot spot the ball.  Thus I cannot make a good training dataset from raw video frames shown on regular TV broadcasts.  Please refer to this other question if you are also trying to do the same task:  concatenation of consecutive video frames


texts:
Consider points x1 = (1,1), x2=(1,0), x3=(1,-1) from class C1 and points x4 = (-1,1), x5=(-1-1) from class C2. Classify
the given data with SVM

How do we manually classify data by finding the equation of the line that will separate the classes?


texts: How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn

texts:
I'm trying to get mean test scores from scikit-learn's GridSearchCV with multiple scorers.

grid.cv_results_ displays lots of info. But
grid.cv_results_['mean_test_score'] keeps giving me an error.

I've checked the docs and similar questions with no luck. What am I doing wrong?

Code:

scorers = {
    'r2': 'r2',
    'nmsle': 'neg_mean_squared_log_error',
    'nmse': 'neg_mean_squared_error'
}

params = [
    {
        'regressor': [GradientBoostingRegressor()], 'preprocessing': [None],
    },
]

grid = GridSearchCV(pipe, params, cv=5, scoring=scorers, refit='nmse')
grid.fit(X_t, y_train)

grid.cv_results_['mean_test_score']

Error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-161-a021fe7b05bb> in <module>
     20 print(f'Val score set score rmse: {rm}')
     21
---> 22 grid.cv_results_['mean_test_score']
     23
     24 # for mean in means:

~/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py in __getitem__(self, key)
    124             warn_args, warn_kwargs = self._deprecations[key]
    125             warnings.warn(*warn_args, **warn_kwargs)
--> 126         return super(DeprecationDict, self).__getitem__(key)
    127
    128     def get(self, key, default=None):

KeyError: 'mean_test_score'
```


texts: Do zero weights receive zero gradient in ReLU neural networks?

texts: Suppose I have a deep neural network using the ReLU activation function, that is $\sigma(x) = max(x, 0)$. Suppose some weight $w_i$ becomes exactly $0$ at some point. Am I getting something wrong here, or is it the case that the gradient w.r.t. $w_i$ will be zero at all times and hence $w_i$ won't get any further updates? I feel like I am missing something here.


texts: Confused about how to graph my high dimensional dataset with Kmeans

texts:
PLEASE NO SKLEARN ANSWERS

So I have a dataset that is very high dimensional, and I am very confused about to convert it into a form that can be used to plot with Kmeans. Here is an example of what my dataset looks like:

Blk Students %       White Students %     Total Endowment    Full time Students     Tuition Revenue

     35                    60               $4,000,000             50000              $4,999,999
     50                    50               $5,888,888             67899              $200,000,000
     .                     .                    .                     .
     .                     .                    .                     .

.

I have normalized my variables, converted everything to integers, etc... But I am confused about how I am supposed to take this high dimensional dataset and plot it on a 2D scatterplot for kmeans? Thanks in advance.

To answer a question asked in the comments: there are 60 more columns than shown here.


texts: Should I transform my feature into normal distrubition before Isolation Forest

texts: I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?


texts: Model threshold for classification model

texts: How do I extract a model score based on desired precision recall for a classification model? Is there a command to extract it from precision_recall function in sklearn?


texts: Comparison with transfer learning and CNN from scratch on a small dataset

texts:
I am doing image classification with CNN, and I have a training set of 3200 imges and a training set of 400 images. I have used two different approches for doing this classification : Transfer learnoing and bulding a CNN from scratch.

In the case of transfer learning I get an accuracy of 85%, and in the case of building the network from scratch I get an accuracy of 89%.

But usually with a small dataset shouldn't transfer leanring have better performances?

One thing to consider first is that I used early-stopping as regularization technique, and in the case of transfer learning this stopped the training process at the epoch 18, while for the CNN from scratch I arrived at the epoch 100, which is the number of epochs I imposed for training.

Can someone help me with my doubt? Thanks in advance.


texts: Weighted Conditional Expectation in AdaBoost

texts:
I am looking at "Additive logistic regression a statistical view of boosting" paper (https://web.stanford.edu/~hastie/Papers/AdditiveLogisticRegression/alr.pdf)
In page 346, the authors introduce a definition for a weighted expectation. I do not understand the motivation behind this definition and how it is used to get to this result.
It starts introducing it to minimize the following expression:

Can someone explain the idea behind this please?
Thank you


texts: Is it possible that a CNN has better accuracy than RNN in word classification?

texts:
So I found something strange once I compared the accuracy of the prediction of a class for a question between a CNN and an RNN (GRU).

The CNN achieved 0.87 accuracy over the RNN (GRU) with 0.7520 accuracy. In terms of question classification task, would it makes sense if CNN outperforms RNN or is it a rare case?


texts: In Pytorch, if I have a 2D tensor, how to iterate over this tensor to get every value changed

texts:
I have a 2d Tensor, whose size is 1024x1024 and the values in the tensor is 0.3333, 0.6667, and 1.0000, so I would like to change all these values to 0,1,2.

Could some one tell me how to iterate over this tensor.


texts: Qunatify total time saved by prioritizing tasks based on the failure rate probability of each task

texts:
I am trying to solve a problem where I am trying to prioritize the tasks in a job based on the failure rates of each task. For ex:
Task   p(Failure)    TimeTaken(sec)
 A       0.7            10
 B       0.1            15
 C       0.5             3
 D       0.3             5

This is a sequence of tasks and if even one task fails, the entire job fails. So I want to prioritize my tasks to save the max amount of time. To do that I am trying to run the tasks in the order of failure probability. So my current order of performing the tasks is A,C,D and then B. I feel the problem with my approach is I am not considering the time factor. Is there a better way to prioritize my tasks based  on the time taken also into consideration ?


texts: How can I draw bar graph in python on aggregated data?

texts:
Normally when I draw bar plot its simple as

import matplotlib.pyplot as plt
from pylab import rcParams
import seaborn as sb

%matplotlib inline
rcParams['figure.figsize'] = 5, 4
sb.set_style('whitegrid')

x = range(1, 10)
y = [1,2,3,4,0.5,4,3,2,1]

plt.bar(x, y)

When I aggregate data on basis of age feature with the following command

data_ag = data.groupby('age')['age'].count()

It returns the output that is the people belonging to a particular age e.g. 11 people with age 22 years.

age
22     11
23      8
27     28
28      1
29     70
30     13
31     45
Name: age, dtype: int64

How can I treat those as x and y points to draw a bar plot?

x = # what should I write here for age data
y = # what for count

plt.bar(x, y)


texts: Can I add categories to a dataset after and retrain an H5 file? Without retraining the previous ones?

texts:
This is the situation. I'm training a model to recognize letters of the Alphabet. There are 26 classes. When writing the code for 26 classes, and loading nearly 100,000 images to train, I'm having a lot of issues. However, I was able to successfully train the model to work on 10 letter increments. As in, A-J, K-T, and then U-Z. These three work perfectly fine (A-Z does not).

Question: Can I train my A-J model, save the H5. Then, train the K-T, and U-Z after and then MERGE the H5 files together? I understand it's possible to train all A-Z using fewer images and then retrain the model with a different image set, but the issues are coming when I'm doing a 26 class system - hence I'm asking if I can do it in increments of 10 and merge it after.


texts: How to do hidden variable learning in Bayesian Network with Python?

texts:
I learned how to use libpgm in general for Bayesian inference and learning, but I do not understand if I can use it for learning with hidden variable.
More precisely, I am trying to implement approach for Social Network Analysing from this paper: Modeling Relationship Strength in Online Social Networks.
They suggest to use following architecture

Here

S(ij) represents vector of similarity between user i and j - Observed
z(ij) is a hidden variable - relationship strength (Normal distribution regularised by W - weights and similarity vector)- Hidden
yt(ij) is user interaction(1,2…n -> certain type of interaction e.g. 1=I retweeted j) (function of z and a that involves Theta parameter) - Observed
at(ij) is auxiliary variable which represents how often certain interaction occurs - Observed

Approach described in paper for training is quite difficult and involves coding of ascent optimisation. I wonder If I can use libpgm to learn W and Theta parameters. If yes, how to do it? If no, what libraries I can use to do it.


texts: Higher dimension data visualization in Matlab/Octave

texts:
I am working on sparse recovery for a classification task. I use Pine hyperspectral dataset which is a freely available dataset and this image contains 200 Dimension (Depth/channels/bands). In addition, this dataset contains labels (Ground truth information). Thus I extract the corresponding pixels for each class and now I want to visualize them to see really how they look like. But I do know now how to visualize them for all 200 channels. I did this visualization for 3D but I want to know how can I expand this for 3D.

Here is my code for 3D, I would appreciate any help for visualizing my data in 3D
Not: I simulate the dataset.

clear all
% Number of classes = 5
B = [1 1 2; 2 3 4; 2 1 3; 4 3 4; 5 5 3];
B_vec = reshape(B, 5*3, 1);
% Image with 200 depth
A = rand(5,3, 200)*100;
A_vec = reshape(A, 5*3, 200);
%Indexing the desire classes.
indx = find(or(B==1, B==4, B==3));
class_num = B_vec(indx);
%Extracting the corresponding pixels (base on indx)from 200D image.
for k = 1: length(indx)
img_value(k,:) = A_vec(indx(k),:);
end
G_idx_1 = (find(class_num==1));
for k = 1:length(G_idx_1)
C_1(k,:) = img_value(G_idx_1(k),:);
end
G_idx_3 = (find(class_num==3));
for k = 1:length(G_idx_3)
C_3(k,:) = img_value(G_idx_3(k),:);
end
G_idx_4 = (find(class_num==4));
for k = 1:length(G_idx_4)
C_4(k,:) = img_value(G_idx_4(k),:);
end

X1 = [C_1'];
X2 = [C_3'];
X3 = [C_4'];
plot3(X1(1,:), X1(2,:), X1(3,:),'bo'); axis on; grid on; hold on
plot3(X2(1,:), X2(2,:), X2(3,:),'go');
plot3(X3(1,:), X3(2,:), X3(3,:),'ro'); hold off;
title('class1, class2, class3')


texts: When to use Linear Discriminant Analysis or Logistic Regression

texts:
The Wikipedia article on Logistic Regression says:

  Logistic regression is an alternative to Fisher's 1936 method, linear
  discriminant analysis.  If the assumptions of linear discriminant
  analysis hold, application of Bayes' rule to reverse the conditioning
  results in the logistic model, so if linear discriminant assumptions
  are true, logistic regression assumptions must hold.  The converse is
  not true, so the logistic model has fewer assumptions than
  discriminant analysis and makes no assumption on the distribution of
  the independent variables.

Could someone help me to understand what the assumptions of linear discriminant are, an example of where they hold, and how application of Bayes' rule results in the Logistic Model.

Would it be correct to say that Logistic Regression is always the preferred choice?  Are there conditions where Linear Discriminant Analysis is to be preferred?


texts: k-fold cross-validation: model selection or variation in models when using k-fold cross validation

texts:
I am new to machine learning, though I have a background in statistics. But I had a question about $k$-fold cross-validation. So I understand the basic idea that we divide the dataset into $k$ partitions and then train a model on $k-1$ partitions while testing on the $k$th partition that was left out. So we don't want to train the model over the entire dataset, but instead over just our $k-1$ partitions.

My question was how do we handle changes in the model parameters with each fold in the process. Another way of asking this question is how do we choose the model to test? So when you train on $k-1$ folds, in each iteration there will be a subtle change in the parameter estimates--namely the $\beta_{0...p}$s will change. So when we test the model against the testing partition, we are not testing the same model each time--because the parameter coefficients are different.

Given that the $\beta$s change for each fold, how do we choose which model to use? And once we choose a model, do we need to test only this model for the $k-fold$ cross-validation, for otherwise we are testing different models and then averaging their prediction error.

Any clarification would be appreciated.


texts: How to correctly lemmatize the text column in R?

texts: I'm working on a project in Natural Language Processing. I have a data frame that has a text column. I have to lemmatize that text column. I'm using lemmatize_strings() function in R. However, there's one case where word 'wound' in one of the text is being lemmatized to 'wind' which is not correct. Can anybody tell me what can be done to correct this ?


texts: Using a Subset of Categories in a Categorical Column

texts:
I have a XGBoost model and I'm going to retrain it by adding new features. There is a column in my data and it's about professions of the customers. It has 60 categories. I suppose there is no need to convert them to dummy variables because tree based models can handle them, but I thought that there should be many splits in order to do it and I decided to use a subset of categories and group other categories under one category. To decide categories which I'll keep I applied one-hot encoding to all of them and applied chi-square test by using scipy.stats.chi2_contingency to each generated dummy column and target variable. Then I sorted columns by test statistic in ascending order and picked first 10 of them. Then in the original column I kept the values that also in the subset and assigned same category to others. I'm not sure it's a proper method or is there any inconsistency in it? Any suggestions?

My code is as below:

from scipy.stats import chi2_contingency

def get_n_cats(df, col_to_cat, n, target="TARGET_GPL_SATIS"):
    # Apply one-hot encoding
    dummies = pd.get_dummies(df[col_to_cat])

    # Calculate chi-square statistic for each dummy column
    scores = pd.DataFrame(index=dummies.columns, columns=["Score"])
    for col in dummies.columns:
        cont_table = contingency_table(dummies[col], df[target])
        score = chi2_contingency(cont_table)[0]
        scores.loc[col,"Score"] = score

    # Sort by score and get first n columns
    scores.sort_values(by="Score", ascending=True, inplace=True)
    first_n_columns = scores.index[:n]

    # Create mapping dict
    mapping = create_mapping(first_n_columns)

    # Preserve only first n columns as categories
    return [mapping[val] if val in mapping.keys() else mapping["other"] for val in df[col_to_cat]]

def contingency_table(c1, c2):
    """Calculates contingency table between provided columns."""
    df = pd.DataFrame({"c1":c1, "c2":c2})
    return df.groupby(['c1','c2']).size().unstack(fill_value=0).values

def create_mapping(first_n_columns):
    """Creates mapping dict."""
    mapping = {cat:code+1 for code, cat in enumerate(first_n_columns)}
    mapping[np.nan] = 0
    mapping["other"] = len(first_n_columns) + 1
    return mapping


texts: Why BERT tokenizers function differently?

texts:
While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer:
transformer_bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

tokenizes the text differently from the tokenizer that I use to construct for my BERT models in this way:
!wget --quiet https://raw.githubusercontent.com/tensorflow/models/master/official/nlp/bert/tokenization.py
import tokenization
FullTokenizer = tokenization.FullTokenizer

and then
BERT_MODEL_HUB = 'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2'
bert_layer = hub.KerasLayer(BERT_MODEL_HUB, trainable=True)
to_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
vocabulary_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()

tokenizer = FullTokenizer(vocabulary_file, to_lower_case)

as an example:
sequence = "Systolic arrays are cool. This  is cool too."
transformer_bert_tokenizer .tokenize(sequence)
# output: ['s', '##ys', '##to', '##lic', 'array', '##s', 'are', 'cool', '.', 'this', '[UNK]', 'is', 'cool', 'too', '.']
tokenizer2.tokenize(sequence)
# output: ['sy','##sto','##lic','arrays','are','cool','.','this','[UNK]', 'is','cool','too','.']

Does anyone know why there is a difference? Aren't both tokenizers using the same vocabulary? which way is preferred?


texts: How to use hours of the day as a continuous feature?

texts: I would like to use the hour of the day (0-23) as a continuous feature, so the model will know that 12pm comes before 13:00, and that 8:00 is further from 21:00 than from 10:00. How do I engineer this feature so it will also understand that 0:00 comes after 23:00?


texts: What does "large sparse" and "small dense" means?

texts:
In a paper I'm reading today it's written :

  For a fixed parameter count, we discover that large sparse WaveRNNs
  significantly outperform small dense WaveRNNs and that this
  relationship holds up to high level of sparsity greater than 96 %.

What does it all mean ? Does it have something to do with the number of layers ?


texts: I want to try the skipgrams approach on my dataset. But I do not know how to vectorize it. For example, I have my cleaned document for which I got it's skipgrams. Now, how do I know vectorize it so that I can use it further for classification? I use sklearn for all the above purposes.


texts: How to learn to score new documents based on a existing set of scored documents?

texts:
I have a 50 000 documents of 1000 words or more ranked between 0 and 2000. They all deal with a similar topic. I'd like to create an algorithm that can learn to score new documents.

What approach do you think I should take?

I am a newbie in the field of machine learninig so if you could point me to introduction material about the particular solution you think of. I will be glad.


texts:
I was planning to use a graph theoretic and page-rank approach to finding the most influential person in an organization.

  Influential person is someone who drives a lot of activity in the
  organization. When he assigns a task, most people do it. When he sends
  a mail, most people reply to him. When he assigns some training, most
  people do it.

But there is one drawback to using graph theory here. For example, say Person A is very influential, and Person B has just joined an organization and works under A. B puts in a leave request and sends it A for approval. A approves the leave request. This seems contradictory, since A is very influential but A responded to B. According to graph theory, wouldn't this make Person B also very influential? This approach would give very bad results.

How can i overcome this limitation? Can anyone suggest an alternative approach?

I am using number of interactions to calculate an affinity scores between two users. Based on this, I suggest what content should be shown to a user (like FaceBook does). User A responds to User B most of the time.  so (A->B) has a high affinity score. So, in user B's newsfeed I intend to show content coming from User A.

is there a better way than just counting number of interactions?

If two users A & C have same affinity scores with B, if A is more influential than B, A's content will be shown first to user B.


texts: Can anyone please help me implement arcface loss function in siamese architecture for face recognition problem?


texts: Normalization and Outlier on Target variable which is continuous

texts: I have doubt that should I perform outlier analysis and normalization even on target variable which is continuous ?


texts: Multivariate time series forecasting with the extended Theta method

texts:
I am looking for an implementation (better if in Python) of the extended Theta method for multivariate time series forecasting, presented in the following paper:
D. Thomakos, K. Nikolopoulos, Forecasting Multivariate Time Series with the Theta Method, J. Forecast. 34, 220–229 (2015)
Any suggestion?


texts: Loaded model predicts well in colab but gives same label and accuracy when downloaded

texts:
I have developed a Recurrent Neural Network to perform sentiment analysis on tweets using the Kazanova/sentiment140 dataset in Kaggle.
The model looks like this:
def scheduler(epoch):
  if epoch < 10:
    return 0.001
  else:
    return 0.001 * tf.math.exp(0.1 * (10 - epoch))

callback1 = tf.keras.callbacks.LearningRateScheduler(scheduler)
callback2 = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss',patience=10, verbose=0, mode='auto',min_delta=0.0001, cooldown=0, min_lr=0)
callback3 = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', min_delta=0, patience=3, verbose=0, mode='auto',baseline=None, restore_best_weights=True)

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size+1, embedding_dim, input_length=max_length, weights=[embeddings_matrix], trainable=False),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Conv1D(64, 5, activation='relu'),
    tf.keras.layers.MaxPooling1D(pool_size=4),
    tf.keras.layers.LSTM(64),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()
num_epochs = 50

training_padded = np.array(training_sequences)
training_labels = np.array(training_labels)
testing_padded = np.array(test_sequences)
testing_labels = np.array(test_labels)

history = model.fit(training_padded, training_labels, epochs=num_epochs, validation_data=(testing_padded, testing_labels), verbose=2,callbacks=[callback1,callback2])

print("Training Complete")

model.save('sentiment_final.h5')

The model runs fine and predicts output perfectly when loaded from colab itself
The loaded colab code:
load_model= tf.keras.models.load_model('sentiment_final.h5')
#load_model.summary()

def decode_sentiment(score):

    if score < 0.5:
        return "NEGATIVE"
    else:
        return "POSITIVE"

def predict(text):

    x_test = pad_sequences(tokenizer.texts_to_sequences([text]), maxlen=16)

    score = load_model.predict([x_test])[0]

    return {"label": decode_sentiment(score), "score": float(score)}

predict("I love this day") #Outputs -> {'label': 'POSITIVE', 'score': 0.793081521987915}
predict("I hate this day") #Outputs -> {'label': 'NEGATIVE', 'score': 0.38644927740097046}
predict("I shouldn't be alive") #Outputs -> {'label': 'NEGATIVE', 'score': 0.12737956643104553}

But If I load the model in VSCode , the output is the same for all the models.
VSCode Implementation:
import tensorflow
from tensorflow.keras.models import load_model
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
import os

tokenizer=Tokenizer()
model = load_model('sentiment_final.h5')

def decode_sentiment(score):

    if score<0.5:
        return "Negative"
    else:
        return "Positive"

def predict_score(text):

    x_test=pad_sequences(tokenizer.texts_to_sequences([text]),maxlen=16)
    score=model.predict([x_test])[0]
    return {"label":decode_sentiment(score),"score": float(score)}

def call_predict_function(text):
    return predict_score(text)

print(call_predict_function("I love this day")) #Outputs -> {'label': 'POSITIVE', 'score': 0.793081521987915}
print(call_predict_function("I hate this day")) #Outputs -> {'label': 'POSITIVE', 'score': 0.793081521987915}
print(call_predict_function("I shouldn't be alive")) #Outputs -> {'label': 'POSITIVE', 'score': 0.793081521987915}

Where am I going wrong? Can somebody resolve this problem?


texts: Actor-critic architecture: How is the policy updated?

texts:
I am going through the ddpg baseline code to try and gain an intuitive understanding of how the actor and critic networks function.

DDPG has two components: the actor which is the deterministic policy \pi and the critic which is the state-value function Q(s, a). The way you update the actor \pi is by computing the gradient of Q(s, \pi(s)). The idea is that the policy can be seen as a continuous equivalent of argmax and so you try to update it such as it takes the action that maximizes the Q-function in a given state.

This can be depicted as shown below.

The code shows three different neural networks created.

actor_tf = create_neural_net(observations) # Maps states to desired actions

critic_tf = create_neural_net(observations, actions) # Updates value function
critic_with_actor_tf = create_neural_net(observations, actor_tf) # Used for policy updating

My question is with how the policy is updated, and more specifically with critic_with_actor_tf.

As explained here,

So critic_with_actor_tf represents Q(s,\pi(s)) the action-state value in a state s (here observation = state) following the policy pi (the actor) (a = \pi(s)). This is what is used to compute the gradient for the actor:

 self.actor_loss = -tf.reduce_mean(self.critic_with_actor_tf)

So, it seems like the actor updated by reducing the mean of critic_with_actor_tf.

This raises the question, what does the TD Error shown in the diagram above represent and how is that related to updating the policy?


texts: use classification to improve clustering

texts:
I am from information security field, and have some introductory level understanding of machine learning field. My problem is to identify behavioural patterns from network traffic. If I use supervised learning ( classification ) the results are very promising but I encounter of a problem of missing out a pattern with slight change in behaviour. And If I use clustering then i need to do a much work on manual tuning of centroids position to get better clusters.

I am thinking to ensemble classification and clustering and use classification output to tune cluster centroids. Is this is a good idea to address the problem?


texts: How to create a multi label classification network in Keras if I have the training data with various accuracy?

texts:
I'm trying to create a neural network that finds the most effective treatment for each patient.
I have a medical database for training. The inputs are histological and pathological data (mostly 0/1 data of having some diseases, conditions) and the outputs are the treatments. And there is an accuracy score, which is the patient's reaction to the treatment (this shows how good was the treatment for the symptoms, so how accurate was the input-output pair).
How can I create a multi label classification network that finds the best treatment if the training data pairs are not always totally correct, rather having these variable accuracy scores?


texts:
Can anybody please explain the affect of multicollinearity on Decision Tree algorithms (Classification and regression).
I have done some searching but was not able to find the right answer as some say it affects it and others say it doesn't.


texts: too few data while too many degrees of freedom in linear regression

texts:
To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for training (with gradient descent) and got excellent results.

However, I just realized that my model had far more degrees of freedom than data points. Say the data are all nn pixel pictures to recognize, so the number of degrees of freedom is nn+1. I use n=50, but I only use less than 20 data points (20 training pictures). From the point of view of linear fitting/regression, the number of data points should at least surpass that of degrees of freedom. But now in my model the reverse happens, no solution should be found. What happened


texts: logistic like curve fitting using machine learning

texts:
I have a set of points of a function k(x). I am trying to do some curve fitting to find the exact k(x) function. It seems that the data points fit to a logistic like curve only a little shifted and stressed.

So far I have tried polynomial regression, but I don't feel the fitting is correct. I have attached a snap of the fitted curve here.

So my question is, is logistic regression only used in classification tasks? Or can it be used for curve fitting?

If not what are the other available techniques to fit a logistic like curve to a set of data points?


texts: How to choose the number of features to select and number of features to drop?


texts:
I am struggling to understand what the rankings in Weka are representing. I.e. the coefficients for each attribute in the rank. What is the output in the Weka program for PCA telling me with these rankings? And how does this help me feature select attributes? Because right now its making no sense how they are ranked.

My data set is 31000 rows with 13 attributes. My class is Income_brack > or <= 50k


texts: Using Machine Learning to play the game SET

texts:
I recently tried to make a CNN which could play the game Set: https://en.wikipedia.org/wiki/Set_(game)

However, I quickly learned after using my own convolutional neural network and trying to retrain Inception-V3 and AlexNet to classify pictures of three cards into "set" and "notSet", that a CNN is not useful for this task.

Which machine learning algorithm would be useful for playing this game? Would an SVM work?

Here are links to my last question (with some code) and the Github repo which includes sample images of the dataset.

Keras val_acc unchanging when training (same label assigned to all images)

https://github.com/Keyan-Rhm/MLWithSet


texts:
I am having a problem during feature engineering. Looking for some suggestions. Problem statement: I have usage data of multiple customers for 3 days. Some have just 1 day usage some 2 and some 3. Data is related to number of emails sent / contacts added on each day etc.

I am converting this time series data to column-wise ie., number of emails sent by a customer on day1 as one feature, number of emails sent by a customer on day2 as one feature and so on. But problem is that, the usage can be of either increasing order or decreasing order for different customers.

ie., example 1: customer 'A' --> 'number of emails sent on 1st . day' = 100 . ' number of emails sent on 2nd day'=0

example 2: customer 'B' --> 'number of emails sent on 1st . day' = 0 . ' number of emails sent on 2nd day'=100

example 3: customer 'C' --> 'number of emails sent on 1st . day' = 0 . ' number of emails sent on 2nd day'=0

example 4: customer 'D' --> 'number of emails sent on 1st . day' = 100 . ' number of emails sent on 2nd day'=100

In the first two cases => My new feature will have "-100" and "100" as values. Which I guess is good for differentiating. But the problem arises for 3rd and 4th columns when the new feature value will be "0" in both scenarios Can anyone suggest a way to handle this.

One way to handle this:

I can add "No change" in those scenarios, but I am confused about one thing. If I do that, I will have to make the new feature as categorical, which is not ideal as the other values will be continuous.

Instead, I can have absolute values in the new feature and indicate the trend as "+1" or increasing "-1" for decreasing "no change" for no change and "0" if both the values have been "0". Would that be a good approach though?

The end goal is to predict if a user would continue using the application or not. So it basically would be a two-class model. And I would want to capture even the scale of usage i.e., "A user sending 100 emails every day" should be different from "B user sending 10000 emails every day"


texts: Should Principal components be normalized before applying K means on them?

texts: I want to get the Principal components of a dataset and apply K mean clustering on them. Do I need to Normalized the PCA output before applying Kmeans on them ?


texts: What is ERR@K metric and how to interpret it?

texts:
What is ERR@K (e.g. K=10)?
what does this metric's value tell me?


texts: Does employment of engineered immediate rewards in RL introduce a non-linear problem to an agent?

texts:
Suppose we operate with state-action pairs called 'S', and a reward function R() as follows:

R(S1) <- 0
R(S2) <- 0
...
R(Sn) <- 1

so that

E(R(S)) << 1, i.e., non-zero rewards are very sparse

In that case a RL agent faces a task of accumulating as many as possible 1s in order to maximize

sum(gamma^i * Ri), i ->> infinity

Now consider immediate rewards introduced by one to make the agent better behaved:

R(S1) <- 0
R(S2) <- -1
R(S3) <- -10
R(S4) <- 0.1
...
R(Sn) <- 1

so that

E(R(S)) ->> -10, i.e., a simple average of random rewards is strongly negative.

Is it the case that in order to maximize the sum of discounted rewards the agent may choose to avoid the states with strongly negative rewards even at the cost of not getting positive rewards in some delayed states?

Is this really a non-linear problem to the model that approximates the agent's actions in that varying level of negative/positive immediate rewards (let's say, from -1 to 1, VS. from -100 to 100) may distract the agent from getting the delayed rewards which must be maximized under optimal policy in favor of maximizing immediate rewards, even if gamma ->> 1?


texts:
I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset):

date                   free_space (GB)
2019-05-15 09:00:00    102.65
2019-05-15 09:05:00    102.69
2019-05-15 09:10:00    103.11
2019-05-15 09:15:00    102.58
2019-05-15 09:20:00    102.55

I would like to detect whether the new value of disk space is an outlier or not. There are several outlier analysis methods (ref link):

Box plot analysis
Based on Z score
IQR based analysis (this is similar to box plot analysis)

Above methods are more statistical approach to detect outlier. There are several ways using an unsupervised machine learning algorithm to detect outlier (ref link). For example,

K-mean
Markov Chain
Isolation Forest etc.

Which method is suitable for the above dataset? Should I implement unsupervised based machine learning algorithm or statistical approach?


texts: I'm writing my own CNN code from scratch. Though I got fast, converged and satisfactory results, the trained weights change very little in value (while cost/loss function drops in time rapidly in a seemingly converged manner). My initial weights: convolution kernels as non zero unit matrices; fully connected layer weights as 0's. The activation function is sigmoid. The data scale from 0 to 1. Why do the weights change so little?


texts: How to leverage description data in multi-class classification (dimensionality reduction)

texts:
I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as would be expected in many cases of description data:

in>>
df[['size description', 'weight Description', 'height Description']].nunique()

out>>
 size Description       4066
 weight Description      736
 height Description     3173
 dtype: int64

some examples of these values could be:

   Product                  Product Description
  ---------               ------------------------
   Ball                 Round bouncy toy for kids
   Bat                  Stick that kids use to hit a ball
   Go-Kart red/black    Small motorized vehicle for kids
   Go-Kart blue/green   Small motorized vehicle for kids
   Wrench               Tool for tightening or loosening bolts
   Ratchet              Tool for tightening or loosening bolts
   Reclining arm-chair  Cushioned seat for lounging

I think that the descriptions are standardized if they fall within a particular category but at this time I cannot confirm if the number of unique descriptions are finite. At this time, my assumption would be to treat these as nominal-categorical, as these are literally descriptive and not qualitative.

To that end, my question is what are some best practices for handling categorical features such as these?

Things I have considered:

Label encoding is obviously not viable in this situation, as the descriptions have no hierarchy.
One-hot encoding seems an unlikely solution as it balloons the shape of the dataset from (55300 , 6) to (55300 , 65223) due to the high cardinality of the other variables. However, I tried it anyway and generated 98% accuracy on my test set but very poor results on an out-of-sample validation set (5k records, 0-5% accuracy). Seems pretty clear it's over-fitting and thus, not viable.
Hashing, for whatever reason, will not apply to one of the columns, but I suppose it could still be viable. I just need to figure out why it's not hashing all of my features (I suppose this would be suited best for a separate question?)
PCA - could be viable, but if I'm understanding correctly the cardinality after one-hot encoding is too great and PCA will throw an error. In fairness, I have not tried this yet.
Binning doesn't seem feasible since I could have a value of 3.5 or three and 1/2. Each one would be considered a separate bin and thus not a solution to my problem.

Thanks to all that can share their insight/opinion.


texts: What is the accuracy majority class classifier?

texts:
I have an SFrame and a model:

train_data,test_data = products.random_split(.8, seed=0)
selected_words_model = graphlab.logistic_classifier.create(train_data,
                                                     target='sentiment',
                                                     features=selected_words,
                                                     validation_set=test_data)

After computing the accuracy of the model with `selected_words_model.evaluate(test_data) I'm asked "What is the accuracy majority class classifier on this task?" Yet I don't even know what this "means accuracy majority class classifier", shouldn't it be "accuracy of the majority class classifier" ?

Here is my attempt.

All these materials come from this coursera ML fundations course exercise.


texts:
What happens when we do repartition on a PySpark dataframe based on the column. For example

dataframe.repartition('id')

Does this moves the data with the similar 'id' to the same partition? How does the spark.sql.shuffle.partitions value affect the repartition?


texts: Could Deep Learning be used to crack encryption?

texts: Say you have a dataset with millions of rows and the attributes Plain Text, Key, and Output Ciphertext. Could Deep Learning, theoretically, be used to find patterns in the outputs that help decipher the ciphertext? Are there any other potential approaches?


texts:
Let's say I've a dataset with 800 rows(40 entries for each of 20 users). The entries are user session log ( columns are - browser, os, time, date etc for a specific session).

Now each user has unique id (1-20). Let's say user_id=1 is special one and I need to detect it whenever new data comes.

So for new data I need to predict whether that session is of user_id=1 or not.

My question is How to do that ?

One way I thought is make a feature representation for each of the 20 users, and whenever new data comes, take the distance of the data from each of feature and see the minimum distance.

But the problem is when I make a unique feature representation for each user, how to deal with columns like browser, os - because a user can have used multiple browsers in all those 40 sessions ?


texts: Convolution and Cross Correlation in CNN

texts: What would be the intuition behind using the convolution and cross correlation operation inside Convolutional Neural Networks? I am interested in putting together the theory from Digital Image Processing where these 2x operations are defined, and CNNs. Could anyone help me connect the dots?


texts:
I am going through the derivation of normal equation for multivariate linear regression. The equation is given by :

$\theta = (X^{T}X)^{-1}X^{T}Y$

The cost function is given by:

$J(\theta) = \frac{1}{2m}(X\theta-Y)^{T}(X\theta-Y)$

Simplifying,

$J(\theta) = \frac{1}{2m}(\theta^{T}X^{T}X\theta - 2(X\theta)^{T}Y + Y^{T}Y)$

Differentiating w.r.t $\theta$ and equating to zero

$\frac{dJ(\theta)}{d\theta} = \frac{d}{d\theta}(\theta^{T}X^{T}X\theta)-\frac{d}{d\theta}(2(X\theta)^{T}Y) = 0$

I want to specifically understand the differentiation of the left term:

$\frac{d}{d\theta}(\theta^{T}X^{T}X\theta) = X^{T}X\frac{d}{d\theta}(\theta^{T}\theta) $

$\frac{d}{d\theta}(\theta^{T}\theta) = [\frac{d}{d\theta_1}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}),\frac{d}{d\theta_2}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}) ,...., \frac{d}{d\theta_n}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2})]$

$\frac{d}{d\theta}(\theta^{T}\theta) = [2\theta_1,2\theta_2,...,2\theta_n] $

$\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta^{T} $

But the final equation is obtained by using $\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$

How is $\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$ and not $2\theta^{T}$


texts: Why would you split your train data to compute a value on half of the data to then fill the Nan values on the other half?

texts:
I was checking a kernel written in python from the Bosch kaggle competition (kaggle link to python kernel) and I came across with a weird (at least to me) way to fill Nan values.

The train data is split into two halves and then some kind of average is computed in one half by using the non-Nan values of a field along with the target value and then fill Nan values on the other half with these computed values.

Then, when training the model after filling Nan values, the model is only trained on the half data where the Nan values have been replaced.

The question is, why would you split the data into two halves to compute on one and then fill the other?
Are you introducing some kind of leakage when mean values are related to the target value and that's the reason why just the half part with filled values (the one in which you haven't computed anything, just filled Nans) is used to train?
Is this procedure prone to overfitting if you perform this operation on all the train data?

Thanks in advanced.


texts:
(Please pardon the pun in the question title.)

I'm currently teaching an introductory class to data science.

I understand the differences between
a left join, a right join, an inner join and a full join.
However, I am not sure how to explain to students when
a particular join (say an inner join)
is the appropriate join for that particular situation.

I am looking for three real-life examples:

In example A, an inner join is the most appropriate.
In example B, a left/right join is the most appropriate.
In example C, a full join is the most appropriate.

I don't need all the details,
but I do want the big picture ideas that show that
in that real-life example, this join is most appropriate.

Note: I am teaching using the R package dplyr,
so I have used the names of the dplyr join functions.


texts:
I am clustering time-series datasets which are not labeled (No Ground truth) and I want to measure the quality of the clusters.
Could you please suggest any Clustering performance evaluation methods that can be used in time-series clustering ??


texts: High performance R libraries for interactive graphics

texts:
In the past few years there has been an explosion in the quantity and quality of available data visualization tools in R. I've long ago stopped using plot base graphics in favor of the wonderful ggplot2 package, and I'm considering going for interactivity now, perhaps using one of the JS-integration packages, and outputting HTML.

However, I'm often in the business of creating long HTML report pages with dozens of graphics in there, and I care about a number of factors:

load-time for the graphics generated cannot be too long.
should work decently with Shiny and Rmarkdown (ideally flexdashboard).
should have a non-insane syntax for tweaking options (i.e no 20 line JS code inserts to change legend location).
should be relatively well-documented.
should be actively maintained.

What are your go-to packages for interactive data viz?


texts: Time-series multi-step generalization from single step model

texts:
I have built a generic stacked lstm model of the form:
self.model = tf.keras.Sequential([
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons),
            tf.keras.layers.Dense(n_output)
        ])

with the data shaped as x_train = (total_samples,timestep, features) and y_test = (total_samples,n_output,1) and then I fit the model with a validation split of 30%.
My question is:
Is there a good theoretical reason for why the model perfectly predicts one step ahead (n_output = 1) but when I try to set n_output>1 the model becomes very, very imprecise? (loss<val_loss from epoch 1)
Is it a model/data dependant thing or am I botching the data preparation?


texts: In a linear model, regularization decreases the slope. Do we just assume that fitting a lin model on training data overfits by almost always creating a slope which is higher than it would be with infinite observations instead? What is the intuition?


texts: Process melspectrograms with convolutional neural network

texts:
I am trying to do audio classification with a convolutional neural network. There are six classes. With librosa, I have created melspectrograms for the one second long .wav audio files. It returned 640x480 .jpg files.
My question is now how to proceed with the input, since I think it is too large as input for the network. If so, what would an adequate resolution be? Something around 60x60? Does it even have to be quadratic?

Options from my perspective:

Re-encode melspectrograms from librosa with smaller resolution
Use cv2 and simply do a cv2.resize() before passing it to the input layer.
Leave the resolution untouched, and introduce more convolutional layers.
?


texts: When selecting a probability threshold to maximize the F1 score prior to deploying a model (based on the precision-recall curve), should the threshold be selected based on the training or holdout dataset?


texts: Does batch_size in Keras have any effects in results' quality?

texts:
I am about to train a big LSTM network with 2-3 million articles and am struggling with Memory Errors (I use AWS EC2 g2x2large).

I found out that one solution is to reduce the batch_size. However, I am not sure if this parameter is only related to memory efficiency issues or if it will effect my results. As a matter of fact, I also noticed that batch_size used in examples is usually as a power of two, which I don't understand either.

I don't mind if my network takes longer to train, but I would like to know if reducing the batch_size will decrease the quality of my predictions.

Thanks.


texts:
I have a binary classification task. I have shown the loss curve here. I have decreased the learning rate by 1/10 every 15 epochs. There is also dropout put in the model. As you can see, I am trying to figure out the optimal point for model training. My initial assumption was that the point came at around epoch 28 since the validation error almost remains constant and then increases ever so slightly. However, I still wanted to know if this is fine or the model is indeed overfitting.

Another concern I have is that the training and validation curves are very very close to each other. Is this an expected behavior?

Being a newbie I would really appreciate any help in here


texts: Using a pre trained CNN classifier and apply it on a different image dataset

texts:
How would you optimize a pre-trained neural network to apply it to a separate problem? Would you just add more layers to the pre-trained model and test it on your data set?

For example, if the task was to use a CNN to classify wallpaper groups, I'm sure that it wouldn't work to directly classify off a pre-trained network trained on images cats and dogs, even though both are image classifiers.


texts: What GPU specifications matter when training and using neural networks?

texts:
I need to purchase some GPUs, which I plan to use for training and using some neural networks (most likely with Theano and Torch). Which GPU specifications should I pay attention to?

E.g.:

one should make sure that the VRAM is large enough for one's application
the more teraflops, the faster programs running exclusively on the GPU will run
the higher the thermal design power (TDP), the higher the electricity bill
the compute capability, which determines which version of CUDA and cuDNN one can use.

What else matter, and what else doesn't matter?

E.g., is it reasonable to assume that the number of sockets, clock speed and the number of cores do not bring any additional useful information (since we already consider the number of Tflops)?


texts: How to find a probability distribution the parameters of which do not impact each other like mean and variance in normal distribution do?

texts:
I need to find a probability distribution to fit my data. My data has two important features, duration and activity count. Duration means how long one sequence lasts and activity count means the number of activities in one sequence.
I want to draw a curve, which should be (but not definitely necessary) like normal distribution. The height of the peak is related to the activity count. The breadth of the peak (confidence area) is related to the duration. In my context, more activity count means higher peak, but the breadth has no change. Also, longer duration means wider breadth, but the peak has no change.
In other words. peak is just related to the number of activities, while the breadth is just related to the duration.
So, you can tell the difference between normal distribution and the distribution I expected. For a normal distribution, variance will change both the breadth of the peak and the height of the peak, which is not what I want.

For example, on the above figure, the same height means that the number of activities are the same for two sequence. But one sequence has longer duration than the other, so the blue curve's peak is wider.
If one distribution cannot do this, can i use high dimensional distributions or the combine of multiple distributions to do it?
So, could anyone suggest a probability distribution for me?


texts: YOLO Pre-trained weights using Open Images Dataset V4

texts:
I want to use an object detection model for some use case.

I started with the YOLOv3, because I need to be able to perform detection on multiple images in less than few seconds (Even if it is not for real-time / video).

So, I successfully loaded and generated predictions using a keras model with pretrained weights.

The problem is that the pre-trained weights for this model have been generated with the COCO dataset, which contains very few classes (80).

What really surprises me is that all the pre-trained weights I can found for this type of algorithms use the COCO dataset, and none of them use the Open Images Dataset V4 (which contains 600 classes).

Is there a reason for that ? And if there is not, is there a place where I can find such pre-trained weights ?


texts: How to separate overlapping curves more effectively?

texts:
So, I am trying to create a Neural Network which will effectively separate 2 Gaussian curves with somewhat different means and standard deviations. My basic aim is, for some given input vector the Neural Network will select 2 points from the 2 Gaussian curves (selection based on the input vector) and compare them, and output a value depending on the Gaussian curve which gives a greater value.

Now, if I implement a simple Neural Network with 2 hidden layers the accuracy of my predictions is about 68.75% for 2 Gaussian curves with N(0,1) and N(0,2000) respectively and 100% for N(0,1) and N(500,1)  (my NN outputs 0, 1 and hence I can easily measure accuracy by true_positives/total_samples). Clearly more the overlapping curves, less the accuracy as the NN cannot distinguish between the 2 curves.

I have tried increasing the number of hidden layers, changing learning rate but still there is no improvement in accuracy. I am using back-propagation with momentum.

So my question is what structure, learning algorithm and modifications in the Neural Network will help me to effectively separate 2 curves which is kind of overlapping for a better accuracy?

Clarification 1: I have generated the data myself for testing purposes, in real situation I will only have an input/output pair from a black-box, and all I know the black-box follows some Gaussian distribution. Also the input vector has to be arranged in some sort of way, and then if we fit the curve to it, it turns out to be Gaussian.

Clarification 2: I am looking for possible hints. Any experienced person in this field might make a guess for some better methods. I know you cannot have a definite method, so suggestion of methods which might work are welcome.


texts: How to convert unstructured texts to structured data?

texts:
I would like to write an algorithm to convert unstructured texts (with contest descriptions) to structured data with the following fields:

contest start date (optional)
contest end date
main prize
additional prizes (optional)

I have hundreds of text examples, that could be used for model learning.

How to approach this task? Just in case this is important - the preferable language is Python. But I never worked on such tasks before.
What is the best practice to do that?


texts: How to delete a row if a values in a column is not NaN

texts:
In the given code it displays only the rows without NaN values but i want only the rows with NaN values in a provided column everything has to be removed

df1 = df1[pd.notnull(df1['column_name'])]


texts: BERT word embedings for finding word definition

texts:
Can BERT, GPT or other contextualised embedings be used for finding word definitions? What would be the most effective and not complicated approach for tackling a sample task as described below.

Map the meaning of the word 'bank' in the sentence "I was walking along the bank of the river" with one of the definitions listed in the WortNet database (or other word-sense lookup table).


texts:
I want to include a plot in my thesis to highlight regional differences in bicycle collision outcomes. Now there are several plot visualizations I could use, and I am not sure which one could be the most appropriate. Having a "scale free" y axis better highlights the differences between each region, but it may also confuse the reader.

I am aware that there might not be an optimal solution to this, but I would appreciate to hear your thoughts.(If you have a completely other suggestion for an appropriate plot than the ones below, I would also be happy to hear those).

These are the plots I have made:

Plot 1

Plot 2

Plot 3

Plot 4


texts: Best Python NLP library for supervised topic classification

texts:
I have a labeled dataset that I have ingested into a dataframe. It consists of news articles,
>>> df.columns

Index(['title', 'headline', 'byline', 'dateline', 'text', 'copyright',
       'country', 'industry', 'topic', 'file'],
      dtype='object')

where the text column contains the body (text) of the article and the topic column contains a list of associated topics.
I want to train a model from this dataset to predict the article topics. I was considering using transformers (https://huggingface.co/transformers/index.html) to do this, along with tensorflow, but I from what I know of transformers, it's not really good for this.
What would be the best NLP library to perform this task with high accuracy?


texts: What are features for state-action pairs in RL?

texts:
I read this answer:  What are features in the context of reinforcement learning?

But it only describes features for the state only in the context of cartpole, ie.   Cart Position, Cart Velocity, Pole Angle, Pole Velocity At Tip

On slide 18 here:  http://www.cs.cmu.edu/~rsalakhu/10703/Lecture_VFA.pdf

It states:

But does not give examples.  I started reading from p. 198 in Sutton's book for Value Function Approximation but also did not see examples for "features of state-action pairs" .

My best guess is for example in Cartpole-V1 (discrete action space) would be to add one more number to the tuple describing the state-action pair, ie. (Cart Position, Cart Velocity, Pole Angle, Pole Velocity At Tip, push_right) .
In the case of Cartpole I guess each state action pair could be described with a feature vector of length 3 where the final input for the tuple is either "push_left", "do_nothing", "push_right".

Would the immediate reward from taking one of the actions also be included in the tuples that form the state-action feature vector?


texts: How does ,the Mutlinomial Bayes's alpha parameter, affects the text classification task?

texts:
I would like to know how the alpha parameter, in Multinomial Bayes, affects the text classification task.

I know that this parameter is correlated to the algorithm's ability in classifying unseen words during training. How changes text classification using low or high values of alpha?


texts: Can we used both cross validation/nested cross validation technique and early stopping with patient at the same time?

texts:
Can we use both cross validation/nested cross validation technique and early stopping with patient at the same time? Using early stopping for each (training, validation) fold  and get best result of each  (training, validation) fold and finally get average result as usual?
I have read an article about this situation Machine Learning mastery But it is not convincing enough for me.


texts: Is it possible to get prediction intervals in sequenced data RNN forecasting?


texts: Help with deep learning for motorbike inspection

texts:
First of all, I am very new in machine learning and data science, so I am really sorry if my question is completely stupid.

I am doing an internship in machine vision, and people of my office want me to implement a Deep Learning model to inspect a motorbike after being fully assembled. Basically, they want to inspect different parts of the motorbike, and detect if there is a defect or not. However, before doing it in real life, they want to use a miniature of a motorbike in order to study the viability of this project.

This being said, I was thinking about creating one model for each region to be inspected, programing a robot with a camera to take pictures of these regions, and letting the DL models evaluate them. In addition, the DL models would be used for One-Class Classification to detect if the region inspected is OK or Not OK only by analyzing images labeled OK.

However, there are several problems that I will list as follows:

1 – We don’t have (any) data (yes, I told them it is stupid to try DL without any data);

2 – In internet, I could not find dataset of motorbike, much less specifically of the regions to be inspected;

3 – They want to inspect a specific model of motorbike, so they are asking me to do something very very specific, and I suppose that even if I find a dataset of the specific regions of different motorbikes, it will be kind of useless.

Finally, with all the conditions and problems that I mentioned above, it seems that it is impossible to do what they want, but I would like to ask you about it before giving up, because as I said, I am very new in this subject and I might be wrong. Could you give me your opinion/advice about everything I mentioned here?

Thank you very much!!


texts: How propagate the error delta in backpropagation in convolutional neural networks (CNN)?

texts:
My CNN has the following structure:

Output neurons: 10
Input matrix (I): 28x28
Convolutional layer (C): 3
feature maps with a 5x5 kernel (output dimension is 3x24x24)
Max pooling layer (MP): size 2x2 (ouput dimension is 3x12x12)
Fully connected layer (FC): 432x10 (3*12*12=432 max pooling layer flattened and vectorized)

After making the forward pass, I calculate the error delta in the output layer as:

$\delta^L = (a^L-y) \odot \sigma'(z^L) (1)$

Being $a^L$ the predicted value and $z^L$ the dot product of the weights, plus the biases.

I calculate the error deltas for the next layers with:

$\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) (2)$

And derivative of the error w.r.t. the weights being

$\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j (3)$

I'm able to update the weights (and biases) of $FC$ with no problem. At this point, error delta $\delta$ is 10x1.

For calculating the error delta for $MP$ , I find the dot product of $FC$ and the error delta itself, as defined in equation 2. That gives me an error delta of 432x1. Because there are no parameters in this layer, and the flattening and vectorization, I just need to follow the reverse process and reshape it to 3x12x12, being that the error in $MP$.

To find the error delta for $C$, I upsample the error delta following the reverse process of the max pooling ending with a 3x24x24 delta. Finding the hadamard product of each of those matrixes with each of the $σ′$ of the feature maps gives me the error delta for $C$.

But now, how am I supposed to update the kernels, if they're 5x5, and I is 28x28? $I$ have the error delta for the layer, but I don't know how to update the weights with it. Also for the bias, as it's a single value for the whole feature set.


texts:
Im a noob at tableau. I have a quick question

I have data that doesn't explicitly list the date. (There is no column labelled "date" )

However there are columns with the following headers:

"Income 2015", "Income 2016", "Income 2017" (the cells within these columns have financial data and do not have a "date")

Is there a way I can write a rule or do manually that will allow me to create dates (by year) based on these header names?

And then use the newly created dates to map to their respective columns?

So lets say I create the date with a value of 2017. Could I map this date to the column "Income 2017"? And make all the cells within this column also map to date 2017?

I hope this makes sense. Its been killing me trying to figure out if this is possible.

Thank you,


texts:
Hello and thank you in advance for any answer. I am building a NN for a multi class classification problem. I have pretrained my data through a Word2vec and generated a 300 dimension vector with values. I have 6 classes .. My data consist of 15000 rows(x300 dimensions). My first question is :

What is the number of units? Is it something that we can extract from theory?
Also i have managed to gain 0.85-0.90 training set accuracy but the validation set accuracy is always low: 0.22-0.25. I do not know what way to follow as I am newbie to these kind of stuff. My code is:

import numpy as np
import pandas as pd
import keras
from sklearn.preprocessing import Imputer
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense,Dropout
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
from keras import regularizers

dataset = pd.read_csv('word2vec.csv')

X = dataset.iloc[:, 0:300].values
y = dataset.iloc[:, 300].values

imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0)
imputer = imputer.fit(X[:, 0:300])
X[:, 0:300] = imputer.transform(X[:, 0:300])

counter = 0
for iterations in range(1):

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

    classifier = Sequential()

    classifier.add(Dense(units=70, kernel_initializer='uniform',kernel_regularizer=regularizers.l2(0.0001),
                activity_regularizer=regularizers.l1(0.001), activation='relu', input_dim=300))

    classifier.add(Dropout(0.3))

    # classifier.add(Dense(units=70, kernel_initializer='uniform', activation='relu'))

    classifier.add(Dense(units=6, kernel_initializer='uniform', activation='softmax'))

    classifier.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

    history = classifier.fit(X_train, y_train,validation_split=0.1, batch_size=10, epochs=100,shuffle=True)

    y_pred = classifier.predict(X_test)
    # y_pred = (y_pred > 0.1666)

    classifier.summary()
    counter+=1
    print(classifier.layers)
    print("Running RNN with Dropout Layer")
    print("Number of layers used: "+str(len(classifier.layers)))
    # list all data in history
    print(history.history.keys())
    # summarize history for accuracy
    plt.plot(history.history['acc'])
    plt.plot(history.history['val_acc'])
    plt.title('model accuracy')
    plt.ylabel('accuracy')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.show()
    # summarize history for loss
    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title('model loss')
    plt.ylabel('loss')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.show()


texts: Predict time series using NN trained with multiple TS

texts:
I'm trying to predict time series using RNN from keras and I have several time series to work with (these are sales from some stores, just usual 1D TS).
  Right now I can train one simple model using one TS and then predict this one TS using this one model, so it is just one model for one series. But it is not effective and I need some way to train ONE model with all the time series I have and then to predict new time series using this model.
I've read about multivariate time series but couldn't get it to work. Also I tried to aggregate the time series into one, but result of prediction wasn't good at all.
I am using keras with python3.


texts: Controlling number of channels in weight/kernel in tensorflow

texts:
To implement a specific function, I need "input_channels" number of kernels in my layer, each having only a single channel depth, and not depth = "input_channels". I need to convolve one kernel with one channel of the input, thus the output of the layer would have "input_channels" number of kernels.

Image attached for reference -

Thanks in advance for any help.

(if anyone wishes to know what all i have tried yet -
In the conv2d function of tensorflow, if I specify number of kernels = 1 to do this, then it will sum over all input_channels and number of output_channels will be 1, since it always initialises kernel depth = "input_channels".

Another option is to specify number of number of kernels = input_channels in conv2d function but this would create "input_channels" number of kernels of depth "input_channels", thus adding lot of complexity and incorrect implementation of my layer.

Yet another thing I tried was to initialise a kernel of volume (kernel_height, kernel_width, input_channels) and loop over the third dimension to convolve only a single input channel with a single kernel. But the tensorflow conv2d function requires a rank 4 kernel to work and gives the following error -

ValueError: Shape must be rank 4 but is rank 3 for 'generic_act_func_4/Conv2D' (op: 'Conv2D') with input shapes: [?,28,28], [28,28].

)


texts: What are the libraries in Deep Lerning and AI and How do we train a Machine Learning model into reinforcement using Python Programming?

texts: I am an aspiring Data Scientist with basic knowledge about Machine Learning libraries. I need to know about the different libraries used in training Machine Learning models with deep learning and neural networks. Please try explaining it from how deep learning and AI is evolved compared to basic machine learning models.


texts: Batch Normalization will disrupt multi-threading?

texts:
Question:

In a Feed Forward network, assume we have a Mini-batch of 64 examples.
Our layer $l$ contains 20 neurons
Because every neuron in layer will require Mean and Variance from the whole minibatch, we must  to run all 64 'examples' in parallel (is this true?). We must wait until each such 'example' will have the appropriate sum (for that particular neuron!) available across all these minibatches.

Wouldn't this break parallelism? Previously, each of our example-threads could have computed everything, up to the activation values for the entire layer $l$. However, now we have to:

Wait for all example-threads to finish computing the sums for our layer $l$ (hence, our speed slows down to match the slowest example-thread)
Pause multithreading (!) and compute mean & variance for the layer $l$
Apply it to the sums, resume example-threads, so each thread completes the job, by activating its corresponding adjusted sums

Do I miss a part of the puzzle, or it really has to be done like this?

$$$$
my understanding of Batch Norm:

Batch norm (link to original paper) makes the search space more even & simpler to navigate during forward & backward passes.

It is usually applied to the sum $Z$ going into activation function (although there is a lot of debate (1) (2)). This is done for every neuron in a layer.

in usual settings, the neuron's output on layer $l$ is $$H_{i} = \phi(W\cdot X +b)$$  where W is a vector of weights leading into our $i$'th unit, $X$ is a vector of outputs from previous layer $l_{-1}$ and $b$ is a bias

With Batch norm, we still have our usual non-activated sum for neuron $i$:

$$Z_i = W\cdot X$$

but also have our mean for that neuron across all minibatches (Just for that neuron!)

$$\mu_i = \frac{1}{m} \sum_{p}^m Z_{ip} $$

Similar, for our Variance as:

$$\sigma^2_i = \frac{1}{m}\sum_{p}^m (Z_{ip} - \mu_i)^2$$

Our non-activated normalized sum for neuron $i$ (here, $\epsilon$ is a tiny number, in case if variance is too small, to avoid division by zero):

$$Z_{norm} = \frac{Z_i - \mu_i}{\sqrt{\sigma^2_i + \epsilon}}$$

We give a network a chance to learn the actual distribution it needs, so it's different from default distrib. We introduce 2 learnable parameters, $\gamma$ and $\beta$, and network can use it to 'squash and shift' the distributions as it needs (even canceling-out the batch-norm if required).

$$Z_{adjusted} = {\gamma}Z_{norm} + \beta $$

Finally neuron's output is computed as:

$$H_i =  \phi (Z_{adjusted}) $$

Bias isn't included, because we already have $\beta$ for the whole layer $l$.


texts:
My understanding of Dynamic Time Warping is that the algorithm always requires calculation with each comparison/training series and that there is no way to extract the "essence" from a given training series in the form of coefficients, which could then be compared to coefficients for the test series similar to what might be possible with different wavelet transforms, Discrete Cosine Transform, etc.  In experimenting with Python's fastdtw package, DTW does appear to be somewhat slow (despite the name of the package).
What I'm wondering is whether there are any methods that produce coefficients that still preserve the dynamic warping aspect.  Perhaps there are certain modifications to wavelet transforms that could do the trick?


texts: Performing anomalie detection on a battery volatge using LSTM-RNN

texts:
I am trying to detect anomalies in a battery output voltage for one month.
I have the next data frame, as it is shown the data is collected each minute for each day so I have almost 1420 sample per day.

Should I use the 'time' or the 'date' column in my time series analysis?
I could not find an example like my situation all dataframes I found looks like the next :

It is all about a unique value for each day. Please any help is appreciated!
Thank you in advance!


texts: Rationale behind most published works in medical imaging trying to reduce false positives

texts: In medical image processing, most of the published works try to reduce false positive rate (FPR) while in reality, false negatives are more dangerous than false positives. What is the rationale behind it?


texts: How to iterate over the rows of a data frame?

texts:
I want to perform perform the following operations on a dataFrame in R

if(new$Period==0)
{
  print(" ")
  prevBal = new$EndingBalance

  print(new$LoanID)

  print(new$EndingBalance)

}else
{
  cat("else block execution")
  if(new$Period < 41)
  {

    if(new$NumberOfClaims>0)
    {
      cat("executing number of claims block")
      print((new$LossAmt)*(new$SimulationCount)/(new$NumberOfClaims)) #loss amount per claim
    }else
    {
      cat("executing else aftern no. of claims that prints 0")
      print(0)
    }
    print("end of if block point")
    print(new$BalanceInClaims) #balanceinclaims
    print(new$EndingBalance) #ending balance

  }
}

However, I get the warning in:

if (new$Period == 0) { :
  the condition has length > 1

and only the first element will be used.
How do I make sure this is executed row by row?


texts: Converting a speech recognition model from CNNs to GRUs

texts:
I am trying to convert the simple audio recognition example from TensorFlow to use GRUs instead of CNNs.
The idea is to classify an audio clip onto a set of 8 labels: ['go', 'down', 'up', 'stop', 'yes', 'left', 'right', 'no']
The original code builds a model as follows:
norm_layer = preprocessing.Normalization()
norm_layer.adapt(spectrogram_ds.map(lambda x, _: x))

model = models.Sequential([
    layers.Input(shape=input_shape),
    preprocessing.Resizing(32, 32),
    norm_layer,
    layers.Conv2D(32, 3, activation='relu'),
    layers.Conv2D(64, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Dropout(0.25),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(num_labels),
])

The input shape is (124, 129, 1) - single channel spectrograms with 124 time steps and 129 frequency bins. The X data is the spectrogram and Y data is an integer label which is an index into the labels array.
I have tried converting the above into using GRUs as follows:
for spectrogram, _ in spectrogram_ds.take(1):
  input_shape = spectrogram.shape #(None, spectrogram.shape[0], spectrogram.shape[1])
print('Input shape:', input_shape)
num_labels = len(commands)

model = models.Sequential([
    layers.Input(shape=input_shape),

    # Step 1: CONV layer (≈4 lines)
    layers.Conv1D(196, 15, strides=2, input_shape=input_shape),
    layers.BatchNormalization(),
    layers.Activation('relu'),
    layers.Dropout(0.8),

    # Step 2: First GRU Layer (≈4 lines)
    layers.GRU(128, return_sequences=True, input_shape=input_shape, reset_after=True),
    layers.Dropout(0.8),
    layers.BatchNormalization(),

    # Step 3: Second GRU Layer (≈4 lines)
    layers.GRU(128, return_sequences=True, reset_after=True),
    layers.Dropout(0.8),
    layers.BatchNormalization(),
    layers.Dropout(0.8),

    # Step 4: Time-distributed dense layer (see given code in instructions) (≈1 line)
    layers.TimeDistributed(layers.Dense(num_labels, activation = "sigmoid"))

])

The input shape in this case is (m, 124, 129). The Conv1D layer reduces the time steps from 124 to 55.
In this case, the X data is still the spectrogram data. For Y data, I had to replicate the label index over 55 time steps. So Y has shape: (m, 55, 1).
The training is done as follows:
model.compile(
    optimizer=tf.keras.optimizers.Adam(),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy'],
)

EPOCHS = 10
history = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=EPOCHS,
    callbacks=tf.keras.callbacks.EarlyStopping(verbose=1, patience=2),
)

The issue is that I am getting a very low accuracy for the GRU model compared to the CNNs. The GRU training is also very slow. My feeling is that I have not set up the model correctly - especially I am not sure if I have the Y data setup correctly for the GRU.
I'd appreciate any insights in setting up this model correctly. Thanks!


texts: What are some good fields to research in data science?

texts: I am recently working on my thesis. I had started with recommender systems, but figured out that there is nothing more to optimize in this field since researchers have found around 8.5 RMSE for netflix data-set. Thus I am not going to reach a lot in this field. So I am wondering what are some good field to research on in machine learning or data mining? Or what are some novel and famous fields that researchers are interested in nowadays? I am personally interested in interconnection between nutrition and machine learning. I will appreciate it if anyone can point me in the right direction.


texts:
Here is the GitHub link to the most recent data.table benchmark.
The data.table benchmarks has not been updated since 2014. I heard somewhere that Pandas is now faster than data.table. Is this true? Has anyone done any benchmarks? I have never used Python before but would consider switching if pandas can beat data.table?


texts: Neural net layer that preserves spatial information

texts:
I'm attempting to use a neural network as a kind of interpolator for a high-dimensional function.

We're doing this to circumvent the need for a physical model that calculates this function exactly, but is expensive to run. We have a dataset of past runs of this physical model that we can now use as the basis for interpolation.

The input is a set of 28 parameter values.

The output we're trying to estimate consists of the shape of 432 periodic functions (i.e. value of 432 different functions along the circumference of a circle). We know that the output is uniquely determined by the 28 input values; there's no noise.

For example, here is one of these periodic functions. For this example, it's only sampled with 12 points (though in production we will be sampling it more finely, perhaps 36 points around a circle).

My current neural network Keras implementation is:

Input layer: 28 units (one per input parameter)
One hidden layer: fully connected, 1000 units
Output layer: fully connected, 432 periodic functions × 12 samples = 5184 units

As illustrated above, this works okay (even before optimizing hyperparameters).

But I'm concerned that with these fully-connected layers, I'm treating the 5184 outputs as unrelated numbers, completely disjointed from one another. In reality, there's some spatial relationship between the 12 samples describing each of the 432 underlying functions in the output. But I'm just throwing this spatial information away.

Is there another kind of layer which could take advantage of this spatial relationship, and thus lend itself better to this kind of regression?

If I were going the other way round, i.e. from spatial information to distinct values, I would consider convolution layers (perhaps with a periodic boundary). But what about this reverse case?


texts:
I am using tensorflow-serving to write a server to consume models in production.
I have a question about consuming the service by clients: does tensorflow-serving support a REST API? Is there is anyway to modify it?

I have checked several github projects: here, here, and here.


texts: How to build a classifier with a rejection class

texts:
Let's say I need to build a food classifier, and I want a rejection class for the inputs that are not food.

What is the best way to do that?
Should I just add a new class label that includes pictures of anything but food? Is there a best way to chose which pictures I should use?


texts: Natural Language Processing (NLP) in .net environment

texts:
I'm developing a project on .net (using web API and MVC). now I have to add another module which users can make queries of Data we have in the DB and retrieve the relevant answer.

Is their a easy way to Do NLP developments in .net
If yes, It would be a great help if someone can give a brief idea on how to get started

any suggestion is greatly appreciated. Thank you.


texts: How to visualize grouped clusters in a graph

texts:
I was trying to use Gephi, which seems to have a 'group' feature in some of the WIKI explanations, but actually does not appear in my GUI. There is a tool that creates groups from partitions, but it ignores the node attributes when creating those groups, and importantly removes the actual name of the partition and picks some random node or edge name instead.

I also tried collapsing the graph in networkx using quotient_graphand making the partitions manually based on the node attribute. However this also removed the attributes and made subgraphs instead of just reducing it.

Then I tried cytoscape, which had the autoannotation plugin. It was pretty close, although did some weird name changes as it wouldnt just let me select the node attribute name to make the groups. However after I was able to collapse the nodes, I am not able to use the layouts well.

Is there a simple approach to this, or something I am missing in the ways that I tried?


texts: Convolutional Neural Network: learning capacity and image coverage

texts:
I was looking through a CNN tutorial and towards the end they refer to learning capacity and image coverage during network learning diagnostics

What do those 2 terms mean in the context of a convolutional neural network?


texts: How to generate fixed number of superpixels?

texts:
A lot of work regarding Graph Neural Networks require fixed number of nodes. In the case of image processing using graph, the image representation is often super pixels (like in this work https://arxiv.org/pdf/2002.05544.pdf)
The other example is in this work (https://arxiv.org/pdf/1611.08402.pdf) with MNIST Superpixels which also ends u having same number of superpixels.
A lot of algorithms including recommended SLIC doesn’t follow this and generate different number of superpixles.
How to handle this? Are there some implementations which generate same number of superpixels in place or there is other mechanism used?


texts: Adding a feature increased amount of transaction however how do you prove it is not simply due to seasonal changes?

texts:
I've been always curious about similar topic however still do not fully understand it.
Say in a car sharing company I am implementing promotion A from Sept ~ Dec. I want to know whether promotion had an effect on increased number of car shares. However even w/o promotion A there will be increased number of car shared in winter times then in summer time.
As a result total number of car shares increased during promotion A compared to previous months. How do I prove that promotion A has an effect?


texts:
I have a dataset with 77 different labels. Each sample has one or more of these labels.

I did some data analysis and found out that the dataset is highly imbalanced - there are a large number of examples that have a particular label, whereas the other labels don't occur so frequently across the data samples.

I'm trying to use SMOTE to synthesize new data samples for the minority labels but apparently, imblearn's SMOTE doesn't support multi-label data. Is there an alternative to SMOTE that I can use for multilabel classification, or should I treat my problem as 77 different binary classification problems, and apply SMOTE on each iteration separately?


texts: Dimensionality reduction convolutional autoencoders

texts:
I don't understand how convolutional autoencoders achieve dimensionality reduction. For FFNN based autoencoder, the reduction is easy to understand: the input layer has N neurons, and the hidden ones have M neurons, where N is greater than M.
Instead, in a convolutional autoencoder, the input image is wide and thin, and it becomes small and thick. It results in an amount of information that is greater than the initial ones.
I report a practical example to explain better what I mean:
# INPUT: 28 x 28 x 1 (wide and thin)

conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img) #28 x 28 x 32
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1) #14 x 14 x 32
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(pool1) #14 x 14 x 64
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2) #7 x 7 x 64
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool2)

# OUTPUT: 7 x 7 x 128 (small and thick)

In this example, we start from a 28x28 single-channel image (784), and the encoder output will be 7x7x128 (6272). Is this a dimension reduction?


texts: How to make custom callback in keras to generate sample image in VAE training?

texts:
I'm training a simple VAE model on 64*64 images and I would like to see the images generated after every epoch or every couple batches to see the progress.

when I train the model I wait until the training is done and then I look at the results.

I tried to make a custom callback function in Keras that generates an image and saves it but couldn't do it. is it even possible? I couldn't find anything like it.

it would be awesome if you refer me to a source that explains how to do so or show me an example


texts: How does Sigmoid activation work in multi-class classification problems

texts: I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I don't understand is how does it work?


texts: What is the Q function and what is the V function in reinforcement learning?

texts:
It seems to me that the $V$ function can be easily expressed by the $Q$ function and thus the $V$ function seems to be superfluous to me. However, I'm new to reinforcement learning so I guess I got something wrong.

Definitions

Q- and V-learning are in the context of Markov Decision Processes. A MDP is a 5-tuple $(S, A, P, R, \gamma)$ with

$S$ is a set of states (typically finite)
$A$ is a set of actions (typically finite)
$P(s, s', a) = P(s_{t+1} = s' | s_t = s, a_t = a)$ is the probability to get from state $s$ to state $s'$ with action $a$.
$R(s, s', a) \in \mathbb{R}$ is the immediate reward after going from state $s$ to state $s'$ with action $a$. (It seems to me that usually only $s'$ matters).
$\gamma \in [0, 1]$ is called discount factor and determines if one focuses on immediate rewards ($\gamma = 0$), the total reward ($\gamma = 1$) or some trade-off.

A policy $\pi$, according to Reinforcement Learning: An Introduction by Sutton and Barto is a function $\pi: S \rightarrow A$ (this could be probabilistic).

According to Mario Martins slides, the $V$ function is
$$V^\pi(s) = E_\pi \{R_t | s_t = s\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s\}$$
and the Q function is
$$Q^\pi(s, a) = E_\pi \{R_t | s_t = s, a_t = a\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s, a_t=a\}$$

My thoughts

The $V$ function states what the expected overall value (not reward!) of a state $s$ under the policy $\pi$ is.

The $Q$ function states what the value of a state $s$ and an action $a$  under the policy $\pi$ is.

This means,
$$Q^\pi(s, \pi(s)) = V^\pi(s)$$

Right? So why do we have the value function at all? (I guess I mixed up something)


texts: Intuitive reasoning behind inverted dropout in neural networks

texts:
I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as follows: let's say we decide on a $0.5$ probability of dropping any particular node from the first hidden layer onwards. Let's also assume that we're using linear activation for now with no bias.

The activations in the second hidden layer will be linear functions of activations in the first hidden layer and if the dropout probability is $0.5$, the expected value of the activation of any node in the second hidden layer is half of the "actual" magnitude (by "actual" magnitude, I mean the magnitude of the activation of the node in the full (no dropout) neural network).

Again, since the nodes in the second hidden layer have $0.5$ probability of being dropped out, the expected magnitude of the third hidden layer activations will be $0.5\times 0.5$ of the "actual" magnitude, and so on. This effect gets carried forward through the layers all the way to the output layer, the expected magnitude dropping by a factor of $0.5$ in each layer.

Thus the final output activation value and hence $\hat y$ will be grossly underestimated and the loss will be overestimated. Thus we need to scale up the activation values of each layer before performing a forward propagation step.

Is this correct or have I missed something or got something completely wrong? Also, are there any references that give a rigorous mathematical explanation of the need for scaling up in dropout, and why dropout has a regularizing effect?

Thanks!