texts: Is converting a categorical value into numerical needed to find a correlation?

texts:
I have a small dataset of 1300 observations x 20 features. They are all numerical but one, which is categorical; this was calculated independently and relates to each observation in any case.

I'm now attempting to find the correlation of each features in my dataset, but a simple dataframe.corr() would omit the categorical from the calculation.

I have two choices as far as I can see:

Do not consider the categorical value, but this means not being able to suggest whether the adoption of internal processes from which that feature infers the value are optimal or not
Convert into a numerical

The categorical value looks like the school grading system:

 A: higher
 B: ...
 ...
 E: low

I don't think that converting into a numerical would result in a loss of magnitude so long I how what that conversion has been made.
But here's my crux.

Should I do something like: A = 1 & E & 5 .. or A = 5 & E = 1?
Would the two different values eventually affect the correlation process in the end?

I've been seeing minimal differences.
For instance, on the same dataset with the A starting at 1 I got Rating correlated to my Y variable at -0.33; when A starts at 5 it returns -0.32.
What I noticed is the correlation varies, and goes positive the more I refine the dataset.

Also, do consider I am also after using this dataset to later do some linear regression, and calculate the RMSE.

Any advice is welcome.

UPDATE:

I was able to further play around with the dataset, and forked it in two way, replacing the Rating score in two ways:

With {'A': 1, 'B': 2, 'C': 3, 'D': 4, 'E': 5}
With {'A': 5, 'B': 4, 'C': 3, 'D': 2, 'E': 1}

The results are NOT what I would have expected (opposed values), which means I am now more confused than before.

Dataset below for you to test:

    Index Ranking   Rating Correlation  # Results   Label
    0   1   0.064138    840 PKW_A1
    1   3   0.087673    245 PKW_A1
    2   5   -0.028258   111 PKW_A1
    3   7   0.017542    117 PKW_A1
    4   9   -0.249403   77  PKW_A1
    5   11  -0.138552   51  PKW_A1
    6   13  -0.090198   41  PKW_A1
    7   15  -0.333333   18  PKW_A1
    8   17  -0.076830   17  PKW_A1
    9   19  -0.113594   24  PKW_A1
    10  1   0.027015    840 PKW_A5
    11  3   0.116202    245 PKW_A5
    12  5   0.134111    111 PKW_A5
    13  7   0.094221    117 PKW_A5
    14  9   -0.070592   77  PKW_A5
    15  11  -0.127137   51  PKW_A5
    16  13  -0.275387   41  PKW_A5
    17  15  0.092450    18  PKW_A5
    18  17  0.055994    17  PKW_A5
    19  19  0.081427    24  PKW_A5


texts: I have one problem regarding segmentation. I would like to put more concentration on the boundary instead of the interior of segmented part. Is that possible to do using Tensorflow 2? For example, if I want to do some card segmentation, I would like to have boundary to be ok more than interior. Which loss function to use and which metric in that direction?


texts: How to incorporate keyboard positions on character level embeddings?

texts:
I am working with NLP and have character level embeddings.

I have embeddings learned from Wikipedia text.

Now, I want to learn embeddings from chat data (where misspellings and abbreviations are way more common). Usually, the character n doesn't follow from character b, however, during texting, this can be common because they are close together on the keyboard, and a misspelling occurs.

So, my questions is: what are strategies to incorporate character keyboard position information to a traditional character level embedding?

Note: it can be assumed that only QWERTY keyboards exist.


texts: Which of the algorithms in the current literature for contextual bandits can be implemented for online learning and which ones can't? I'd really appreciate it if someone could provide a link to papers too! Thanks for the help!


texts: Approach to semantic similarity between documents

texts:
I was wondering what approach people would take, or point me in the right direction on this challenge I have set myself. I am pretty new at this, I have covered some area but want to expand my skillset.

Say you have an abstract from a research paper, which is a summarised form of information of a larger document, can you calculate, from a list of papers, which research paper this abstract belongs to?

Please note: I am not asking how to summarise the research paper. Also, note that the abstract information doesn't necessarily take the same form as the research paper, but is semantically similar.

Would you, encode both datasets with a something like doc2vec to try to get the semantic meaning of the texts and then use cosine similarity?

Would the semantics of the numbers used in these papers get lost in the vectorizing?

IN this case, would a custom encoder work the best, or do you think USE or doc2vec would fair better?

Forgive my basic questions, I just wanted to explore things before I started coding!


texts:
In linear regression we use the following cost function which is a convex function:

We Use the following cost function

in logistic regression because the preceding cost function is not convex whenever the hypothesis (h) is logistic function. We have changed the equation of cost function to have a convex shape to find its global (the only one which exists). There is a fact that I can not understand. In Multi Layer Perceptrons ANNs I have seen a lot that they can be stuck in local minimums. Why is that? We have used this cost function for each perceptron and gotten the rules for updating the values for the weights in back propagation algorithm; So why do we stuck?


texts: Using Machine Learning to Predict Temperature

texts:
I am a beginner in ML and I want to create a smart thermostat, that after collecting enough data from the interaction with the user, it will start to set the home temperature by itself.

What I got so far is the hardware prototype that lets the user set the temperature, and in the same time it posts the Environment and the UserSetTemperature to ThingSpeak (to easily store the data for later access)

The other part is a python algorithm that gets the data from ThingSpeak and it converts it into a Pandas DataFrame.

The data frame looks like bellow:

timeStamp                      environment_temp    user_set_temp

2018-05-27T00:12:43Z               20            21
2018-05-27T00:17:27Z               20            22
2018-05-27T00:17:59Z               20            24
2018-05-27T00:20:01Z               20            21
2018-05-27T00:23:14Z               20            24
2018-05-28T09:39:07Z               20            22
2018-05-28T10:40:17Z               20            23
2018-05-28T20:12:47Z               20            25
2018-05-28T20:14:16Z               23            25
2018-05-30T20:29:30Z               18            24

And here is where I got stuck. I don't know how to use this data with the ML libraries in order to make predictions on how the temperature should be set when the environment temperature is x.

I tried to use the sklearn train_test_split() and LinearRegression(), but with no significant result. I really don't know how to use this data

Every suggestion will be highly appreciated!!


texts: model.score and r2_score giving different values for a regression model

texts:
I am build a linear regression model and a decision tree model using sklearn. I want to compare the performance of these two models, I have calculated the r2_score for both the models. I have calculated the model.score for both the values. I am confused which is a better metric to compare the performance of these models. Also what does model.score gives?

from sklearn.metrics import r2_score
score_DT = r2_score(y_pred_DT,y_test)

dt_score = regressorDT.score(X_test,y_test)


texts:
I have a data set with 3 independent variables and 1 dependent variable. The dependent is play_golf, the independents are Humidity, Pending_Chores, Wind.
   I want to aggregate the probability of playing golf based on those rules of multiple trees. So again, play_golf (this is the dependent value), and there are three independent variables Humidity(High, Medium, Low), Pending_Chores(Taxes, None, Laundry, Car Maintenance) and Wind (High, Low).  A rule would be like (IF humidity = "High" AND pending_chores = "None" AND Wind = "High" THEN play_golf = 77% ).

I was thinking that random forest would weight the rules somehow on the collection of trees and give a probability.  But if that doesn't make sense, then can you just tell me how to get the decision rules with one tree and I will work from that. I believe am I talking about a decision tree vector? I'm not sure.


texts:
My question is in regards to an MIT course example.

The instructor delves into the backpropagation of this simple NN.

I have two questions.

Why do we seem to disregard the weights of the second layer (in blue)?
The red circle, should this not be $\partial y_2$ (followed by $\frac{\partial y_2}{\partial P_3}$)?


texts: Overall AUC higher than all "stratified" AUCs

texts:
For one of my binary classification models, I have observed this (Simpson's Rule-esque) paradox.  The AUC on the test set as a whole is 0.8.
Gender is one of the model's features.  So I decided to produce a "bias" report, for which I calculated AUCs for each of the Male and Female subgroups.  But I noticed that each of these AUCs is around 0.7.  How is this possible, given that the overall test AUC is 0.8?  (In my dataset, every data point belongs to either the Male or the Female subgroup.)  I don't expect the overall AUC to simply be a (weighted) linear combination of AUCs for the individual strata.
I'm hoping to get both a technical/mathematical answer and a high-level explanation.  Please let me know if any further information is needed (if you think I should plot the overall, Male, and Female ROC curves, for instance).  Thank you!


texts: High Cross Validation Score on Training Set, High Score on Test Set, But Low Score on Kaggle?

texts:
I've been trying to complete this regression task on Kaggle. As usual they gave a train.csv(with response variable) and a test.csv (without response variable) file for us to train the model and compute our predictions, respectively.
I further split the train.csv file into a train_set and test_set. I use this subsequent train_set to train a list of models which I will then shortlist to one model only based on 10-fold cross validation scores (RMSLE) and after hyperparameter tuning.  Now I have one best model, which is Random Forest (with best hyperparameters) with an average RMSLE score of 0.55. At this point I have NOT touched the test_set.
Consequently, when I train the same exact model on train_set data, but evaluate its result on test_set (in order to avoid overfitting the hyperparameters I have tuned), it yields an RMSLE score of 0.54. This is when I get suspicious, because my score on test_set are slightly better than the average score of the train_set (test_set results are supposed to be slightly worse, since the model hasn't seen the test_set data, right?).
Finally, I proceed to submit my results using the same model but with the test.csv file (without response variable). But then Kaggle gave me an RMSLE score of 0.77, which is considerably worse than my cross validation scores and my test_set scores!
I am very frustrated and confused as to why this would happen, since I believe I've taken every measure to anticipate overfitting my model. Please give a detailed but simple explanation, I'm still a beginner so I might not understand overly technical terms.


texts: In a convolutional neural network (CNN), when convolving the image, is the operation used the dot product or the sum of element-wise multiplication?

texts:
The example below is taken from the lectures in deeplearning.ai shows that the result is the sum of the element-by-element product (or "element-wise multiplication".  The red numbers represent the weights in the filter:
$(1*1)+(1*0)+(1*1)+(0*0)+(1*1)+(1*0)+(0*1)+(0*0)+(1*1)
= 1+0+1+0+1+0+0+0+1
= 4 $

HOWEVER, most resources say that it's the dot product that's used:

"…we can re-express the output of the neuron as , where is the bias
term. In other words, we can compute the output by y=f(x*w) where b is
the bias term. In other words, we can compute the output by
performing the dot product of the input and weight vectors, adding in
the bias term to produce the logit, and then applying the
transformation function."

Buduma, Nikhil; Locascio, Nicholas. Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms (p. 8). O'Reilly Media. Kindle Edition.

"We take the 553 filter and slide it over the complete image and
along the way take the dot product between the filter and chunks of
the input image.  For every dot product taken, the result is a
scalar."

The best explanation of Convolutional Neural Networks on the Internet!

"Each neuron receives some inputs, performs a dot product and
optionally follows it with a non-linearity."

CS231n Convolutional Neural Networks for Visual Recognition

"The result of a convolution is now equivalent to performing one large
matrix multiply np.dot(W_row, X_col), which evaluates the dot product
between every filter and every receptive field location."

CS231n Convolutional Neural Networks for Visual Recognition
However, when I research how to compute the dot product of matrics, it seems that the dot product is not the same as summing the element-by-element multiplication. What operation is actually used (element-by-element multiplication or the dot product?) and what is the primary difference?


texts:
... assuming that tables allowed for nested non-primitive values.
I'm trying to think of any instance where data cannot be represented as tables (or a series of tables)... and I can't think of any. However, I'm skeptical of myself on this one... its seems too simple to defined a pseudo-law which state "all data can be represented as tables".
Can anybody think of instances where this isn't true?


texts:
I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words.  I want to find bi-grams using nltk and have this so far:

bigram_measures = nltk.collocations.BigramAssocMeasures()
articleBody_biGram_finder = df_2['articleBody'].apply(lambda x: BigramCollocationFinder.from_words(x))

I'm having trouble with the last step of applying the articleBody_biGram_finder with bigram_measures. I've tried multiple iterations of lambda with list comprehension but am getting nowhere.

my most recent attempts:

df_2['articleBody_scored'] = score_ngrams(bigram_measures.raw_freq) for item in articleBody_biGram_finder

df_2['articleBody_scored'] = articleBody_biGram_finder.apply(lambda x: BigramCollocationFinder.score_ngrams(bigram_measures.raw_freq))


texts: Do I need to transform time with sin/cos if I'm using decision tree algorithms?

texts:
According to this post, the time on a 24-hour clock should be decomposed into separate periodic components:

https://ianlondon.github.io/blog/encoding-cyclical-features-24hour-time/

before feeding it into an extratrees algorithm, as 24-hour clock numbers are not ordinal (1 comes right after 24...).

My question is, when using decision tree algos, is this necessary?

My lay understanding is that tree algos work in feature space, are non-linear, and perform arbitrary splits, hence this is an unnecessary operation?


texts: Is there any way to create column based on some previous column in PANDAS dataframe?

texts:
Given: I have Pandas Dataframe as shown below

| Employee_ID | Manager_ID |
|:-----------:|:----------:|
| E068        | E067       |
| E071        | E067       |
| E229        | E069       |
| E248        | E144       |
| E226        | E223       |
| E236        | E241       |
| E066        | E001       |
| E067        | E001       |
| E144        | E001       |
| E223        | E001       |
| E069        | E066       |

Problem Statement:

This problem is to identify the Head of Manager by using Employee and their Manager data.

About:

We have an Employee ID and their Manager ID. Please note that Manager ID are from Employee ID. Since each manager has one Manager above their level.

STEPS:

First, we'll take all UNIQUE ID in Manager ID column.
Then for each ID from Manager ID column, we will look for their respective Manager ID(Manager)
Then we will create a new column say Level 1 we will put manager for each Manager ID on their respective cell.
Similarly, we will repeat the above 3 processes again till there is no Manager ID for that particular ID.
This way we can identify the Head of Manager.

I am able to solve the problem in EXCEL.
By using =IFERROR(VLOOKUP(C2,$A:$B,2,FALSE),"")

But this approach lead me to create new column in excel for each level hierarchy. And putting the formula on first cell of that particular column and then dragging the result for each manager

But incase of big companies there would be n no. of level of hierarchy. So creating the new column in excel for each level of hierarchy would be time consuming task. Hence, I am looking for an optimal solution.

Expected Output:

| Employee ID | Manager ID | Level 1 | Level 2 | Head of Manager |
|:-----------:|:----------:|---------|---------|-----------------|
| E068        | E067       | E001    |         | E001            |
| E071        | E067       | E001    |         | E001            |
| E229        | E069       | E066    | E001    | E001            |
| E248        | E144       | E001    |         | E001            |
| E226        | E223       | E001    |         | E001            |
| E236        | E241       |         |         | E241            |
| E066        | E001       |         |         | E001            |
| E067        | E001       |         |         | E001            |
| E144        | E001       |         |         | E001            |
| E223        | E001       |         |         | E001            |

  The Employee ID column contain UNIQUE ID while Manager ID contain DUPLICATES ID.

Thank you for your time and consideration.


texts: Consolidating values for predictive modelling through Random Forest

texts:
I am working on developing a predictive model using Random Forest. There are a lot of users that log in to the site but only a fraction of them actually monetize on that day. I am trying to predict the probability of monetization of a user during a login. The predictor variables have been selected. Now the issue is that one user can log in on multiple days and have different values of predictor variables and the label (Monetized/ Not Monetized) for each log in.

If I try to train the data on the last 30 days' data, some users would occur multiple times and others would only occur one time. This may lead to a higher weight-age being given to the users who logged in on more days.

How can I ensure that equal weightage is assigned to all the users even though the users might have different numbers of logins and hence different counts of data points.


texts: In XGBoost, how to change eval function and keeping same objective?

texts:
I want to keep objective as "reg:linear" and eval_metric as customised rmse as follows.

def customised_rmse(preds, dtrain):
    N = len(preds)
    preds = np.array(preds)
    actual = np.array(dtrain.get_label())
    crmse = np.sqrt(np.sum(np.power((preds+1)*1.0/(actual+1) - 1.1, 2))/N)
    return "custom-rmse", crmse

When I ran, train function as follows ->

model = xgb.train(param_list, xgb_train, num_rounds, watchlist, None, customised_rmse, early_stopping_rounds=30)

Output I am getting is this ->

[0] train-rmse:15.1904  val-rmse:15.2102    train-custom-rmse:0.607681  val-custom-rmse:0.610993
Multiple eval metrics have been passed: 'val-custom-rmse' will be used for early stopping.
Will train until val-custom-rmse hasn't improved in 30 rounds.
[1] train-rmse:14.4936  val-rmse:14.5103    train-custom-rmse:0.588831  val-custom-rmse:0.589902

and so on ..

My question is, it is getting optimised using "rmse" or "custom-rase" or both ?? What I have to do to remove "rmse" as it comes by default with "reg:linear" ??


texts: Which regularization in convolution layers (conv2D)

texts:
I am using Keras for a project. I would like to know if it makes any sense to add any kind of regularization components such as kernel, bias or activity regularization in convolutional layers i.e Conv2D in Keras.

If yes, then which regularization is most useful for conv2d layers

Kernel
Bias
Activity

As explained here the regularization techniques are useful for the fully connected(dense) layers. Any such intuition/logic for conv2D?


texts: Linear correlation and XGboost regression for time series

texts: I am working with sales time-series data, I have a history of 9 years of monthly data. I am trying to forecast sales for the next 12 months. I am using XGboost regression to build multivariate time series, forecasting model. The problem I am facing is, I can see a strong correlation, Pearson correlation, between my stationery sales and stationary temperature for some month and some years e.g. the strong correlation between my stationary sales which is my target and temperature can be observed between 2004 and 2007 in May, June, and July. I have decomposed the data, removed the trend and seasonality and kept the level only. However XGboost model is not showing this kind of strong correlation between the two. I am feeding original time series of 9 years, non-stationary but providing some trend information but still can't see the good forecast for the 12 months in general and for those months in specific. Could anyone tell me why strong individual strong correlations are not caught by the model?


texts: I'm currently on the reading stage of the deployment of an RTB system. I've seen the problem of a cold start (having no initial guess of how to bid) in several papers, but I haven't really seen it be formally addressed. Could somebody point me to some recent literature on the topic, or briefly discuss how they've faced it? Thanks!


texts: tensorflow categorical data with vocabulary list - Expected binary or Unicode string, got [0,1,2,...]

texts:
I'm brand new to machine learning (having just completed the google machine learning crash course) and thought it would be good to try my hand at a Kaggle competition as a good starter to some real problem solving. I'm using tensorflow and Python 3, all up to date (the kaggle online jupyter notebook)

The data is formatted in a dataframe like below

|Identity | Cuisine | Ingredients                |
|---------|---------|----------------------------|
|1        | italian | [beans, milk,..., tomatoes]|
|2        | indian  | [chicken, curry leaf,...]  |

I have made a vocabulary list generator to create a vocabulary set, and replace instances of those words in the ingredients array with the index of the ingredient in the vocabulary set, so my original data looks like below.

|Identity | Cuisine | Ingredients |
|---------|---------|-------------|
|1        | italian |[0, 1,..., 4]|
|2        | indian  |[5, 6,...]   |

I seperate the labels (cuisine) and the features (ingredients) into 2 seperate dataframes for ease, and I am using a tf.feature_column.categorical_column_with_vocabulary_list and subsequent tf.feature_column.indicator_column for the ingredients array.

I now however have an issue with my model not being able to read the ingredients column, and get the error

TypeError: Expected binary or unicode string, got [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

my input function is as follows

def input_fn(features,labels,batch_size,num_epochs=None,shuffle=True):
    ds = Dataset.from_tensor_slices((features,labels))
    ds = ds.batch(batch_size).repeat(num_epochs)

    if shuffle:
        ds = ds.shuffle(10000)

    feature_batch, label_batch = ds.make_one_shot_iterator().get_next()
    return feature_batch, label_batch

which is fed into a simple function as below

training_func = lambda: input_fn(training_example,training_target,batch_size)
validati_func = lambda: input_fn(validation_example,validation_target,batch_size)

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
optimizer = tf.contrib.estimator.clip_gradients_by_norm(optimizer, 5.0)

classifier.train(
    input_fn=training_func,
    steps=steps_per_period
)

My urgent question is how do I fix this TypeError

In addition I also want to know if there a best practice for handling this format of data? (and if there is any built-in functionality to handle this)


texts: document similarity when the document size less than 30 tokens?

texts:
I was solving a problem to compare 3 million, 2018 documents against the 2019 documents. There are three text attributes to be compared from one item against the other. I used Latent Semantic Indexing (LSI) for one variable, containing about 5 word tokens, with reasonable performance.

What will be the minimum document size for LSI/LDA for a multivariate [Item Description (text) (10 tokens), Item specification (text) (5 tokens)] problem to compute document similarity?
I had used cosine similarity, String Distances to measure the closeness of
match of 2018 Description and 2019 description. Are there any other statistical methods to evaluate the Model?


texts: Coefficients of Linear regression for minimizing MSE

texts:
(I asked this in mathematics site, but nobody responded, it seems the whole problem is more related to data science than math.)

In a regression problem, loss function is:

$$L(a,b) = {\sum_{i=1}^n (y^i - (ax^i +b))^2})$$

In order to minimize this value, we need to set the derivative of L with respect to each of its parameters, equal to zero.

Hence, $\frac{dL}{db}$ would be $y^- + a \cdot x^-$
But what would $\frac{dL}{da}$ be?

$$\frac{\sigma L(a,b)}{\sigma a} = {2 \cdot \sum_{i=1}^n (y^i - (ax^i +b)}) \cdot \frac{\sigma \sum(y^i - (ax^i +b))}{\sigma a}$$
$$\frac{\sigma L(a,b))}{\sigma a} = {2 \cdot \sum_{i=1}^n (y^i - (ax^i +b)}) \cdot -\sum(x^i)$$
$$\frac{\sigma L(a,b))}{\sigma a} = {2\sum(x^i) \cdot \sum_{i=1}^n (y^i) - 2\sum(x^i) \cdot \sum_{i=1}^n (ax^i)  - 2\sum(x^i) \cdot \sum_{i=1}^nb}$$
$$\frac{\sigma L(a,b))}{\sigma a} = {2\sum_{i=1}^n (y^i \cdot x^i) - 2\sum_{i=1}^n (ax^i \cdot x^i)  - 2\sum_{i=1}^n x^i \cdot b}$$
$$\frac{\sigma L(a,b))}{\sigma a} = {2\sum_{i=1}^n (y^i \cdot x^i) - 2\sum_{i=1}^n ((ax^i +b) \cdot x^i)} = 0$$

How this would be equal to $\frac{cov(x, y)}{\sigma^2x}$


texts: Can the size of a pooling layer be learned?

texts:
As far as I understood it, the pooling layer doesn't learn anything. It has several parameters, most important its pool_size and stride (see Lasagne documentation for more), but none of those is learned.

Is it possible to learn these two (or one of those) parameters? Are there papers about it?

(I would guess that it is not possible to add this to the objective function in a meaningful way... but I'd like to see if people thought about it.)


texts:
I am trying to create a new column in this dataset. I would like to have rating to be 1, 2, 3, 4 or 5 instead of 1.25, 2.00, 3.75, ..., 5.00.
Can you help me? The only solution I found is this

chocolate$Rat = gsub("1.25", "1", chocolate$Rating)

but I know it is not a good way to work. Is there something faster which can to the work efficiently?
Thanks


texts: What is the advantage of using Dunn index over other metrics for evaluating clustering algorithm?

texts: There are many metrics to evaluate clustering algorithm like Calinski-Harabaz Index, Dunn index, Rand index, etc. Are there any advantage of using Dunn index over other metrics for evaluating clustering algorithm (K-means in particular)? If yes, what are the advantages and disadvantages?


texts: How to analyse move kindness in python (Logistic regression, neural network, etc.)?

texts:
With a team of researchers we were given the assignment to make a scale for the move kindness (how inviting a room or place is for exercise--e.g., a gym). In order to get objective results we were asked to make a measurement device. This device receives input through various sensors and maybe some true/false questions, and then analyze it with training information in mind. After the analysis it returns a number from, say: 1 to 10, indicating the move kindness. So a gym gets, for example an eight and a classroom a three. The problem with this is, that move kindness is very subjective, so we have conducted some surveys. For example one of the criteria is the temperature of the room/place. While the survey was being conducted we measured the temperature:

  From a scale from 1 to 10, what is your opinion about the temperature?

And then we measured the temperature. We have put all these information into some  spreadsheets:

Rating (Move Kindness): 8
Temperature: 18 degrees Celsius

At the end of this survey we asked them to give the move kindness a rating.
So we have this, for example:
Temperature: 8, 18
Light: 7, 300
Humidity: 8, 50
....
Rating (Move Kindness): 8

So my question is, what's the best way to analyse these data for a reliable measurement device using python?
We were thinking of using neural networks, because they can be trained, but logistic regression or some other machine learning algorithm is also an option. Can anyone give me some direction on this?


texts:
I am new to Machine Learning.  Need a little direction on how to proceed with training a model for object detection.  I have a complete training dataset and test dataset of cars with color images, annotations and bounding boxes.  This data set has 196 classes.

All google search mention use of pre-trained models.  However I want to train the model on this dataset to detect and predict bounding box values.  Did not find any on StackExchange either.

Appreciate any pointers.

Thanks


texts:
So I have a scraper that gets articles. However, it doesn't always work properly. I want to get better at checking when it doesn't work. For example, the following is something like I want it to scrape:

  Hello. This is a sequence of sentences that are put together. They don't have to follow this exact format, but something very close to this would be nice! Just basically stuff like this put together with the occasional weird formatting, which depends on what is scraped.

But I might also get something that is obviously not text:

  REGISTER | LOGIN | LOGOUT | Sign in to your account Forgot your password?  {* #signInForm *}....

Is there any python library that checks the general format of strings? Basically, I am scraping articles and want to see if the text scraped is article-y. If there isn't a python library, would the best way to go be some sort of regex matching? Is this possible to do reasonably well?

Any help would be greatly appreciated, thanks!!


texts: Neural net without hidden layers should be a simple linear model: why do I get so different results?

texts:
I’m reading „Elements of Statistical Learning“ where Hastie et al. describe in Section 11.3 on neural nets (p. 394), that (in short) if there are no hidden layers in a neural net (so without non-linear transformation of features), then the model collapses to a linear model.

I tested this in a Keras model with only the output layer included in the model. I noticed that the results (mean absolute error, mae) differ a lot contingent on the choice of the learning rate and the number of training epochs. For some combinations of training epochs and learning rate, I get a mae which is very close to the one of an ordinary least square (OLS) model.

However, in many cases, results are hugely different (and of course worse than under OLS).
I don’t understand why this happens. Can someone point me to an answer?

Here is my playcode (using Boston housing data):

library(keras)
library(tibble)
# Data
boston_housing <- dataset_boston_housing()

c(train_data, train_labels) %<-% boston_housing$train
c(test_data, test_labels) %<-% boston_housing$test
paste0("Training entries: ", length(train_data), ", labels: ", length(train_labels))
train_data[1, ] # Display sample features, notice the different scales

column_names <- c('CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE',
                  'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT')
train_df <- as_tibble(train_data)
colnames(train_df) <- column_names
train_df
train_labels[1:10] # Display first 10 entries

############################################
# Feature engineering

# Test data is *not* used when calculating the mean and std.
# Normalize training data
train_data <- scale(train_data)

# Use means and standard deviations from training set to normalize test set
col_means_train <- attr(train_data, "scaled:center")
col_stddevs_train <- attr(train_data, "scaled:scale")
test_data <- scale(test_data, center = col_means_train, scale = col_stddevs_train)

train_data[1, ] # First training sample, normalized

######################################
# OLS
olstrain = data.frame(cbind(train_labels, train_data))
olstest  = data.frame(cbind(test_labels, test_data))
colnames(olstrain) = c('y', 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE',
                       'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT')
colnames(olstest) = c('y', 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE',
                      'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT')
ols = lm(y~.,data=olstrain)
summary(ols)
olspred = predict(ols, newdata=olstest)

library(Metrics)
mae(test_labels, olspred)

##################################
# Keras

#lrate=0.02
lrate=0.1

# OLS-like results are sensitive to choice of LR and EPOCHS (both must match)

epochs = 100
build_model <- function() {
  model <- keras_model_sequential() %>%

    # OLSisch
    layer_dense(units = 1, input_shape = dim(train_data)[2])

    # Simple NN
    #layer_dense(units = 64, activation = "relu", input_shape = dim(train_data)[2]) %>%
    #layer_dense(units = 1)

    model %>% compile(
      loss = "mse",
      optimizer = optimizer_rmsprop(lr=lrate),
      metrics = list("mean_absolute_error")
    )

  model
}

model <- build_model()
#model %>% summary()

# Fit the model and store training stats
history <- model %>% fit(
  train_data,
  train_labels,
  epochs = epochs,
  validation_split = 0.2,
  verbose = 1
)

c(loss, kmae) %<-% (model %>% evaluate(test_data, test_labels, verbose = 0))

print("======================================================")
print(paste0("Learning rate: ", lrate))
print(paste0("Keras: Mean absolute error on test set: $", sprintf("%.2f", kmae*1000)))
print(paste0("Keras mean val_mae:                     $", sprintf("%.2f", mean(history$metrics$val_mean_absolute_error[30:100])*1000)))
print(paste0("O L S: Mean absolute error on test set: $", sprintf("%.2f", mae(test_labels, olspred)*1000)))
print(paste0("Diff.                                 : $", sprintf("%.2f", kmae*1000 - mae(test_labels, olspred)*1000)))


texts: Why there are no tutorials on calculating Gini impurity in a three-valued categorical feature variables?

texts:
Trying to learn here so please go easy on me if I asked dumb questions.

As the title says, I was searching for a tutorial that calculates the Gini in a CART algorithm for a three-valued feature variable. (i.e., assuming we have a feature variable named, "Blood pressure" and it has three levels (low, medium, and High)). However, I didn't find one. A perfect example of that is the Play Tennis problem, we can have a solution using the entropy method, but not with the Gini method.


texts: Proper order of optimizing hyperparameters in ANN

texts: I have built a neural network for approximating a certain function and decided on a metric how to evaluate the performance. Now, where do I start with optimizing to get the optimal result? I need to decide on number of layers, hidden neurons, learning rate, learning algorithms and all other hyperparameters. What is the usual  procedure and order in which to optimize all network's parameters? I guess I am looking for some kind of general guidelines to follow to obtain the best possible performance


texts:
I have an image (for example (7x7x3) and a filter (3x3x3)). I convolved the image with the filter and it became a (3x3) output. If I want to do the inverse operation and want it to become the image from the output and the filter. How can I implement this operation in Python with Numpy?

I don't know which operation I should use with the filter (inverse or transpose)?

Here is my code for the Deconvolution:

import numpy as np

def deConv(Z, cashe):

'''

deConv calculate the transpoe Convoultion between the output of the ConvNet and the filter

Arguments:
    Z-- Output of the ConvNet Layer, an array of the shape()
'''
# Retrieve information from "cache"
(X_prev, W, b, s, p) = cashe

# Retrieve dimensions from X_prev's shape
(m, n_H_prev, n_W_prev, n_C_prev) = X_prev.shape

# Retrieve dimensions from W's shape
(f, f, n_C_prev, n_C) = W.shape

# Retrieve dimensions from Z's shape
(m, n_H, n_W, n_C) = Z.shape

#create initial array for the output of the Deconvolution
X_curr = np.zeros((m, n_H_prev, n_W_prev, n_C_prev))

#loop over the Training examples
for i in range (m):

    #loop over the vertical of the output
    for h in range(n_H):

        #loop over the horizontal of the output
        for w in range(n_W):

            #loop over the
            for c in range (n_C):

                #loop over the color channels
                for x in range(n_C_prev):

                    #inverse_W = np.linalg.pinv(W[:, :, x, c])
                    transpose_W = np.transpose(W[:,:,x,c])
                    #X_curr[i, h*s:h*s+f, w*s:w*s+f, x] +=  Z[i, h, w, c] * inverse_W
                    X_curr[i, h*s:h*s+f, w*s:w*s+f, x] +=  Z[i, h, w, c] * transpose_W
                X_curr[i, h*s:h*s+f, w*s:w*s+f, :] += b[:,:,:,c]

X_curr = relu(X_curr)

return X_curr


texts: YOLOv1 algorithm - how to determine predictor responsibility

texts:
I am researching Yolo detector, and have read the original paper, but still have some confusion and a few questions regarding the assignment of bounding box predictors to ground truth at training time (in particular calculating the confidence).
Some resources like Andrew Ngs deep.ai here set the confidence of the corresponding label vector bounding box confidence to "1" if there is an object present at that grid cell. In contrast, here is an article from Oracles engineers that set the confidence to maximum Intersection-over-union value between some object and all bounding box predictors. I think the latter is also in line with what the paper is trying to tell us:

"YOLO predicts multiple bounding boxes per grid cell.
At training time we only want one bounding box predictor
to be responsible for each object. We assign one predictor
to be “responsible” for predicting an object based on which
prediction has the highest current IOU with the ground
truth" (You Only Look Once:
Unified, Real-Time Object Detection; Redmon, Divvala, Girshick, Farhadi)

Which of these methods should be used ?
I made a sketch to try to understand the mapping between ground truth and predictor bounding boxes. Please correct me if I'm wrong. The paper states we should assing a b.b. predictor to a ground truth object, but once we have done so, that bounding box predictor is "taken" and married to that object, so it can't be assigned to any other object in that grid cell?
In my example with 3 ground truth bounding boxes and 2 predictors (B=2), we choose a predictor for object b1, which is b2_hat with highest IoU of 0.7, but b2 also has the highest Iou with b2_hat of 0.6, but b2_hat is already taken, so we take the next best thing, which is b1_hat. Now, all the bounding box predictors are assigned, and b3 gets no predictors, even if it has the highest IoU with all the boxes.


texts: What is the use of function build in custom layers in tensorflow keras?

texts:
I am trying to build my own custom keras layer following the documentation at
https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer
In the custom layers we need three functions call, build and init I am not able to understand where the function build gets called and what is its use.
I checked the source of Dense layer (https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/keras/layers/core.py#L1159-L1190) in keras which is also built on top of the Layer class of tensorflow.keras and the same thing is observed there as well. I cant find any call to the function build.
Can someone please explain me what build function is for ? I mean I understand that add_weights is called inside build and it is important but where is build being called


texts: ML Algorithm or approach to solve timeseries selection

texts:
I am fairly new to ML and I'm working on a problem, but not sure which algorithm to choose.  The dataset contains a set of incremental time-series events, in consistent and set intervals, with a list of potential observations.  Something akin to what the output of an audio recognition program might produce where it looks something like:

listener_id
interval_timestamp
candidate_song

12345
00:00:00
Hey Jude (The Beatles)

12345
00:00:05
Hey Jude (The Beatles)

12345
00:00:10
Hey Jude (The Beatles)

12345
00:00:15
Hey Jude (The Beatles)

12345
00:00:20
Hey Jude (The Beatles)

12345
00:00:20
Yesterday (The Beatles)

12345
00:00:25
Hey Jude (The Beatles)

12345
00:00:30
Hey Jude (The Beatles)

...

12345
00:01:30
Yellow Submarine (The Beatles)

12345
00:01:30
Hey Jude (The Beatles)

12345
00:01:30
Circle Sky (The Monkees)

12345
00:01:30
I'm a Believer (The Monkees)

...

12345
00:03:20
Hey Jude (The Beatles)

12345
00:03:25
Imagine (John Lennon)

12345
00:03:30
Imagine (John Lennon)

12345
00:03:35
Imagine (John Lennon)

12345
00:03:40
Imagine (John Lennon)

I want to be able to have the program pick the best possible option based on previous values, so the output would like something like:

listener_id
interval_timestamp
candidate_song

12345
00:00:00
Hey Jude (The Beatles)

12345
00:00:05
Hey Jude (The Beatles)

12345
00:00:10
Hey Jude (The Beatles)

12345
00:00:15
Hey Jude (The Beatles)

12345
00:00:20
Hey Jude (The Beatles)

12345
00:00:25
Hey Jude (The Beatles)

12345
00:00:30
Hey Jude (The Beatles)

...

12345
00:01:30
Hey Jude (The Beatles)

...

12345
00:03:20
Hey Jude (The Beatles)

12345
00:03:25
Imagine (John Lennon)

12345
00:03:30
Imagine (John Lennon)

12345
00:03:35
Imagine (John Lennon)

12345
00:03:40
Imagine (John Lennon)

1 observation per time interval, with the best guess based on the previous values.
I did some research on RNN and LSTM, but those seem to be mostly used for predictions and forecasts on what the next interval probably is, not choosing from a set of captured observations.
Would anyone be able to point me in the right direction in terms of algorithms/approches to best solve this problem?
Any help is appreciated.


texts: Calculating derivative of error at point x with respects to weight w_j

texts: I don't know how the equation below goes from line 2 to 3 after the derivative term is moved inside the brackets.  Specifically, how is it calculating the derivative of log(y_hat)?  Also, if anyone can point to a good textbook or website to learn this stuff.  I've just started on this free course (option to pay for extras) at edx that is so far pretty good because it has easy-to-understand lectures and jupyter notebook assignments that get you coding:  https://courses.edx.org/courses/course-v1:Microsoft+DAT256x+1T2019/course/


texts: Why am I getting very different results between SVC, LinearSVC and Naive Bayes?

texts:
I am doing classification by using bag-of-words model. The goal is to locate users based on their tweets. Splitted the data as 80% training and 20% test.

I did experiments with sklearn's SVC and Naives Bayes. The results 35% and 42% accuracy respectively. However, when I try the sklearn's LinearSVC algorithm, it gives me 80% which is shocking.

This is the part of the code:

text_clf = Pipeline([
    ('vect', CountVectorizer(stop_words='english')),
    ('tfidf', TfidfTransformer()),
    ('clf', LinearSVC()),
])
text_clf.fit(train_data, train_target)

What might be the reason for that? Why LinearSVC performs really good?


texts: Random Forest, Type - Regression, Calculation of Importance Example

texts:
I am trying to use a Random Forest Model (Regression Type) as a substitute of logistic regression model. I am using R - randomForest Package. I want to understand the meaning of Importance of Variables (%IncMSE and IncNodePurity) by example.

Suppose I have a population of 100 employees out of which 30 left the company.
Suppose in a particular decision tree, population is split by an attribute (say location) into two nodes. One node contains 50 employees out of which 10 left the company and other contains 50 employees from which 20 left the company. Can someone demonstrate me a calculation of %IncMSE and IncNodePurity. (if Required for averages etc., please consider another decision tree)

This may look like a repeated question but I could not find a worked out example.


texts: How much can bias decrease performance of the network at the beginnng of the training?

texts:
I am writing a custom framework and in it I'm trying to train a simple network to predict the addition function.

The network:

1 hidden layer of 3 Neurons
1 output layer
cost function used is Squared error, (Not MSE to avoid precision problems)
Identity transfer function to make things simple at first
no specal updaters, just the step size
no learning rate decay
no regularization

The training set:

~500 samples
inputs: [n1][n2]; labels: [n1 + n2]
Every element is between 0 and 1. e.g.:
  [0.5][0.3] => [0.8]

The algorithm I'm using to optimize:

samples 64 elements for an epoch
for each sample: it evaluates the error

then propagates the error back
and then based on the error values calculates the gradients
the gradients for each elements are added up into one vector, then normalized by dividing by the number of samples evaluated

After the gradients are calculated a step size of 1e-2 is used to modify the weights.
The training stops when the sum of the errors for the 500 data elements are below 1e-2

I don't have a test dataset yet, as first I'd like to overfit to a training set, to see if it could even do that. Withouot bias the training converges to an optimum in about ~4k epochs.

When I include the tuning of bias into the training, it seems to have a much worse performance,
the network is not converging to the optimum, instead the biases and the weights oscillate next to one another..

Is this a normal effect of introducing a bias?

Here is a chart abuot the weight values throughout the training:


texts: Advanced Activation Layers in Deep Neural Networks

texts: I'm wondering about the benefits of advanced activation layers such as LeakyReLU, Parametric ReLU, and Exponential Linear Unit (ELU). What are the differences between them and how do they benefit training?


texts: Can I forecast with discontinued data using ARIMA?

texts:
I have data for sales on monthly basis, but a few months' information is not in the CSV file or data file. Can I forecast or fill that missing month with other calculated values from present records?

Part of the code I am using:

AIC = []
SARIMAX_model = []
for param in pdq:
    for param_seasonal in seasonal_pdq:
        try:
            mod = sm.tsa.statespace.SARIMAX(train_data,
                                            order=param,
                                            seasonal_order=param_seasonal,
                                            enforce_stationarity=False,
                                            enforce_invertibility=False)

            results = mod.fit()

            print('SARIMAX{}x{} - AIC:{}'.format(param, param_seasonal, results.aic), end='\r')
            AIC.append(results.aic)
            SARIMAX_model.append([param, param_seasonal])
        except:
            continue
print('The smallest AIC is {} for model SARIMAX{}x{}'.format(min(AIC), SARIMAX_model[AIC.index(min(AIC))][0],SARIMAX_model[AIC.index(min(AIC))][1]))

# Let's fit this model
mod = sm.tsa.statespace.SARIMAX(train_data,
                                order=SARIMAX_model[AIC.index(min(AIC))][0],
                                seasonal_order=SARIMAX_model[AIC.index(min(AIC))][1],
                                enforce_stationarity=False,
                                enforce_invertibility=False)


texts: Is it meaningful to use word2vec for non-string inputs like time series analysis?

texts: I am working on a project that detects anomalies in a time series. I wonder if I can use word2vec for anomaly detection for non-string inputs like exchange rates?


texts: Initializing neural network with singular vectors

texts:
I'm thinking about ways to initialize my neural networks for faster convergence, and I was wondering about initializing the weights with the singular vectors of the data in order to immediately start with "useful" features. (I couldn't find any such paper) Obviously that's pretty vague, so specifically:
Example: for a CNN with $k$ $n$ by $n$ kernels and p images of $m$ by $m$, assemble the $p(m-n)(m-n) \ by \ n^2$ matrix A consisting of every window of the kernel (or a random subset thereof) and initialize the convolutional filters to the first k left singular vectors of A
My question is: would this create a stable convergence for training, or is it likely to lead to exploding or vanishing gradients? Further, would it be useful to identify features more quickly, or just lead to overfitting?


texts:
I have data consisting of light wavelength and some coefficient. The wavelength in the data I have is in steps of 10s and I need the coefficient for a wavelength that is between the values I have (eg. 432.5 nm). Data:
wavelen    coeff
380         0.982
390         0.564
400         0.378
...         ...

I was thinking of regressing the data I have to find a function that represent the data, but not quite sure what type of function I should choose. I tried exponential and power functions, but the results not accurate for the first few wavelengths due to the coefficient drops extremely fast for earlier wavelengths.

What would be the best type of regression for this data? I was also thinking of splitting the regression for the first few data points, like a piecewise function. Is this a valid method?


texts: Does the count of items in a transaction matter to apriori?

texts: When preparing my data for funneling into the Microsoft Association Rules algorithm, I was not sure if I should group by data by Transaction and Item, or have a record for every instance of an item in a transaction. Does the algorithm care and add weight if an item appears 3 times in a transaction  Or is it just looking for the existence of an item with another item, regardless of how many are present?


texts:
In pymc3 documentation, it specifies that the .dist() member for distribution allows the distribution to be used without a model for sampling and use of the logp functions e.g.

d_dist = pm.HalfCauchy.dist(beta=2.5, shape=3)

however, an example I copied, used .dist() functions for model elements, but when I remove the .dist() and use the model version of the distribution, it fails.

so:

data = [tuple(x) for x in self.pointdata[['x','y', 'z']].values]
print("Data {}".format(data))
with pm.Model() as model:
   mu1 = pm.Normal('mu_wall1' ,mu=0.0, sd=10.0, shape=3)
   sd_dist = pm.HalfCauchy.dist(beta=2.5, shape=3)
   chol_packed = pm.LKJCholeskyCov('chol_packed', n=3, eta=2, sd_dist=sd_dist )
   chol= pm.expand_packed_triangular(3, chol_packed)
   wall1 = pm.MvNormal('wall1', mu=mu1, chol=chol, observed=data)
   trace = pm.sample(5000, tune=1000)

will work, generating samples and providing a solution. but:

data = [tuple(x) for x in self.pointdata[['x','y', 'z']].values]
print("Data {}".format(data))
with pm.Model() as model:
   mu1 = pm.Normal('mu_wall1' ,mu=0.0, sd=10.0, shape=3)
   sd_dist = pm.HalfCauchy('sd1', beta=2.5, shape=3)
   chol_packed = pm.LKJCholeskyCov('chol_packed', n=3, eta=2, sd_dist=sd_dist )
   chol= pm.expand_packed_triangular(3, chol_packed)
   wall1 = pm.MvNormal('wall1', mu=mu1, chol=chol, observed=data)
   trace = pm.sample(5000, tune=1000)

will throw

  line 29, in _alignPointsToMajorAxes
    chol_packed = pm.LKJCholeskyCov('chol_packed', n=3, eta=2, sd_dist=sd_dist )
  File "build/bdist.linux-x86_64/egg/pymc3/distributions/distribution.py", line 42, in __new__
  File "build/bdist.linux-x86_64/egg/pymc3/model.py", line 816, in Var
  File "build/bdist.linux-x86_64/egg/pymc3/model.py", line 1492, in __init__
  File "build/bdist.linux-x86_64/egg/pymc3/distributions/transforms.py", line 95, in apply
  File "build/bdist.linux-x86_64/egg/pymc3/distributions/distribution.py", line 52, in dist
  File "build/bdist.linux-x86_64/egg/pymc3/distributions/transforms.py", line 125, in __init__
  File "build/bdist.linux-x86_64/egg/pymc3/model.py", line 1209, in __init__
  File "build/bdist.linux-x86_64/egg/pymc3/distributions/multivariate.py", line 1041, in logp
AttributeError: 'TransformedRV' object has no attribute 'logp'

I would like to understand why using dist() helps and why utilizing the model version of the distribution causes errors.


texts: training accuracy greater than validation accuracy

texts:
The problem that I'm facing is that the training accuracy of my model is way higher than the validation accuracy, were talking about an approximate value of 0.2. And I can't understand why, yet I'm still a newbie when it comes to this so bear with me, please.
The data comes from two datasets which were created using f.data.Dataset, one for training another for validation since that's how the dataset had his folder layout.
model = keras.Sequential([
    keras.layers.Flatten(),
    keras.layers.Dense(256, activation='relu'),
    keras.layers.Dropout(.1),
    keras.layers.Dense(2, activation="softmax")
])

model.compile(optimizer='adam',
            loss="categorical_crossentropy",
            metrics=['accuracy'])

model.fit(train_ds, steps_per_epoch=STEPS_PER_EPOCH, epochs=10, validation_data=test_ds, validation_steps=VALIDATION_STEPS)

Train for 163.0 steps, validate for 20.0 steps
Epoch 1/10
163/163 [==============================] - 3s 21ms/step - loss: 3.9965 - accuracy: 0.8468 - val_loss: 0.3582 - val_accuracy: 0.8406
Epoch 2/10
163/163 [==============================] - 3s 19ms/step - loss: 0.3197 - accuracy: 0.8930 - val_loss: 0.5207 - val_accuracy: 0.7641
Epoch 3/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2009 - accuracy: 0.9191 - val_loss: 0.4350 - val_accuracy: 0.8062
Epoch 4/10
163/163 [==============================] - 3s 19ms/step - loss: 0.1815 - accuracy: 0.9270 - val_loss: 0.5521 - val_accuracy: 0.7516
Epoch 5/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2122 - accuracy: 0.8986 - val_loss: 0.9616 - val_accuracy: 0.7156
Epoch 6/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2405 - accuracy: 0.9082 - val_loss: 1.2039 - val_accuracy: 0.7109
Epoch 7/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2013 - accuracy: 0.9183 - val_loss: 0.7242 - val_accuracy: 0.6406
Epoch 8/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2055 - accuracy: 0.9176 - val_loss: 0.4830 - val_accuracy: 0.6891
Epoch 9/10
163/163 [==============================] - 3s 19ms/step - loss: 0.1901 - accuracy: 0.9250 - val_loss: 0.3925 - val_accuracy: 0.8313
Epoch 10/10
163/163 [==============================] - 3s 19ms/step - loss: 0.1861 - accuracy: 0.9202 - val_loss: 0.5492 - val_accuracy: 0.8000

Could anyone explain to me please what could cause this big gap in between accuracy and val_accuracy, please?


texts: Create a random chi-Square independence distribution with a given p-Value

texts:
I want to randomly create a table of data that has a predefined p-Value and chi-Value of a chi-square distribution.
For example this would have a p-Value of 1 on a chi-square independence test:
[[25,25],
 [25,25]]

Trying arround some random values I see that:
[[50,0],
 [30,20]]

has a p-Value of 2.02E-6 and a chi-Value of 22,56.
But how would I do it the other way arround? I have given p-Value of 0.05 for example from that I want to get a table with random values that has exactly that p-Value.
I already tried the following in python: np.random.default_rng().chisquare(dof,size_arr)*homogenous_expected_value specifically:
np.random.default_rng().chisquare(1,[2,2])*25
which is a good start and gives me something like this:
[[ 4.83409623 42.00549597]
 [ 5.17981502 20.08927103]]

or
[[ 11.59607448   1.58326752]
 [ 25.40019747 111.6858744 ]]

But as these are random values without and limits whatsoever the p-Values differ widely. For now I have no better option than to repeat this process until I get a table with a random distribution that is in a from me predefined desired range. But there must be a better way, right?


texts: Approaches to high dimension pattern matching problem

texts:
My apologies in advance as I am new to this. I have searched the internet and tried various processes and nothing seems to work or address this situation.

I have a dataset of 30,000 transactions and 500,000 items. Average item size for a transaction is 50. The dataset is sparse, so the support number must be set quite low. Furthermore, the rules become more valuable the larger the number of items in the rule.

I have tried running this in arules and the tests fail after exceeding 64 gb of RAM (the limit of the machine). I have tried reducing items and transactions to smaller subsets, but still hit this memory limit.

Ultimately, I am looking for ways to cluster large groups of similar accounts by selection of items and generate confidence and lift of various next items selected from those clusters.

My question: are there alternative, more efficient ways to do this, or other approaches to consider?

Thank you.


texts: What are the advantages of the rectified linear (relu) over the sigmoid activation function in networks with many layers?

texts: The state of the art of nonlinearity is to use rectified linear units (ReLU) instead of a sigmoid function in deep neural networks. What are the advantages?


texts: Coefficient of determination is close to 1 but the value of RMSE is large. What does it mean?

texts:
I am working with the DecisionTreeRegressor and trying to understand how well the data fits the model. I calculated both RMSE and coefficient of determination. At a certain depth,  coefficient of determination has the value equal to 0.9918744073066561 but the value of RMSE is equal to 75.0025. I cannot understand this. The value of RMSE is quite large but the value of the coefficient of determination is close to 1.0. What does it really mean? Is the model/fit good enough?

from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import AdaBoostRegressor
from sklearn.metrics import mean_squared_error

def rmse(y_true, y_pred):
    return math.sqrt(mean_squared_error(y_true, y_pred))

sample_depth = np.linspace(1,40, num = 40, dtype=int)
dt_score_list = []
for index, depth in enumerate(sample_depth):
    boosted_regressor = AdaBoostRegressor(DecisionTreeRegressor(max_depth=depth), random_state=1)
    boosted_regressor.fit(X_train, y_train)
    dt_score_list.append(boosted_regressor.score(X_test, y_test))
    print(boosted_regressor.score(X_test, y_test), rmse(y_test, boosted_regressor.predict(X_test)), depth)


texts:
I'm beginning my data science journey and I've faced a challenge that confuses me a bit.
I have a set with few features and a target variable whose raw distribution is highly skewed.

I've read that it's possible to use a log transformation to normalize the target variable (loss in $) and thus increase the accuracy.

When I train my model with "y_raw", using MAE I get an error of 306k. When I log-transform
y = y.transform(np.log) I get MAE accuracy of around 2 (log-transformed units I suppose?), which is e^2 = 7.39 (y_raw). This is a significant drop from 306k to only 7.39 ($) (or am I getting it wrong?), so I am a bit suspicious about it.

So here are my questions:
1) Did I get it correct that the error rate drop from 306k to only 7.39 is real and is valid?
2) How do I make a predictions from there? If I feed a sample to my model, receive a log-transformed output, lets say it returned a prediction of y_log = 10. Do I then simply use an inverse of it by placing e^10 = 22,026.5 and will it be my final prediction?


texts: What are some useful but less known libraries when working with machine learning?

texts:
What are some useful but less-known python libraries when working with machine learning?

For example, I saw these two libraries to help plot the ML plots.

Yellowbrick: Machine Learning Visualization
scikit-plot

What are some other useful packages you know that can make my life easy? It can be related to visualization, preprocessing or something else.

Note: I already know about popular packages like (numpy, pandas, sklearn, matplotlib, seaborn etc)


texts: How to perform convolution with kernel bigger than image?

texts:
In this question I've seen an example of convolution by the kernel with the shape bigger than initial image's one:
import numpy as np
from scipy import signal

x = np.array([(0.51, 0.9, 0.88, 0.84, 0.05),
              (0.4, 0.62, 0.22, 0.59, 0.1),
              (0.11, 0.2, 0.74, 0.33, 0.14),
              (0.47, 0.01, 0.85, 0.7, 0.09),
              (0.76, 0.19, 0.72, 0.17, 0.57)])

y = np.array([(0, 0, 0.0686, 0),
              (0, 0.0364, 0, 0),
              (0, 0.0467, 0, 0),
              (0, 0, 0, -0.0681)])

gradient = signal.convolve2d(np.rot90(np.rot90(y)), x, 'valid')

So, we get this:

array([[ 0.044606, 0.094061], [ 0.011262, 0.068288]])

I understand that y is flipped by 180 degress. But how does "valid" convolution work here? When we can get (2x2) shape from (4x4) convolved by (5x5)?


texts: How to plot two distribution curves in a faceted way in R/ggplot2?

texts:
I have two probability distribution curves, a Gamma and a standarized Normal, that I need to compare:

library(ggplot2)

pgammaX <- function(x) pgamma(x, shape = 64.57849, scale = 0.08854802)

f <- ggplot(data.frame(x=c(-4, 9)), aes(x)) + stat_function(fun=pgammaX)
f + stat_function(fun = pnorm)

The output is like this

However I need to have the two curves separated by means of the faceting mechanism provided by ggplot2, sharing the Y axis, in a way like shown below:

I know how to do the faceting if the depicted graphics come from data (i.e., from a data.frame), but I don't understand how to do it in a case like this, when the graphics are generated on line by functions.
Do you have any idea on this?


texts:
I'm finding a quicker method to export plots

import numpy as np
from matplotlib.backends.backend_pdf import PdfPages
import matplotlib.pyplot as plt

X = ... #define X
with PdfPages('MyFile.pdf') as export_pdf:
   plt.figure()
   plt.plot(np.arange(0,60+1), X)
   plt.xlabel('t')
   plt.ylabel('x(t)')
   plt.title('title')
   export_pdf.savefig()
   plt.show()
   plt.close()

I'm trying to find simpler code than the one I wrote attached. Any advice?


texts: Detecting abundance of a certain periodic pattern in a time series?

texts:
I am really stumped at the moment about how to solve a particular problem. I have many time series like this:

This represents the number of hours a person spends on a website each day throughout the year. Any days where they are not seen to be using the website have zero values, rather than missing values.

What I really want to do is to calculate a metric telling me to what extent there is a consistent "1 hour per day, once per week" pattern, or, say a "10 minutes a day 6 times per week" pattern and I have written some code that produces this pattern. I don't have any background really in machine learning or deep learning, but I have looked into some techniques from spectral analysis and time series. I also found a technique called "Dynamic Time Warping" which I could in theory use to find out how much of a cost there is in mapping this time series onto the "ideal" one (representing the usage patterns I just mentioned).

I tried plotting the periodogram and correlogram, but there seems to be very little correlation indicating periodicity of any kind:

Can anyone shed some light onto how I could use certain techniques to get the information I need, or any introductory texts on periodicity detection or data mining for those who have never really used it before?


texts: In Neural network, if one node is deleted, where should other nodes be connected?

texts: If it's a fully connected neural network, should we just remove those lines that were originally connected to the deleted node, and hence no actual changes on the remaining nodes? Except there weights and bias will be updated?


texts: Keras next(); what does (2, 256, 128, 128, 3) mean

texts:
I have used the next() method on a Keras generator. I then turned this into a numpy array to get the shape:
data = generator.next()
data = np.array(data)
print(data.shape)

>>> (2, 256, 128, 128, 3)

256 is my batch size, and (128, 128, 3) is my image size. But what does 2 represent?


texts:
Let's say I'm trying to classify some data with logistic regression.

Before passing the summed data to the logistic function (normalized in range $[0,1]$), weights must be optimized for desirable outcome. In order to find optimal weights for classification purposes, relatively minimizable error function must be found, this can be cross entropy.

From my knowledge, cross entropy measures quantification between two probability distributions by bit difference between set of same events belonging to two probability distributions.

For some reason, cross entropy is equivalent to negative log likelihood. Cross entropy loss function definition between two probability distributions $p$ and $q$ is:

$$H(p, q)=-\sum_{x}p(x)\,log_e(q(x))$$

From my knowledge again, If we are expecting binary outcome from our function, it would be optimal to perform cross entropy loss calculation on Bernoulli random variables.

By definition probability mass function $g$ of Bernoulli distribution, over possible outcome $x$ is:

$$g(x|p)=p^{x}(1-p)^{1-x} \ \textrm{for} \ x\in [0, 1]$$

Which means that probability is $1-p$ if $x=0$ and $p$ if $x=1$.

Bernoulli probability distribution is based on binary outcome and therefore process of cross entropy being performed on Bernoulli random variables is called binary cross entropy:

$$\mathcal{L}(\theta)= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i)\log(1-p_i) \right]$$

Is this true? why are negative logarithm likelihoods associated with cross entropy? why does Bernoulli random variable perform so well?

In short, how does binary cross entropy work?


texts: Solve a pair of coupled nonlinear equations within certain limits

texts:
This answer to this question works only for situations in which the desired solution to the coupled functions is not restricted to a certain range.

But what if, for example, we wanted a solution such that 0 < x < 10 and 0 < y < 10?

There are functions within scipy.optimize that find roots to a function within a given interval (e.g., brentq), but these work only for functions of one variable.

Why does scipy fall short of providing a root solver that works for multi-variable functions within specific ranges?  How might such a solver be implemented?


texts: ValueError while using linear regression

texts:
I have loaded a dataset and converted into data frame while I am using linear regression I am receiving the following value error as shown in my code below. I am at the moment doing a tutorial and could not figure out how come the arrays are 1-D as the error shows.

from sklearn.datasets import load_boston
boston_dataset=load_boston()

#create a pandas dataframe and store the data
df_boston=pd.DataFrame(boston_dataset.data)
df_boston.columns=boston_dataset.feature_names

#append price, target as a new column in the dataset
df_boston['Price']=boston_dataset.target

#print first five observations
df_boston.head()

CRIM    ZN  INDUS   CHAS    NOX RM  AGE DIS RAD TAX PTRATIO B   LSTAT   Price
0   0.00632 18.0    2.31    0.0 0.538   6.575   65.2    4.0900  1.0 296.0   15.3    396.90  4.98    24.0
1   0.02731 0.0 7.07    0.0 0.469   6.421   78.9    4.9671  2.0 242.0   17.8    396.90  9.14    21.6
2   0.02729 0.0 7.07    0.0 0.469   7.185   61.1    4.9671  2.0 242.0   17.8    392.83  4.03    34.7
3   0.03237 0.0 2.18    0.0 0.458   6.998   45.8    6.0622  3.0 222.0   18.7    394.63  2.94    33.4
4   0.06905 0.0 2.18    0.0 0.458   7.147   54.2    6.0622  3.0 222.0   18.7    396.90  5.33    36.2

#assign features on x-axis
X_features=boston_dataset.data

#assign target on y-axis
Y_target=boston_dataset.target

#import linear model-the estimator
from sklearn.linear_model import LinearRegression
lineReg=LinearRegression()

#fit data into the estimator
lineReg.fit(X_features,Y_target)

ValueError                                Traceback (most recent call last)
<ipython-input-16-4b5b068e587b> in <module>()
      1 #fit data into the estimator
----> 2 lineReg.fit(X_features,Y_target)

/usr/local/lib/python3.4/dist-packages/sklearn/linear_model/base.py in fit(self, X, y, sample_weight)
    480         n_jobs_ = self.n_jobs
    481         X, y = check_X_y(X, y, accept_sparse=['csr', 'csc', 'coo'],
--> 482                          y_numeric=True, multi_output=True)
    483
    484         if sample_weight is not None and np.atleast_1d(sample_weight).ndim > 1:


texts: Comparing 2 columns from separate dataframes and copy some row values from one df to another if column value matches in pandas

texts:
Im relatively new to pandas andpython . I have a simple problem but I can not find the solution . I have 2 dataframes DF1 and DF2 as shown in image
Both dataframes have ID coluumns. I wannt to check if the ID for a row in DF1 matches any of ID in DF2, then I want to copy the diameter value from DF2 and paste it to DF1 in a new column. Eventually what I want to achieve at the end is DF3 shown in image below.
Help will be highly appreciated.


texts: Multiclassification Error: NotFittedError: This MultiLabelBinarizer instance is not fitted yet

texts:
After picking the model, when I try to use it, I am getting error -

  "NotFittedError: This MultiLabelBinarizer instance is not fitted yet.
  Call 'fit' with appropriate arguments before using this estimator."

X = <training_data>
y = <training_labels>

# Perform multi-label classification on class labels.
mlb = MultiLabelBinarizer()
multilabel_y = mlb.fit_transform(y)

p = Pipeline([
('vect', CountVectorizer(min_df=min_df, ngram_range=ngram_range)),
('tfidf', TfidfTransformer()),
('clf', OneVsRestClassifier(clf))
])

# Use multilabel classes to fit the pipeline.
p.fit(X, multilabel_y)


texts:
I am looking for an algorithm that would be able to extract meaningful keyphrases from web articles. Each article has more than 2000 words and information is structured using paragraphs, h1, h2 tags and so on. I would like to find 2-3 word phrases that describe particular content. H1 tag plays an important role since it is like a title, but sometimes the body may contain some important information. I was thinking, that maybe TextRank algorithm could be adapted to this problem by including HTML tags and also Named Entity Recognition (NER). Do you know of any research paper in thus direction or open source implementation?
BTW. RAKE performs poorly.


texts: What are your thoughts on SKLearn's dismissal of GPUs for machine learning?

texts:
SKLearn has this broad claim in its FAQs:

  Outside of neural networks, GPUs don’t play a large role in machine learning today, and much
  larger gains in speed can often be achieved by a careful choice of
  algorithms.

Anyone care to add color for or against this claim?


texts: Which different visualizations to make for time series analysis?

texts:
I hope you're doing great. I have recently started working on a time series project and I have just finished preparing my data and calculating trends (daily, weekly, yearly) and I wonder what are the different plots I should make using the the absolute values of my variables and the calculated trends?

Also, I'm seeking anyone to tell me what are the best practices to work on time series and what are the details I should pay attention to!

Thanks !


texts:
I run Boruta with RandomForestClassifier the previous day on my data (nb features = 36) and got 17/36 confirmed. Now I run it again and there is 0/36 and stop at the 9th iteration. Any idea why this is happening?

%%time
rfc = RandomForestClassifier(n_estimators=200, n_jobs=-1, class_weight= 'balanced_subsample' , max_depth=10)

boruta_selector = BorutaPy(rfc, n_estimators='auto', verbose=2, random_state=1)

boruta_selector_before.fit(X, y)

the current output look  is the follow

Iteration:  1 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  2 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  3 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  4 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  5 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  6 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  7 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  8 / 100
Confirmed:  0
Tentative:  0
Rejected:   36

BorutaPy finished running.

Iteration:  9 / 100
Confirmed:  0
Tentative:  0
Rejected:   36
CPU times: user 1.95 s, sys: 377 ms, total: 2.33 s
Wall time: 2.79 s


texts: What can weka do that python and sklearn can't?

texts:
I would like to build a variety of classification and regression decision trees. My use case focuses on extraction and communication of decision rules. Previously weka was used in my organisation for decision tree learning. What can weka do that Python or Sklearn can't?
I currently use pandas, numpy, scipy, and sk-learn and other libraries for the majority of my workflow.


texts: I'm interested in data science market. I was expecting that there would be a lot of companies who are making algorithms and models for companies like in kaggle competitions. But i struggle to find any. Practically every data science company is selling some automated software which makes algorithms/models by its own. So why is that so? Did i understand situation correctly? and if there are companies who make algorithms and models which are the leaders and best companies in that field. thanks p.s. sorry for my bad english


texts: How backpropagation works in case of 2 hidden layers?

texts:
Imagine the next structure (for simplicity, there's no bias, and activation formatting with sigmoid or relu, just weights). The input has two neurons, the two hidden layers have 3 neurons each, the output layer has two neurons, so a cost ($\sum C$) with two "subcosts" ($C^1$, $C^2$).
(I'm new at machine learning, and super confused with the different notations, and formatting, indexes, so to clarify, in case of activations, the upper index will show the index of it in the layer, and the lower index will show the layers index which is in, so the third neuron in the second layer will look like $a^2_1$ (0 starting). In weights, the upper index will show which is the index of neuron it's "coming from", the first index of the lower index will show which is the index of neuron it's "going to", and the second index of the lower index will show which is the index of layer it's "going to", so the third layers first neurons weight, which connects it with the second layers first neuron will look like $w^0_{0,2}$.)
Just for the example, I would like to propagate the first line's weights back. Just to illustrate:
a0-w1-a1-w2-a2-w3-a3->y0->C0
ax    ax    ax    ax
      ax    ax

So to know the first weights (of the output layer) slope in the backpropagation process:
$$\frac{\partial\sum C}{\partial w^0_{0,3}} = \frac{\partial a^0_3}{\partial w^0_{0,3}}\frac{\partial\sum C}{\partial a^0_3}$$
As I learned, because $w^0_{0,3}$ doesn't affect $C^1$, just $C^0$, $\frac{\partial\sum C}{\partial a^0_3}$ is just $2(a^0_3-y^0)$. Not sure how can it be generalized though, to be used dynamically, not having to check if a weight has direct connection with all costs, or not.
As I somehow glued the tutorials together, calculating the second layers (backwards) first weight:
$$\frac{\partial\sum C}{\partial w^0_{0,2}} = \frac{\partial a^0_2}{\partial w^0_{0,2}}\frac{\partial\sum C}{\partial a^0_2}$$
Where $w^0_{0,2}$ has connection with $a^0_2$, which has connection with all neurons of the output layer, therefore, to both costs, so:
$$\frac{\partial\sum C}{\partial a^0_2}=\frac{\partial C^0}{\partial a^0_2}+\frac{\partial C^1}{\partial a^0_2}$$
Where:
$$\frac{\partial C^0}{\partial a^0_2}=\frac{\partial a^0_3}{\partial a^0_2}\frac{\partial C^0}{\partial a^0_3}$$
and:
$$\frac{\partial C^1}{\partial a^0_2}=\frac{\partial a^1_3}{\partial a^0_2}\frac{\partial C^0}{\partial a^1_3}$$
Clear so far, what I don't understand, and can't split down from this point is:
$$\frac{\partial\sum C}{\partial w^0_{0,1}} = \frac{\partial a^0_1}{\partial w^0_{0,1}}\frac{\partial\sum C}{\partial a^0_1}$$
Because $a^0_1$ has connections to three neurons, in the second layer backwards, in other words
$$\frac{\partial C^0}{\partial a^0_1}=\frac{\partial a^0_2}{\partial a^0_1}\frac{\partial C^0}{\partial a^0_2}$$
can't be true, as $a^0_1$ affects $C^0$ not just through $a^0_2$, but $a^1_2$, and $a^2_2$ as well. So how this can be solved? Do I have to add them up, or it needs to be solved in a different way than the previous solutions?


texts: Would there be any reason to pretrain BERT on specific texts?

texts:
So the official BERT English model is trained on Wikipedia and BookCurpos (source).
Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain a new BERT model from scratch on movie-related dataset?
Can my model become more accurate since I trained it on movie-related texts rather than general texts? Is there an example of such usage?
To be clear, the question is on the importance of context (not size) of the dataset.


texts: What is the role of $W_{ax}, W_{aa}, W_{ay}$ in forward propagation in RNN? Are they hyperparameters? Why are they needed?

texts:
In RNN introduction in Coursera sequence model course, the following formula for forward propagation in RNN was introduced.  What exactly is the role of $W_{ax}, W_{aa}, W_{ay}$? What do they do?

In the lecture, it was told that:

$W_{ax}$: parameter governing connection from $x$ to hidden layer (not sure
what exactly does that mean: what happens if it is not governed?)
$W_{aa}$: governing activations (Why govern activations? What happens
if it is not governed?)
$W_{ay}$ governs output prediction (What is the point of governing that? What happens if it is not governed?)

In standard neural network these were the formula of forward propogation

$$
z_1 = w_1 X_1+b_1 \\
A_1 = g(Z_1)
$$
Consider there were only 4 layers then last layer
$$
z_4 = w_4X_4 + b_4 \\
A_4 \text{ or } \hat{y} = g(z4)
$$
I'm able to correlate these equation with  $a^{<1>}$ in RNN
but I am unable to correlate the role of $W_{ya}$ present in  $\hat{y}^{<1>}$  highlighted in yellow.
Please explain in very simple terms with an example on what is the job of $W_{ax}, W_{aa}, W_{ay}$ in forward pass in standard RNN.


texts: Site for implementing a research papers without code

texts:
Before a couple of months, I found a web page for implementing papers which are published without code. Someone posts a paper on it and then community competes to implement that paper.

But I don't remember the name of the page and I cannot find it anymore. Any help?


texts: What is a "variable index" in the Gaussian perspective?

texts:
I was going through this article about Gaussian processes, in which the author explains about the "variable index" in the form of a plot while writing about 2D Gaussian.

The explanation and plot are as below:

I understood the y-axis in this plot, but I'm having problems understanding the x-axis (variable index).

Where did the values 1 and 2 come from in that axis and how is the y-value 2 for both of them?


texts: Clarifying my understanding of on-policy RL (online SARSA)

texts:
I want to clarify I have understood how SARSA works in nuances. Consider an original definition taken from ON-LINE Q-LEARNING USING CONNECTIONIST SYSTEMS. G. A. Rummery & M. Niranjan. CUED/F-INFENG/TR 166. September 1994 (which is the first publication where SARSA wss mentioned, according to a Wikipedia article).

The authors proposed an update rule which "... differs from normal Q-learning in the use of the Qt+1 associated with the action selected, rather than the greedy max(Qt+1 | a) used in Q-learning." (A citation from page 6.)

Please note that this is essential to on-policy nature of the algorithm. The term SARSA mentioned in the footnote for this definition.

And a later pseudo-code of SARSA update used in many readings:

begin

  initialize Q[S,A] arbitrarily
  observe current state s
  select action using a policy based on Q

repeat forever:

carry out an action a
observe reward r and state s'
select action a' using a poicy based on Q
Q[s,a] <- Q[s,a] + alpha * (r + gamma * Q[s', a'] - Q[s, a])
s <- s'
a <- a'

end-repeat

end

source: http://www.cse.unsw.edu.au/~cs9417ml/RL1/algorithms.html

I want to understand if I must use exactly the same Q function (and policy) to get A and A'. If I update Q function in each iteration, it follows that the next action in a subsequent iteration will be derived using the latest Q updated, while the previous action was obtained using the former Q.\
On the other hand, I really can make A and A' with exactly the same Q, and only after that update the Q. So I will always consider A and A' derived using the same function.

Which is more orthodox / correct?


texts: How does BERT and GPT-2 encoding deal with token such as ,

texts:
As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode the token as one single piece.

However, when I use pytorch BertTokenizer it seems the encoder also separate token into pieces. Is this correct behaviour?

from pytorch_pretrained_bert import BertTokenizer, cached_path
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)
tokenizer.tokenize('<s> This is a sentence <|endoftext|>')

The results are:

['<',
 's',
 '>',
 'This',
 'is',
 'a',
 'sentence',
 '<',
 '|',
 'end',
 '##oft',
 '##ex',
 '##t',
 '|',
 '>']


texts:
I've been doing a few DataScience competitions now, and i'm noticing something quite odd and frustrating to me. Why is it frustrating? Because , in theory, when you read about datascience it's all about features, and the careful selection, extraction and engineering of those to extract the maximum information out of raw variables, and so far, throwing every variable as it is in the mix seems to work fine with the right encodings. Even removing a variable that has 80% nulls ( which in theory should be an overfitting contributor ) decreases the performance of the regression model slightly.

For a pratical case : I have long/lat for a pickup point and destination point. I did the logical task of calculating the distance ( all kinds of them ) from these points. And dropped the long/lat. Model performs way better when you include both ( coordinates & distance ) in the features list. Any explanations? And a general thought on my dilemma here with the real utility of feature selection/engineering/extraction

EDIT : could it be that the information we can get out of the coordinates is bigger than the distance? Is it just possible to extract features that are more beneficial to my model that plain long/lat?


texts: What is the difference between shuffle in fit_generator and shuffle in flow_from_directory?

texts: I am using Keras to create a deep learning model and I would like to know that what is the difference between shuffle argument in fit_generator() method and shuffle argument in flow_from_directory() method?


texts: How can I find similarities in two graphs?

texts:
Let's say that I have two 1 dimensional arrays, and when I plot the two arrays they look like this:

If you look at the top and bottom graphs, then you can see that the highlighted parts are very similar (in this case they're exactly the same). I need to find a way to find these sections using some sort of algorithm or method.

I've tried searching everywhere in the numpy and scikit docs, I even searched everywhere on stackexchange and couldn't find a solution for this problem. I don't think anyone published a solution for this yet.

Does anyone have any idea how I can find similar sections in two graphs? My dataset is a 1 dimensional data array for each graph, and I need a algorithm that tells me where the similar parts are. Just remember that the similar sections are never 100% the same, sometimes they're a little bit off and sometimes there's anomalies so a small part would be different but everything else will still look the same. Also you can ignore the curvature of the graph, that's irrelevant. Only the X and Y coordinates of the data points are important.

I can't read explanations that have a lot of maths inside of them and I also can't turn explanations that have a lot of maths into code, I'm still learning how to do that at University. But I'm really good at reading pseudo-code and other programming languages so please give me an answer with real code.


texts: approach for predicting machine failure using maintenance history

texts:
I have been struggling with this problem for a while now and I finally decided to post a question here to get some help. The problem i'm trying to solve is about predictive maintenance.

Specifically, a system produces 2 kinds of maintenance messages when it runs, a basic-msg and a fatal-msg, a basic message indicates that there is a problem with the system that needs to be checked (its not serious), a fatal-msg on the other hand signals that the machine must be shut down and the fatal problem be addressed before it is allowed to run again. All maintenance messages have a unique message-id associated with them.

For example:

Basic-msgs:

112234: Check Valve 3

162746: Main switch power fluctuating

Fatal-msgs:

7432955: Valve 3 Broken

3358399: Startup Failed

Some key Details:
A basic-msg can occur without a fatal-msg, but a fatal-msg must occur with a corresponding basic-msg. Every time the machine is run it gets a unique run-id.

Problem: By looking at the basic-msgs occurred in the previous 4 runs of the machine, predict what fatal-msgs could occur in the next run. I have the data for previous 8 years (includes basic-msgs, fatal-msgs, corresponding run-ids.

I am stuck with this for about 3 weeks now, can someone guide me about a machine learning approach I can take to solve this, and what tools should i use.

Thanks!


texts: Image preprocessing: How to resize / align / cut images of various sizes?

texts:
I want to create a new dataset for image recognition.

If I have Object A that I want to recognize in images, and multiple images of various different sizes, how do I preprocess them so that

they are of same size
aligned (occurences of Object A rotated appropriately)
zoomed in / out, depending on how big Object A occurence is
?

And what would be the best way to solve the problem when there are multiple occurences of Object A on a single image?


texts: Can Precision-Recall be improved for imbalanced sample?

texts:
I tried out a few models on a highly imbalanced sample (~2:100) where I can get decent AUC from ROC (test sample). But when I plot precision-recall (test sample), it looks horrible. Kind of like the worse PR curve in box (d).

This article contains the picture from below and describes that ROC is better suited since it is invariant to class distribution.

My question is if there's anything than can be done to improve precision-recall?


texts: Clarification on the Keras Recurrent Unit Cell

texts:
I paste below the Keras documentation on Recurrent layer

model= Sequential()
model.add(LSTM(32, input_shape=(10, 64)))

# now model.output_shape == (None, 32)
# note: `None` is the batch dimension.

What does the first argument of LSTM 32 mean here? Is it creating 32 LSTM blocks (By block I mean consisting of input, forget and output gate)? Could you please explain the meaning of the first argument and how that contributes to the output dimension of the LSTM both when return sequences is True and when return sequences is False


texts: Neural Network for Multiple Output Regression

texts:
I have a dataset containing 34 input columns and 8 output columns.

One way to solve the problem is to take the 34 inputs and build individual regression model for each output column.

I am wondering if this problem can be solved using just one model particularly using Neural Network.

I have used Multilayer Perceptron but that needs multiple models just like linear regression. Can Sequence to Sequence be a viable option?

I am using TensorFlow. I have code but I think it is more important to understand what I am missing out in terms of the multilayer perceptron theory.

I understand that in MLP if you have one output node it will provide one output. If you have 10 output nodes then it is a multi class problem. You pick the class with the highest probability out of the 10 outputs. But in my case it is certain there will be 8 outputs for same input.

Lets say, for a set of inputs you will get the 3D coordinate of something (X,Y,Z). Like, Inputs = {1,10,5,7} Output = {1,2,1}. So for the same input {1,10,5,7} I need to make models for X value Y value and Z. One solution is to have 3 different models using MLP. But I would like to see if I can have one model. So I thought about using seq2seq. Because the encoder takes a series of input and the decoder provides series of output. But it seems seq2seq in tensorflow cannot handle float values. I can be wrong about this though.


texts:
I'm using this dataset and i'm trying to do logistic regression

heart_data = pd.read_csv('../input/heart-disease-uci/heart.csv')
X = heart_data.iloc[:,:-1]
y = heart_data.iloc[:,-1]
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20,random_state=5,shuffle=True)

from sklearn.preprocessing import MinMaxScaler

nr = MinMaxScaler()
X_train = nr.fit_transform(X_train)
X_test = nr.transform(X_test)

from sklearn.metrics import accuracy_score

def cal(method,c=1):
    lr = LogisticRegression(C=c,solver=method)
    lr.fit(X_train,y_train)
    pre_test = lr.predict(X_test)
    pre_train = lr.predict(X_train)
    train_score = accuracy_score(y_train,pre_train)
    test_score = accuracy_score(y_test,pre_test)
    return train_score,test_score
for i in method:
    print(i,'->>>>>>>>',cal(i))

First is training accuracy and second one is testing accuracy.
why am i getting more testing accuracy over training?

And Is there another way to increase the both accuracy?
I'm using min-max scaling so are there any other normalization to increase the accuracy more or this is the best accuracy we can get using logistic?


texts:
Is it possible to get  recommendation on similar product using Mahout ?

eg :

I have data set of movies with following attributes

Movie_name, Actor_1, Actor_2, Actress_1, Actress_2, Director, Theme, Language

Now given a Movie_name the system should  recommend top 3 similar movies based on the attributes .

Can this be done using Mahout. If yes how ?


texts: Creating Domain specific Question Answering Systems

texts:
I want to create a domain specific QA system.

I am working on a project to create an QA system for a given text book (on a specific domain and answer all related questions)

One approach I am considering for this is to create an Ontology/Knowledge base and then use this for answer retrieval.

For this :

How can I extract data from documents and populate this to the KB ?
The question will be in natural language, how can I use this question to query from the knowledge base ?
Is this the best approach ? because domain knowledge is required here for creating ontologies. And there are n number of domains. So shall I need to create different ontologies for each domain ?
Can a single QA model can be used to cover different domains ? Is there any other Machine learning and Deep learning approaches can be used for this ?


texts: Cross validation when training neural network?

texts:
The standard setup when training a neural network seems to be to split the data into train and test sets, and keep running until the scores stop improving on the test set.  Now, the problem:  there is a certain amount of noise in the test scores, so the single best score may not correspond to the state of the network which is most likely to be best on new data.

I've seen a few papers point to a specific epoch or iteration in the training as being "best by cross-validation" but I have no idea how that is determined (and the papers do not provide any details).  The "best by cross-validation" point is not the one with the best test score.

How would one go about doing this type of cross validation?  Would it be by doing k-fold on the test set?  Okay, that gives k different test scores instead of one, but then what?


texts:
I'm trying to delete part of the name of some cells of a data frame with gsub() but the function doesn't "have effect". The code I used is the following:
data<-data.frame(a1=rnorm(10),b1=rnorm(10),c1=rnorm(10))
gsub("-","",data$c1)

At the end the "-" remains in the cells of the column, what am i doing wrong?


texts:
I've an array like this:

 array([[ 0,  1],
        [ 2,  3],
        [ 4,  5],
        [ 6,  7],
        [ 8,  9],
        [10, 11],
        [12, 13],
        [14, 15]])

I want to make normalize this array between -1 and 1. I'm currently using numpy as a library.


texts: Features reduction for the not correlated data set

texts:
I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of features:

I am trying to find the right way to decrease the number of features.
All the methods of the important feature selection return different sets of features. For example:

The MARS returns only five important features, but,
Correlation with threshold >1% selects twenty of them.
Lasso selects near twenty but selected features are different from the feature set, returned by correlation method.
RandomForest selects near twenty-five features different from previous methods.
The feature rating in the returned feature sets is different.
The PCA method is not applicable, because there are not any visible linear correlated features.
etc.

My point is the different tools return different important features. One can combine/union them, one can intersect them. One can use N methods of selection and as a result, one will get N feature sets.

The method: "Check them all" is not applicable there. As an example, one can calculate the time effort if N sets multiplied M prediction models, multiplied 3 rounds of the model tuning. The time effort will be even more if one will union or intersect feature sets. It will take forever! There should be some tactics, an algorithm for the filter a final selection.

How to select the best set of the features if the data set is large, non-correlated and noisy?


texts:
I am working with a large dataset (approximately 55K observations x 11K features) and trying to perform dimensionality reduction to about 150 features. So far, I tried PCA, LDA, and autoencoder. The autoencoder that I tried was 12000-8000-5000-100-500-250-150-, all layers were Dense with sigmoid activation, except the final layer, which had a linear activation in order to reproduce the continuous data from the input. The autoencoder loss effectively plateaus after 10-15 epochs, regardless of the learning rate (here, I used the ReduceLROnPlateau feature in Keras). For the record, I am normalizing each feature by z-score prior to the training. I'm not sure how to get this loss to stop reaching a plateau.
Should my next attempt be to use a convolutional neural network on this dataset to see if I can reduce the dimensionality more successfully? Are there any pre-trained convolutional autoencoders that I could use? Training a convolutional autoencorder from scratch seems to require quite a bit of memory and time, but if I could work off of a pre-trained CNN autoencoder this might save me memory and time.


texts: In this post, https://cs231n.github.io/linear-classify/#webdemo it mentions "Softmax classifier interprets the scores inside the output vector f as the unnormalized log probabilities." Aren't we taking the log afterward and f is equal to just Wx+b?


texts: Multivariate Time-Series forecasting using LSTM

texts:
I have a dataset of hourly measures of pollution('Sample_Measurement) and weather condition. If I want to predict the pollution level of the current hour using the weather and pollution data of the previous n hours i have no problem. In fact, let's say that n = 24 and the number of features is 5. The 3D vectors will have the following shape (number of elements, 24,5).

Until this point I have no problem, but what i want to do is using also the weather condition of the current hour(together with the previous n = 24 hours), since in the previous case I was using only the data for the previous n hours. The problem is that i don't know how to do it since the 3D vector has as third dimension 5 (4 weather features plus 1 one that represents the pollution level), so i cannot simply add the current hour weather features because the number of feature(4) is different(since for the current hour we don't have the current pollution that is what we want to predict) from 5.
I hope I have been clear but it was not easy to explain.


texts: Is empirical risk the same thing as loss function?

texts:
I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk
$E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$
where $l(f(x),y)$ is a loss function, that measures the cost of predicting $f(x)$ when the actual answer is $y$.
Then, there is written:
We seek the function $f \in \mathcal{F}$ that minimizes the loss
$Q(z,w) = l(f_w(x),y)$ averaged on the examples.
I fail to see the difference between empirical loss and the $Q$ loss function, can anybody provide explanation? Also I am not native speaker so there might be misunderstanding in "averaged on examples".