texts: Training machine learning models from log files

texts:
Im starting to learn about ml and most of examples show very simple examples on how to train your ml modell. For me this looks more like statistic calculations than actual ml. These kind of examples could very easily been written in sql server etc.

What Im looking for is the ml to find pattern within data that i dont know about. For example supose i have a lot of devices which is logging temperature, humidity, hours used and much more. At some point these devices fails for some unknown reason and Will be marked as "failed"

So what om looking for is if there is a way to use ml to analyze all the log files and find common patterns to why these devices failed, without me have to "tell it what to look for first"  in this way I Will be able to predict feature failures based on log files.

Would this be something I can use ml for and would Azure be a good starting point for this?


texts: Calculating Feature Importance of Time Series Data

texts:
I am new to time-series modeling, and I was wondering what the standard way of quantifying feature importances are in a time-series setting? What types of models allow for the greatest interpretation of the feature space?

I am looking for something, which does not necessarily function like the Random Forest Regressor's feature importance call, but provides a similar insight.


texts: Binary classificaiton for weather data if its class 1 or class 0 alert

texts:
I am working on weather data and it has few features that are independent variables such as severity, severity_id, urgency_id etc ... Based on these values, I would like to classify alerts into class 0 or 1. For example, below is row item from data source

Alert | Severity | Sev_Id | Urg_Id | Event      | Sys_Rec(Target Variable)
--------------------------------------------------------------------------
dummy | Extreme  |   1    |    1   |   STORM    |      1
dummy | Minor    |   3    |    5   |   RIPTIDE  |      0
dummy | Extreme  |   1    |    1   |   HURRICANE|      1

For severity_id 1 it should be class 1 (Yes) and for others its class 0 (No).The objective is to build a general binary classifier using decision trees. So I started with DTClassifier, but later I realized it could also be done with logistic regression. I am confused which would be a better fit for this kind of data for classification.

Please advice and give me some starting points.


texts: Canopy clustering implementation in Python

texts:
As far as I understand, canopy clustering preprocessing procedure is not implemented in scikit-learn.

Is there a python library for this?


texts: How to handle non consistent time series( using LSTM )

texts:
The time series dataset I am working on has missing samples.
I am trying to use keras and LSTM for prediction.
How should I handle the missing timestamp samples ( sometimes there are missing weeks even months)?

Does this affect the overall performance of the LSTM predictor?


texts: NLP - Simple approach to identify commonalities in text comments between people

texts:
For something we are working on, we were looking for a simple way to compare from review/feedback data against a question (for which there are multiple responses from multiple people), the following:

What are the common things (things defined as phrases/sentences) they are saying (Some way to quantify the commonality too if possible). The point is to identify what seems to be areas of agreement about their review
What are things that are not common (basically...what are those on-off sentences/phrases that have been told that are very uncommon)
Where is there disagreement (i.e. are there sentences/phrases where there is disagreement potentially between the responses)

The goal is to find a simple solution to this and not necessarily model driven (there is paucity of data). Also...it needs to be directional at this time...as the goal is to prove that this can work and can product reasonable results.

Any help advise?

Thanks much!

PS: We would need some 'intelligence' in identifying the commonality or vice-versa (i.e. different words meaning same within a phrase should be considered common).


texts:
On this site of Otto Group Product Classification Challenge, it is shown that best accuracy was possible with RandomForest method, but it was relatively low at 0.83. Accuracy with ANN and with Naive Bayes was even lower (0.72 and 0.65, respectively).

What is the cause of low accuracy achievable in some data? Are there any other methods with which accuracy can be increased here?

Edit: On this site: https://www.kaggle.com/c/otto-group-product-classification-challenge/leaderboard best model seems to have a score of 0.38. What will be corresponding accuracy for that?


texts:
I am using the stepAIC function in R to do a bi-directional (forward and backward) stepwise regression. I do not understand what each return value from the function means.

The output is:

          Df     Sum of Sq    RSS       AIC
<none>                        350.71   -5406.0
- aaa      1     0.283        350.99   -5405.9
- bbb      1     0.339        351.05   -5405.4
- ccc      1     0.982        351.69   -5400.5
- ddd      1     0.989        351.70   -5400.5

Question Are the values listed under Df, Sum of Sq, RSS, and AIC the values for a model where only one variable would be considered as the independent variable (i.e. y ~
aaa, y ~ bbb, etc.)?


texts: What is the best technology to be used to create my custom bag of words with N-grams to apply to. I want to know a functionality that can be achieved over GUI. I cannot use spot fire as it is not available in the organization. Though i can get SAP Hana or R-hadoop. But R-hadoop is bit challenging, any suggessions.


texts: KNN scoring low compared to Logistic regression in MNIST challenge

texts:
KNN gives me a score of 0.76100 while it shows 94% accuracy for my training data (splitted with test_size =0.3) in my jupyter notebook while logistic regression gives me a score of 0.91485 with an accuracy of 92 %.I do not understand the reason. Please can anybody help me or suggest anything.Here is my model:

   from sklearn.neighbors import KNeighborsClassifier
   knn=KNeighborsClassifier(n_neighbors=1)
   knn.fit(X_train,y_train)
   pred = knn.predict(X_test)

below is the classification report :

                    precision    recall  f1-score   support

               0       0.94      0.99      0.96      1213
               1       0.95      0.99      0.97      1422
               2       0.95      0.92      0.93      1258
               3       0.92      0.94      0.93      1284
               4       0.93      0.94      0.94      1209
               5       0.93      0.91      0.92      1121
               6       0.96      0.97      0.97      1242
               7       0.93      0.93      0.93      1315
               8       0.97      0.89      0.93      1227
               9       0.91      0.91      0.91      1309

        accuracy                           0.94     12600
       macro avg       0.94      0.94      0.94     12600
    weighted avg       0.94      0.94      0.94     12600

Confusion matrix :

[[1190    2    2    4    1    2   10    0    0    2]
 [   2 1404    7    3    2    1    2    0    1    0]
 [  14   11 1157   25    8    0   10   15   11    7]
 [   0    1   19 1191    1   34    1   18   12    7]
 [   2    8    6    0 1125    2    4   11    3   48]
 [   2    2    3   36    5 1030   16    2   15   10]
 [  18    4    4    1    4    9 1200    0    2    0]
 [   1    8    7    6   10    1    0 1224    3   55]
 [   8   14    8   30   13   31    9    6 1093   15]
 [   4    2    4    6   38    6    0   51    8 1190]]

Thanks in advance


texts: On ttest why t-stat is greater than one?

texts:
I have my data in two culomns as below:

On excel did this ttest and this is my result:

I can not Understand why Hypothesized Mean Difference =0 but tStat = -1.6367...?

I mean my NULL/Zero Hypothesized has been accepted and t-Stat must be close to zero, am I wrong?


texts:
I'm looking for Jupyter extension to plot interactive graphs. For example I need to plot twenty time series lines with order to examine data. Only plugin which I've found is bqplot https://github.com/bloomberg/bqplot but its documentation is too complicated.

Could you advice some other free tool or help with bqplot?

I can't plot lines with different colors.

unitIds = commonSeries['UnitId'].unique()
commonSeries = commonSeries.sort_values(by=['Date','UnitId'], ascending=True)
commonSeries = commonSeries.set_index(pd.DatetimeIndex(commonSeries['Date']))

plt.clear()
x_ord = OrdinalScale()
y_sc = LinearScale()
lines = []

for unitId in unitIds:
    timeseries = commonSeries.loc[commonSeries['UnitId'] == unitId]
    line = plt.Lines(x=timeseries.index, y=timeseries['Value'], scales={'x': x_ord, 'y': y_sc}, colors=['red','green','blue'])
    lines.append(line)

plt.Figure(marks=lines)


texts: I understand that GPU can speed up training for each batch multiple data records can be fed to the network which can be parallelized for computation. However, for inference, typically, each time the network only processes one record, for instance, for text classification, only one text (i.e., a tweet) is fed to the network. In such a case, how can GPU speed up?


texts: How to add extra constraints to an equation？


texts: Performance issues when merging two dataframe columns into one on millions rows with Pandas

texts:
I am trying to merge two address columns into one and separate the resulting string with '--'. The dataset has 10 million rows and 33 columns - but the number of rows grows for a million or so a month. This line in Pandas/Python is very slow. Any ideas of how can I make it faster and scalable for future use?

df['address'] = df[['address1', 'address2']].apply(lambda x: '--'.join(x.dropna().astype(str).values), axis=1)


texts: How to implement single Imputation from conditional distribution?

texts:
In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as:

I did not find any code implementation of this approach. My question is, how to implement it? Should we integrate the distribution w.r.t an assumed interval of $\bf{x}_{mis}$? Otherwise, is this just an intuitive mathematical representation that should be understood but the implementation is different.

[*] Theodoridis, S., & Koutroumbas, K. “Pattern recognition. ” Fourth Edition, 9781597492720, 2008


texts: which algorithms can be used to extrapolate non-linear data?

texts:
I have a dataset, where target value changes in time in following way:

I need to predict target value for upcoming month, however I struggle to find a method to extrapolate the function that defines target. It is certainly not linear.

I tried to extrapolate the max/min peaks separately using linear regression, but not sure what to do with all other values.


texts: Can feature importance change a lot between models?

texts:
I have a random forest classifier and Multinomial Naive Bayes. For feature importance, I used gini index for random forest and for Multinomial Naive Bayes I used the coefficients of each feature. Then normalized to compare the two lists but there is a big difference between the two. Is this normal? Something like

RF vs NB

A - C
B - D
C - A
D - B


texts:
i'm currently doing dual apprenticeships. My main mission is to represent the health of a company based on accounting records for multiple companies over multiple years. The part of an accounting record i'm interested in is simply the date and the amount spend or receive. We're in a context of time series.
The objective is to build scores, each one representing a part of the company. I have a few indicators for each one of this company's part.
I didn't find any scientific articles about it, so i proposed an algorithm to do so. Here is it :
For each indicator

Build the mean for this indicator for each company
Use a regressive model to get the trend
Compute the difference to this trend (the objective is to increase the amount of data)
Make a forecast
Build the mean for this forecast
Use a regressive model on this forecast to build the trend
Compute the difference to this trend for each company
Based on this difference, build the score

I know this is not perfect, but after a lot of thinking, that's what i ended up with.
For the forecast part, i'm planning to use an LSTM.
I would like to hear your thought about my algorithm, even if you think it is completely crap. My objective is to improve my skills and to build the best system i can. I'm still a bit lost : should i try to transform the problem and go for more classical, like a simple classifier, or should i do that just for the last part ? Well, i don't know.
Thank you.
EDIT
As Julio made me realize, i forgot to mention one important thing. In France, each one of the record has an identifier, which represent what type of of income/outcome it is. For example : each accounting record with the id 40 are accounting charges. In this example i showed you a simple 2 digits identifier, but it can go up to 8.


texts: Comparing transition matrices for Markov chains

texts:
I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions.

My population can be segmented into various subpopulations of interest. I've obtained the transition matrices for each of these subpopulations, and would like to know if these subpopulations differ from the general population in some principled way.

I don't know of any principled way of comparing transition matrices in this way. Comparing the transition matrices row-wise to that of the general population seems like one approach, but I'm not sure how to go about interpreting this. Another approach might be a spectral/eigendecomposition approach, which is much more readily interpretable to me, but might be harder to squeeze insight/stylised facts from.

I've had a cursory search of the literature without much luck. Any suggestions?


texts: What is the meaning of "The number of units in the LSTM cell"?

texts:
From Tensorflow code: Tensorflow. RnnCell.

num_units: int, The number of units in the LSTM cell.

I can't understand what this means. What are the units of LSTM cell? Input, Output and Forget gates? Does this mean "the number of units in the recurrent projection layer for Deep LSTM"? Then why is it called "number of units in the LSTM cell"?   What is an LSTM cell and how is it different from an LSTM block, what is the minimal LSTM unit if not a cell?


texts: Information Gain of a Customer ID Attribute

texts:
Suppose I have a dataframe like the following
CustID  ...  Class
1       ...     C0
2       ...     C1
3       ...     C1
4       ...     C0
5       ...     C0
6       ...     C1

I am trying to find the information gain, and I was wondering if this is the correct process?
I began by finding the information gain of the parent node
$$I(\mathrm{parent}) = -(3/6) log_2 (3/6) - (3/6) log_2 (3/6) = 0.5 + 0.5 = 1.$$
Then, the information of the children node, Customer ID.
$$I(\mathrm{children}) = -(1/6) log_2 (1/6) - (1/6) log_2 (1/6) - (1/6) log_2 (1/6) - (1/6) log_2 (1/6) - (1/6) log_2 (1/6) - (1/6) log_2 (1/6) = 6 \times (- (1/6) log_2 (1/6)) = 2.584963$$
However, $$\Delta = 1 - 2.584963 = -1.584963.$$ Is it possible to have a negative information gain?


texts: Different types of Ensemble learning methods

texts:
I've been reading and searching information about different types of Ensemble learning methods however I am a bit confused and want to make sure my understanding is correct.
Below is graph of how I understand ensemble learning methods. Is it correct?

Also I have some questions:

for Bagging and Pasting do we train different subsets of data with all same models? can we use different models for each subsets?
Does Boosting use whole dataset?
Is there hard voting/ soft voting in regression problems? or do we just average output of models?


texts: Using the whole dataset for testing (not validation) in case of small datasets

texts:
for an object detection task I created a small dataset to train an object detector. The class frequency is more or less balanced, however I defined some additional attributes with environmental conditions for each image, which results in a rather uneven distribution of classes depending on the viewed attribute (e.g. class X occurring with attributes A and B).
I used a typical training/validation split and the loss curve lets me conclude that no over-fitting is occurring. I am aware that in general for testing unseen images with a realistic feature distribution should be used, however due to the small size of the dataset, splitting off additional 10 or 20 % for testing or randomly drawing examples would result in certain combinations of classes and attributes not or only barely occurring in the test dataset and manually selecting examples would be very time-consuming.
I am therefore now wondering, if it reasonable to use the whole dataset (train+val) for testing and calculation of performance metrics as a compromise since I see no signs over-fitting?


texts:
As per my understanding, while learning the features, the Convolution part of the CNN works-on and preserves the 'spatial relationship' between the pixels.
But, I want to identify object(s) in my image whose definition might be all scattered throughout the image.
By scattered I mean - The identity of object may exist in separate disconnected islands of pixel-sets. Where each of these island in itself is a connected component of pixels.

In other words- my 'Cat' exists, not as a whole, but is broken up into pieces.

Would traditional CNN still identify my 'Cat'?
If yes, upto what level? If my Cat is broken into 4 pieces? 100 pieces?
If not, what adjustment(s) do I need to make so that I can solve this problem?

Thanks,


texts: Why the accuracy decreased with more data

texts:
When working with a random forest model, it is surprising that the cross-validation score on the training dataset is much higher than the score on the whole dataset.

Here is the code:

X_train, X_holdout, y_train, y_holdout = train_test_split(X, y, test_size=0.2)
model = RandomForestClassifier(n_estimators=100)
cv_score1 = cross_val_score(model, X_train, y_train, cv=3).mean()
print('CV score for the training data is : {:.3f}'.format(cv_score1))
cv_score2 = cross_val_score(model, X, y, cv=3).mean()
print('CV score for all data is : {:.3f}'.format(cv_score2))

The first score is 0.887 while the second score is 0.645.
The trend is the same with a linear support vector machine model.
Is this normal? Or I have done something wrong? How can I fix it?
Thank you!


texts: Fuzzy and FuzzyWuzzy: what are the differences in text comparison?

texts:
I have found a lot of information about fuzzy logic, but less information about fuzzywuzzy. I would like to know more about this, the function which determines the logic, if possible, and understand what partial_ratio in Python does.
Any information will be well welcome.


texts: Is it valuable to normalize/rescale labels in neural network regression?

texts: Have there been any papers, or does anyone have any specific experience to know whether normalizing labels in a regression problem is likely to improve the performance of a neural network? I have labels that are in the range (0,1000) applying square loss in a ConvNet. I want to know if it might be useful to normalize these to a (0,1) range, or whether that's known to not matter.


texts: What are some reference sources for understanding Markov switching models?


texts: How to force pytorch model to predict only positive values

texts:
I am doing a prediction on a data set where labels have positive values (time values). After training a simple Linear pytorch model I get negative values for time despite being 0 negative values in the training set.
Is there some way to punish the model hard for making negative predictions so that the model would understand to make positive predictions (because taking ReLU does not feel correct to me)?


texts: Should I give regularly-spaced or irregular-timestamped data to a price predicting neural network?

texts:
I am building an application to predict the price of an item. Data is collected at regular 5-minute intervals while the application is running. Unfortunately, there is downtime, so there is not a full track record of data points. I'm using a multi-layer perceptron network to do the prediction, my question is:
Should I give the network continuous series of data? Or should I give it timestamped, but irregularly spaced data?
index | timestamp | price
----- | --------- | -----
1     | 5:20      | 0.80
2     | 5:25      | 0.78
3     | 5:30      | 0.66
4     | 5:35      | 0.72
5     | 6:24      | 0.91
6     | 6:29      | 0.89
7     | 6:34      | 0.92

Training with regularly spaced data would give prices 1-4, and 5-7 as separate data sets (the network sees '0.8, 0.78, 0.66, 0.72' and '0.91, 0.89, 0.92'). Training with irregularly spaced data would give the network '0.80 @ 5:20 , 0.78 @ 5:25 ... 0.92 @ 6:34' in one clump, but the network would know the times.
Ideally, I'd like to be able to make a prediction at multiple distances into the future, not just "what will it be in 5 minutes' time".
Note: The data I have access to each time I collect data, for both buy and sell, is 'orders placed', 'price per unit', 'units transacted in the last week', and 'avg volume of items per order.'


texts:
I am trying to use DBSMOTE(Density-Based Synthetic Oversampling TEqnique) to on a data set of short text--tweets to be specific. This will be used to train a classifier model in a multiclass classification model. This will be done at feature level augmentation, using TF-IDF as features. I have read though, that to use SMOTE on NLP, the feature vectors must be reduced in dimension. What is an optimal size of a feature vector to use in a SMOTE family algorithm?

Similar Question: How do you apply SMOTE on text classification?

DBSMOTE Code: https://rdrr.io/cran/smotefamily/man/DBSMOTE.html

DBSMOTE Paper: https://link.springer.com/article/10.1007/s10489-011-0287-y


texts: Low training and validation loss but bad predictions

texts:
what does it mean if in a neural network, the training and validation losses are low but the predictions (so use model on test set) are bad?

Neural network architecture:

model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=100, return_sequences=True, dropout=0.5, recurrent_dropout=0.5), input_shape=(timesteps, features)))
model.add(Dense(1, activation='sigmoid'))

My loss function is:

def get_top_one_probability(vector):
  return (K.exp(vector) / K.sum(K.exp(vector)))

def listnet_loss(real_labels, predicted_labels):
  return -K.sum(get_top_one_probability(real_labels) * tf.math.log(get_top_one_probability(predicted_labels)))

Fitting settings:

model.fit(training_dataset, training_dataset_labels, validation_split=0.2, batch_size=1,
            epochs=number_of_epochs, workers=10, verbose=1,
            callbacks=[SaveModelCallback(), keras.callbacks.EarlyStopping(monitor='val_loss', mode='auto', patience=3)])

Thanks in advance.


texts: What causes the network validation loss to always be lower than train loss?

texts:
I am using a LSTM model to predict values in a time-series based network. However, my problem is that every time, I get consistently the validation loss lower than the training loss. Even at the starting of the epochs, the model always has validation loss lower than train loss which seems weird to me. Basically, I have created a subset of my Training Data (which after splitting into Validation has 42005 samples), which I use for validation (having 10502 samples). And then, there is a separate test set, which is used to evaluate the RMSE and MAPE for the network. I use a test set with 17667 samples for this very purpose.

The following outlines the essence of the network:-

# design network
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.add(Dense(1))

model.compile(loss='mae', optimizer= 'Adam') # also try mae

# fit network
es = keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, verbose=1 , mode='min')
history = model.fit(train_X, train_y, epochs= 100, batch_size= 512  , validation_data=(val_X, val_y), verbose=2, shuffle=False, callbacks= [es])

I have also tried various batch sizes, including 128, 512, 1024, and even lower values like 10,20,40 etc., and also with only 1 dense layer or multiple dense layers, but always the test loss is lower than train and due to early stopping, the model stops at at most 5-6 epochs, which seems very weird. The loss curve for the network is accordingly shown in

I have looked at Stackexchange, Keras Documentation and related blogs, but still, the problem isn't clear to me. Any help and suggestions on how I can resolve this would be highly appreciated.

EDIT

Used activation function 'tanh' as per:

model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1, activation='tanh'))
model.add(Dense(1, activation='tanh'))

New loss curve is still weird as in :-


texts: Relation between Cross Validation and Confidence Intervals

texts:
I've read from a source which I forgot where that 'In cross validation, the model with best scores at 95% confidence interval is picked'.
But according to my stat knowledge, in order for CI (confidence interval) to works, you need normality assumption about the sampling statistics of the experiment.

But how come from that unknown source it seems to simply use results from each flow to construct the sample mean & the confidence interval. It seems to me that neither checking if central limit theorem testing at all. And it seems to me this is what people are doing as well:
i) automatically assume normality in sampling MEANS (instead of sampling distribution)
ii) CLT automatically satisfied.
May I know if it's my misunderstanding or the industry is adopting a norm which is too loose? Thanks.


texts:
When Keras prints the validation loss, it prints the total loss value (regularization loss + fitting loss).

How can I get the two separately? I'm specifically looking to obtain value for regularization loss.


texts: What gradient boosting library is the easiest to implement on mobile?

texts:
I think of implementing training and predicting in an app (both Android and iOS), but existing packages I found not seem to be very mobile-friendly (scikit-learn, xgboost, lightgbm).

Random forest will also work fine.

Maybe you guys could advise an easier way so it does not require a lot of porting?

Would appreciate any help, Thanks!


texts: Predict future value with time period using non linear regression model

texts:
Here I have dataset import from csv file. I want to predict the next value with the time series. Can we use nonlinear regression model to predict the value for next time period or Is there any regression model can we use to predict the value?

Here I upload the subset of my original dataset.

 Time     x     x1   x2     y
0:06:00  63    NaN   NaN   63
0:07:00  63    NaN   20    104
0:08:00  104   11     0     93
0:09:00  93    0      0     ?


texts: How to split pieces of dataframe and create new dataframes based on it?

texts: I have the following dataframe that I need to split bu Product and create a new dataframe for each product contain the series ds and y. How can I do that?


texts: What are some data analytic package & feature in python which helps do data analytic?


texts: what are the step need to be followed inorder to retrain any pretrained neural network model?

texts: what are the step need to be followed inorder to retrain any pretrained neural network model ? (How to load the pre-trained BERT model from local/colab directory? ) I tried to re train few pretrained neural network models . After retraining i get 3 CKPT files but im stuck on how i should proceed further after getting the 3 CKPT file . How to load the model buy pointing to these CKTP files ?


texts: results from "Google Vizier: A Service for Black-Box Optimization"

texts:
In Google Vizier: A Service for Black-Box Optimization it is show what kind of optimizers google uses internally.
Figure 6 shows some benchmark results compared to Random Search. Am I right in understanding that "GP Bandit" has better results as "CMA-ES" in 8 Dimensions? So is a lower number better?

Thank you


texts: Presentation of data - Correctness of using exponentially weighted moving average values in charts

texts:
I have a set of data that I would like to present to an audience in a chart.
The data points span several years and they each year needs to be presented as separate series.
The datapoints are daily, so a single series data looks 'busy':

My intention is to make the chart readable (overlapping multiple high-frequency series would look messy), and it is clear that I could acheive that by removing the high-frequency changes in the data.
I have attempted to group the values by downsampling from 1 day to 10 days:

Is this a valid way to solve the presentation problem? What would you do?
EDIT:
My question is specifically about the correctness of presenting exponentially weighted data to the audience who may not understand the implications of it.
Please consider the grouped values versus e.w. values:

A sharp drop in value becomes a much smaller dip. That looks like potential misrepresentation to me, hence my question about the conventions of presenting the data.
Trying to answer my own question, I would reason that I must choose the decay (span) parameter for the exponential weighted function, so that the result doesn't distort the 'meaningful trends' in data. Woh is to decide what is a 'meaningful trend'? My personal judgement?


texts: Uncertainty in connection to explainability

texts:
When I write "uncertainty" in this post I mean:
If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the uncertainty of this decision is $1-p_i$.
I'd like to inquire about connections of this notion and that of accuracy and explainability.
For example, if I have a classifier that is "very certain" (on mean/median on the test/training set) how often is this property correlated to achieving real-time accurate predictions? What about vice-versa?
Moreover, if my classifier is "certain" how does this affect my ability to explain its decision in any sense?
I couldn't find good resources for this notion of uncertainty and these questions so I will really appreciate some references as well!


texts: How do I calculate a similarity matrix with a Student-t kernel?

texts:
As the title says, how do I calculate a similarity matrix with an un-normalized Student-t kernel? I'm attempting to calculate Kullback-Leibler divergence for different t-SNE runs, but need a Q-matrix for that. A few steps before the Q-matrix, I need the similarity matrices made using the un-normalized Student-t kernel.

I'm using r, not sure if that's relevant to an answer.


texts: Comparing two images and showing the difference in a new image?

texts:
I would like to compare two web pages images using computer vision techniques.
Show what are non-unique portions comparing both images. Which part image1 not exist in  image2 vice versa.


texts: A Derivation in Combinatory Categorial Grammer

texts:
I am reading about CCG on page 23 of Speech and Language processing. There is a derivation as follows:

(VP/PP)/NP , VP\((VP/PP)/NP) => VP?

Can anyone example this please? This make sense if

VP\((VP/PP)/NP) is equivalent to (VP\(VP/PP))/NP

and

(VP/PP)/NP is equivalent to VP/(PP/NP).

But they seem at least non-trivial from the text!

Any help would be greatly appreciated.

CS


texts:
I am performing simple linear regression using PyTorch but my model is not able to properly fit over the training data. please look at the code to find the mistake.
Dataset is here
import torch
from torch import nn
from torch.optim import SGD, Adam
from torch.autograd import Variable

class Linear_Reg(nn.Module):
    def __init__(self):
        super(Linear_Reg, self).__init__()
        self.linear = nn.Linear(1,1)

    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

net = Linear_Reg()

Xt = Variable(torch.Tensor(X[:,0]))
yt = Variable(torch.Tensor(y))
Xt = Xt.view(-1,1)

criterion = nn.MSELoss()
optimizer = Adam(net.parameters(), lr=0.001)

EPOCHS = 500
for epoch in range(EPOCHS):
    pred_y = net(Xt)
    loss = criterion(pred_y, yt)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print('Eopch: {}, \t\t loss: {}'.format(epoch, loss.data.item()))

The loss decreases from ~68.88 to ~30.26
and the resulting fitting is this:
xxt = torch.arange(5,23)
with torch.no_grad():
    a = net(xxt.reshape(-1,1).float())
plt.scatter(X[:,1], y, s=30, c='r', marker='x', linewidths=1)
plt.plot(xxt.data.numpy(),a.data.numpy(), label='Linear regression (Gradient descent)')

What is the problem here?


texts: Encoding of high cardinality multi-label categorical feature?

texts:
This is the problem of binary classification:
"1" - the subscriber is a driver (belongs to the segment of drivers),
"0" - the subscriber is not a driver (does not belong to the segment of drivers).
The files tabular_data.csv and hashed_feature.csv  ̶    here are descriptive characteristics for 4084 subscribers ("ID" is the subscriber ID).
The train.csv file is data about the target label (subscriber id and corresponding binary target).
The tabular_data.csv file contains numerical data on the subscriber's activity for 12 periods.
• period - period number (consecutive periods, 1 - the newest);
• id - subscriber ID;
• feature_0 - feature_49 - data on the subscriber's activity in the corresponding period.
The hashed_feature.csv file is a set of hashed values of one categorical variable for the subscriber.
• id - subscriber ID;
• feature_50 - hash from the value of the categorical variable.
I faced the following problems:

How should I encode feature_50 for ML algorithm? This feature contains approximately 5k unique values and each user can have almost 1k of these values. As I think one hot encoding is not very helpful here, but what technique should I use in this case?
NA filling strategy for feature_0 - feature_49 (I've only tried pandas filling strategies)
Make use of period variable?
I would appreciate any help.


texts:
Ex: Matlab's t-sne tutorials frequently use PCA

https://www.mathworks.com/help/stats/tsne-settings.html

" Process Data Using t-SNE

Obtain two-dimensional analogs of the data clusters using t-SNE. Use the Barnes-Hut algorithm for better performance on this large data set. Use PCA to reduce the initial dimensions from 784 to 50. <- (1) Why are we using PCA here to reduce dimensions to 50 first at all if we are going to use t-sne after PCA to reduce to 2 dimensions anyway?

Matlab Tutorial Code: https://www.mathworks.com/help/stats/tsne-settings.html

rng default % for reproducibility

Y = tsne(X,'Algorithm','barneshut','NumPCAComponents',50);

figure gscatter(Y(:,1),Y(:,2),L)

1) See question bolded above

2) What would you have googled to find this out?

I had googled "Why is PCA often used before t-sne for problems when the goal is only to reduce dimensionality? "


texts: How to normalise(?) an [x,y] time series data set

texts:
Problem:
I'm currently parsing a time series dataset, of [x,y] coordinates. The data isn't complete - it contains gaps and jitter, and I would like to fill these gaps / normalise the jitter using statistical analysis.
Background:
I'm currently reading up on non-linear regression (specifically polynomial regression -> PR) - which seems to be the best fit (pun intended) for my problem.
I realise that PR deals with arcs "turning in one direction" so, I'm going to try to refactor my code to work with smaller sample sizes - and work my way along the time series.
Questions:

Am I on the right track?
Is there a name for what I'm trying to do? i.e. using polynomial regression across an ongoing graph (curve fitting? trendline? continuous regression?)
Is there another technique I can/should use, that provides a better "fit" for my data?


texts: I've heard a lot of buzz for Julia programming language and how it's the future of AI. Should I buy into the hype and start learning Julia? Or do you think its better to stick to a better established language (e.g. python). What are the pros and cons of Julia compared to other languages aimed towards data science (python, R, etc.)?


texts: CNN for image classification with two outputs

texts: Is it possible to classify my images (cars parts) by the type of cars part(door, window ...) and also by the view of the image( front, back, right, left, top and bottom). My pictures are labelled like this: View_idPart, the view is a number from 2 to 7. I want to use a CNN model, but i don't know if this is possible? I hope that I will have some answers, I will be so grateful


texts: Ways to speed up Python code for data science purposes

texts:
Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the data retrieval is not a problem and that it also fits in memory etc). Some of those could be the following, but I would like to receive feedback about any other else:

good practices as always using Numpy when possible on numeric operations and not loops...
more good practices like using 'apply', 'applymap'... instead of loops when applying functions to elements of lists, dataframes, etc
Numba applied on native python loops, numpy arrays...
multiprocessing with multiprocessing library depending on the available logical cores

This is motivated by the fact that, if we mainly use Python with all its advantages, we do not want to switch to other languages like Scala or Julia, unless there is no alternative.


texts:
I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in other words. I have more than a thousand possible product names.

Currently I'm just using regular expressions with a list of product names to find and extract the product names in each row of the dataset, but it's not working well in the cases of misspelled product names.

Since I already have more than 50,000 rows in which the product was extracted by hand in cases where there is some product from the list in the column. I am wondering if after breaking each description into multiple lines (tokenization) could apply some classification/search method to detect if there is some product and which is in the description.

  Example: Medical product used in cancer treatment "PRODUCT XXXXYYYY"
  where the XXXX and YYYY are the first and second name of the product.

I think most of the description would be unused, because as in the example above, simply being a medical product would not help at all since there are several possibilities, but the presence of the wrongly spelled words that describe a product specific would be helpful. I am open to suggestions on how to deal with these spelling errors.


texts: Find intersection of data between rows and columns

texts:
Below is an Excel sheet with crops and countries. "1" means that the crop is grown in that country whiles "0" means it is not grown in that country.

I want to find where there is an intersection (as in I want to find similar crops that are grown in most of the countries or at least most of the countries.) How do I do this ?


texts: What to use pretrained models (Glove) or train my own model?

texts:
I have been using pre-trained models such as google news or Glove 6B model but many words in my text data does not have their vectors representation in those pre trained model. So I was thinking of maybe train my own model with the data I have.

Is there any disadvantages in training our own model for two class classification model? or Should I keep using pre-trained model. What is the difference between training our own model or using pre-trained model.

# This is how I am thinking to train the model
from gensim.models import Word2Vec
w2v_model=Word2Vec(list_of_sentance_train,min_count=5,size=50, workers=4)


texts:
In the papers "Convolutional Sequence to Sequence Learning" and
"Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order of the input sequence. These position embeddings are generated from a sinusoidal signal depending on the absolute position of the word in the sequence and the dimension. We obtain position embeddings of the same dimension as the word embeddings and we simply sum these two.

I can understand that this helps the model to get a sens of the ordering of the input, but I'm quite disturbed by the fact that adding these two might also erase some of the information contained in the word embeddings. Do you have an explanation on why this might work (or not) ? Is there some literature about it ?


texts: How do I implement masking in TensorFlow eager execution?

texts:
I am training a stateful RNN on variable length sequences (optional: see my previous question for more details).

I padded the sequences to a fixed length with the value -1.

The when batches are loaded, some samples will be entirely -1. (e.g. the batches are shape [batchsize, ...] and samples 1,6,8 may be entirely composed of -1's). I would like to:

not include these samples in the loss function calculation
not perform operations on them so as to speedup training.

Attempt 1:

I tried using tf.Keras.layers.Masking as in:

input = tf.Keras.layers.Masking(mask_val=-1)(input)

But, this doesnt seem to do anything. The subsequent operations are still performed, and as far as I can tell the samples are still included in the loss function. Why is this?

Attempt 2:

I tried making my own custom layer which would actually remove samples which are masked (see code below). For example the input shape goes from [batchsize, ...] to [adj_batchsize, ...] where adj_batchsize = batchsize - num_removed_samples. This works, but is extremely slow, since every time the input shape changes, the GPU memory needs to be reallocated which slows training down by a lot.

class MaskLayer(K.Model):

    def __init__(self, mask_val):
        '''
        This layer takes input tensor of [batchsize, ...] and returns tensor of shape [bs_out, ...]
        Where all samples composed entirely of mask_val are removed and bs_out is the number of remaining
        samples
        '''
        super(MaskLayer, self).__init__()
        self.mask_val = mask_val
        self.mask  = tf.keras.layers.Masking(mask_value=mask_val)

    def call(self, data, lbls):

        good_batches = tf.where(tf.math.reduce_all(self.mask.compute_mask(data), axis=range(1,data.ndim-1)))[:,0]
        data = tf.gather(data, good_batches)
        lbls = tf.gather(lbls, good_batches)
        return data, lbls

So what would be the best way to do this?

NOTE: This question was previously asked on stackoverflow.coom, but deleted due to lack of response. I think this will be a better home for it.


texts: When do you use FunctionTransformer instead of .apply()?

texts:
I'm watching a PyData talk from 2017 in which the speaker provides this example for how to use FunctionTransformer for sklearn.preprocessing

from sklearn.preprocessing import FunctionTransformer
logger = FunctionTransformer(np.log1p)
X_log = logger.transform(X)

In other words, she's applying a function over the rows of a column.  I assumed this could be done more simply using .apply().  I feel that there must be something more to the reason why a data analyst would import FunctionTransformer.  Could someone help me understand what differentiates the .apply() method from FunctionTransformer?


texts:
I'm trying to build a model using GBM in r in order to get probability of two classes ( 'yes','no'). My data are unbalanced, and because of this I trained my model using a balanced data(undersampling the 'no' class). I trained my model With the balanced sample and then I evaluate the performance on my test sample ( 30% of the balanaced sample).

My question is:

There is any problem to predict the probability with my fit in the unbalanced sample? When I performe this, the probability distribution are equal of the probability distribution of my balanced sample, and because of this maybe my score is meaningless


texts:
Suppose I am given a set of structured data. The data is known to be problematic, and I need to somehow "score" them on consistency. For example, I have the data as shown below:

fieldA | fieldB | fieldC
-------+--------+-------
foo    | bar    | baz
fooo   | bar    | baz
foo    | bar    | lorem
..     | ..     | ..
lorem  | ipsum  | dolor
lorem  | upsum  | dolor
lorem  | ipsum  | baz

So assume the first row is considered the correct entry because there are relatively more data in that combination compared to the records in second and third row. In the second row, the value for fieldA should be foo (inconsistent due to misspelling). Then in the third row, the value of fieldC should be baz as other entries in the dataset with similar values for fieldA (foo) and fieldB (bar) suggest.

Also, in other part of the dataset, there's another combination that is relatively more common (lorem, ipsum, dolor). So the problem in the following records are the same as the one mentioned before, just that the value combination is different.

I initially dumped everything to a SQL database, and use statements with GROUP BY to check consistency of fields values. So there will be 1 query for each field I want to check for consistency, and for each record.

SELECT    fieldA, count(fieldA)
FROM      cache
WHERE     fieldB = 'bar' and fieldC = 'baz'
GROUP BY  fieldA

Then I could check if the value of fieldA of a record is consistent with the rest by referring the record to the object below (processed result of the previous SQL query).

{'foo': {'consistency': 0.99, 'count': 99, 'total': 100}
 'fooo': {'consistency': 0.01, 'count': 1, 'total': 100}}

However it was very slow (dataset has about 2.2million records, and I am checking 4 fields, so making about 9mil queries), and would take half a day to complete. Then I replaced SQL storage to elasticsearch, and the processing time shrunk to about 5 hours, can it be made somehow faster?

Also just out of curiosity, am I re-inventing a wheel here? Is there an existing tool for this? Currently it is implemented in Python3, with elasticsearch.


texts: Linear Regression Model Validation with Transformed Data

texts:
I worked on a model that I applied a log10 transformation to the dependent variable. I am having trouble with manually calculating the R2 for both train and test dataset. The model looks like this.
Model <- lm(log10(Total_LT) ~ ThreeComb + Ship_Qtr, data = Train_Data)

Additionally here is the summary of the model.
Residuals:
 Min       1Q   Median       3Q      Max
-0.47904 -0.09681 -0.00449  0.09272  0.63265

Coefficients:
                                Estimate Std. Error t value Pr(>|t|)
(Intercept)                         1.178008   0.007786 151.302  < 2e-16 ***

ThreeCombAIR Site A Product C       0.221098   0.042209   5.238 1.85e-07 ***

ThreeCombAIR Site B Product B       0.467222   0.050400   9.270  < 2e-16 ***

ThreeCombAIR Site C Product B      -0.020639   0.013471  -1.532 0.125716

ThreeCombFASTBOAT Site A Product A  0.357324   0.015775  22.652  < 2e-16 ***

ThreeCombFASTBOAT Site A Product C  0.397101   0.015291  25.970  < 2e-16 ***

ThreeCombGROUND Site D Product B   -0.084635   0.010842  -7.806 1.08e-14 ***

ThreeCombOCEAN Site A Product A     0.470911   0.014879  31.648  < 2e-16 ***

ThreeCombOCEAN Site A Product B     0.582689   0.025467  22.880  < 2e-16 ***

ThreeCombOCEAN Site A Product C     0.474703   0.061184   7.759 1.56e-14 ***

ThreeCombOCEAN Site B Product B     0.414655   0.016140  25.691  < 2e-16 ***

Ship_QtrQ2                         -0.039806   0.009264  -4.297 1.84e-05 ***

Ship_QtrQ4                         -0.040277   0.012147  -3.316 0.000935 ***

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1489 on 1535 degrees of freedom

Multiple R-squared:  0.6803,    Adjusted R-squared:  0.6778

F-statistic: 272.2 on 12 and 1535 DF,  p-value: < 2.2e-16

Now I am trying to test this model my calculating Rsquared manually like this.
Train_Data$Residual <- Model$residuals
Test_R2 <- 1 - (sum((Test_Data$Residual)^2)/ sum((Test_Data$Total_LT - mean(Test_Data$Total_LT))^2))

Here is the output that I get when for my R2
[1] 0.9999015

To validate my model, I also did this to calculate my R2 for test dataset.
Test_Data$Predicted <- predict(Model, newdata = Test_Data)
Test_Data$Residual <- Test_Data$Total_LT - Test_Data$Predicted
Test_R2 <- 1 - (sum((Test_Data$Residual)^2)/ sum((Test_Data$Total_LT - mean(Test_Data$Total_LT))^2))

And I get this R2 for test dataset.
[1] -1.964802

I am thinking this was caused by log10 transformation that I applied to my model. What can I do to make my R2 for both training and test close to 0.68 like it actually says on the summary of the model?
By the way, I tried the same thing without log10 transformation, and got a very good R2.


texts: Why encode pitch as one-hot encoding instead of ordinal encoder?

texts:
looking at the state-of-the-art publications on deep learning for synthesizing audio one can see that they always resort to encoding pitch as a one-hot vector. I'm curious what the advantage is on doing this, since pitch (frequency) fits naturally as a scalar value and one could reduce network dimensionality by encoding it as such.

The papers I've studied are:

GANSynth: Adversarial Neural Audio Synthesis

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders


texts: Making outputs of sub-models the inputs of other models LSTM

texts:
I'm trying to create a neural network that is composed of different sub-models. Each sub-model have their own inputs, which some of them can be shared between the sub-models.

For example, sub-model 1 could be something like this:

S1 = Input(shape=(1,50),name="Stats")
S2 = LSTM(300,return_sequences = True)(S1)
S3 = Dropout(0.5)(S2)
S4 = LSTM(200,return_sequences = False)(S3)
S5 = Dropout(0.5)(S4)
S6 = Dense(100,kernel_initializer='uniform',activation='relu')(S5)
S7 = Dense(1,kernel_initializer='uniform',activation='relu')(S6)

And then, the output of this sub-model I want it to go as an input of a hidden layer. The thing is that the hidden layer I'm trying to do is also with LSTM and I read in some forums that using keras concatenate can do this.

So, it would go like this:

merge = concatenate([S7,A7,B7,...],name="conca")

At last, I tried to gave this merge object to a new model that would be treated as a "hidden layer".

It would go like this:

hidden_1_1 = Input(shape=(1,6),name="Hidden_Layer_1")(merge)
hidden_1_2 = LSTM(300,return_sequences = True)(hidden_1_1)
hidden_1_3 = Dropout(0.5)(hidden_1_2)
hidden_1_4 = LSTM(200,return_sequences = False)(hidden_1_3)
hidden_1_5 = Dropout(0.5)(hidden_1_4)
hidden_1_6 = Dense(100,kernel_initializer='uniform',activation='relu')(hidden_1_5)
hidden_1_7 = Dense(1,kernel_initializer='uniform',activation='relu')(hidden_1_6)

And I'm getting this error: TypeError: 'Tensor' object is not callable.

So...how can I make that the outputs of the sub-models to be the inputs of the other models?

In an image, what I'm trying to do is something like this, but the thing is that instead of a neuron, each neuron would be a model.

Thank you in advance!


texts:
hope to find you well ! I am trying to build a model to classiffy customers with propensity to buy, but i cannot get rid of overfitting! My approach is the following: I have created the train dataset with unbalanced approach and have now a target 1 of 6% and a total of 6.755 rows and 252 columns. On the other hand, the test dataset has 313.587 rows and target 1 is only 34 of the cases (really low %). The test set was constructed to reflect the reality : the universe to score is all customers from one month, so for the test set I also chose all customers from one month and actually this is a product with low expression... I used autoML code from H2o in R and I am getting very different AUC between train (0.9) and test (0.6). The code i used is the one below:
aml <- h2o.automl(x = predictors, y = response,
                  training_frame = train_h2o,
          validation_frame=test_h20,
              nfolds=0,
                  max_models = 10,
                  seed = 3)

Any advice on what I can do to prevent from overfitting?


texts: What other approaches are there for abstractive summarization, other then seq2seq?

texts:
I'm researching on abstractive text summarization, and has come across many recent papers. They all seem to be focusing on Sequence to Sequence models based on RNNs. Apart from RNNs, what other approaches are there when it comes to abstractive text summarization?
Does ontology-based summarization revolve around the same seq2seq model?

Most of the material I have come across are research papers on this subject; what better sources are there to get an understanding of the underlining concepts of abstractive summarization?


texts: What approach to use to detect violations of media ethics in news?

texts: I've been trying to come up with a solution to detect violations of media ethics in news articles. For example, I need to detect if an article has mentioned the name of a victim of rape. So far, I've tried to perform a binary classification using a Multinomial Naive Bayes classifier with sklearn. However, it does not produce very accurate results and I believe this is because there are no clear-cut features to be used for classifying. I really appreciate if someone can point me in the right direction or give me some tips on how to proceed.


texts: How is a single element of the training set called?

texts:
This question is only about the vocabulary.

Do / can you say

data item
data sample
recording
sample
data point
something else

when you talk about elements of the training / test set? For example:

  The figure shows 100 data items of the training set.

  Database A contains the same data items as database B, but in another format.

  The remaining data items were removed from the dataset.

  Those 10 classes have 123456 data items.

Please provide papers with examples.

According to Google n-grams:


texts: Segmentation-free character recognition on an image: Multi-label, multi-class or sequential image classification problem?

texts:
I have some images which look like this one:

They exist of 3 possible characters (A-C) and a length of 4.

Now, I would like to run a neural network, which recognizes each character in the picture without any segmentation before that; I would like to use a similar approach as the authors in this paper did: https://medusa.fit.vutbr.cz/traffic/research-topics/general-traffic-analysis/holistic-recognition-of-low-quality-license-plates-by-cnn-using-track-annotated-data-iwt4s-avss-2017/.

Now, I am wondering which kind of classification problem I am facing and how to solve that?

Is this a multi-label classification problem? Or a multi-class problem? Or is it a sequential data labeling problem (I assume this)? Or something completely different?

To differentiate the different problems:
Multi-label classification would mean:
First class is 'AAAA', second class 'AAAB', ... and the last class 'CCCC'. That means, we would have 3x3x3x3 = 81 possible classes. So, we would have one column for each label which contains a string label.

Multi-label classification would mean:
The labels are split in parts for every character. First position: A-D. Second: A-D, Third: A-D, Fourth: A-D. So, we would have 4 columns for each label with the appropriate character.

I think, both are not the approach the authors used. If I understood correctly the paper I linked above, they handled the data as sequential data, and I do not have to split the labels but I can use the whole label. Futhermore, the convolutional neural network is able to learn the spatial positions automatically. Because of that, I do not have to split the labels but can them use as they are. If this is right, then I think the target shape of my output should be something like (n, 4, 3), with n as the number of images, 4 as the length of the text labels and 3 as the possibilities for each character.

Is this correct?


texts: Why is pre and post silence important when collecting speech data?

texts: I've been wondering why it's important, when collecting speech data for AI machine learning, to collect pre and post silence for speech data? Is it just so the machine learns and understands the difference between "silence", "background noise", and actual "speech"?


texts: aggregating multi output regression outputs vs single output (=top down) approach: is it worth it?

texts:
basically I have this problem where I need to forecast the sales of some stores. such stores have multiple product lines of which I have the split data (say Y1,Y2,Y3 where Y1+Y2+Y3=Y). I have also some X that I will use to estimate Y (the stores aggregate sales). tabular data, no time series. Loss function is RMSE. I have 2 approaches possible:

simply use aggregated sales to train -> single-output problem
forecast all the product lines at the same time (in python this is easily doable via deep learning libraries) and then aggregate the split sales into the aggregated sales -> multi-output problem

in theory the second approach should be superior as it leverages a greater amount of information and therefore should be able to forecast more effectively the final result reducing the unexplained variance while the first problem should be way easier in terms of convergence (loss function) and should be orders of magnitudes faster to train.
In practice (my trials so far), the NN is taking forever to train and, as I am forced to increase the ephocs, I am a bit scared overfitting will be there.
given the information available, do you still think it's just a matter of tuning the NN for the multi-output thing or am I missing something major for which apparently the simpler approach works better ?


texts:
I have been looking for a good annotation tool/interface for text annotation. My requirements are as follows:

Display of multiple short-text social media posts  on the same page, as one task;
Annotators are asked to label one short post as good or bad;
Then annotators are asked to label the entire displayed posts as good, ok or bad;

I have tried Doccano and checked out a few other open-sourced interfaces. It seems rather difficult to satisfy the above requirements, and I feel I have to hack Doccano quite a bit to achieve this. Does anyone have any advices please? Thanks!


texts: Multiple choice gap-fill question (with distractors) dataset for evaluating NLP algorithms

texts:
I am looking for a standard gap-filling multiple-choice exercise (with distractors) dataset that can be used to evaluate the NLP gap-filling ML algorithms. I expect the dataset to contain questions that evaluate the NLP aspect of the algorithm and can be answered even by kids that require no special prior technical knowledge. Simple questions like:

I eat an ----- every day.

book
salary
apple
car

I ----- to school yesterday.

went
go
apple
student

Any recommendation?


texts: General equation for getting an idea of the scale of a machine learning project

texts:
I'm writing an application for a project where we intend to teach a model to predict one aspect of an environment (traffic safety) using a database with 10 images (about 300x300px and, say, 256 colors) for each of either 100 000 or 15 million locations.

I must come to grasp with if both, one or none of these projects are feasible with our hardware constraints. What can I expect? Is there some formula or benchmark that one can refer to? Will one be able to do this on a laptop with a decent GPU, a dedicated ML computer or does it require the level of infrastructure that Google and Amazon use?


texts: clustering multivariate time-series datasets

texts:
I am new to clustering.i have data from quality testing of an automobile manufacturing company.

I have 100000 datasets.each dataset has 4 variables force, voltage, current, distance. each variable is a continuous time-series with 8000 data points each(1 to 17000 milliseconds). the length of time series differs from on dataset to another. all variables in one dataset has to be compared with another dataset

I have to find clusters in the 100000 datasets based on similarities in shape of each variable in a dataset.

which type of measuring best suits, in this case, to find similarity in shape of time-series


texts: Predicting credit applications with timeseries

texts:
I was wondering what the best way is to make a model for predicting credit applications.

I have two tables, which look like this:

   client_nr  yearmonth  total_nr_trx  nr_debit_trx  volume_debit_trx  ... etc.
1          1     201201            94            49           6527529
2          1     201202            85            58           3475518
3          1     201203            94            61          31317405
4          1     201204            85            52          18869967
5          1     201205            93            53           2893105

  client_nr  yearmonth  credit_application  nr_credit_applications
1          1     201201                   0                       0
2          1     201202                   0                       0
3          1     201203                   0                       0
4          1     201204                   1                       1
5          1     201205                   0                       1

The goal is to determine which clients are likely to apply for a credit. So far, I have made sequences of size (months, features) for each client,
using the first table.

My questions:

Would it now be a good idea to create train/test folds based on theclient_nr? Or should I make splits by month?
Should I then select the first $n$ months as features, and create one label from $k$ months after $n$ that indicates whether the client has applied for a credit in those $k$ months? Or is there a better way?
Would it be better to use regression using nr_credit_applications, or classification on credit_application?


texts: Big Data and a fwf files reading with filters

texts:
I have searched at all possible areas and haven't been able to find the solution to my problem. I am hoping that someone here would be able to help me out. If you feel that this question is already answered, I would appreciate the link.

I am using R3.2.2 64 bit on a windows 7 machine. I have a fixed width file of size 1.9 GB and has approx 18 million records in it. I need to read the file based on a filter condition RecordID = "Cast". All my attempts to read the file.dat has failed due to memory constraints. Any help on how to use package ff for this purpose?

I am open to other methods also.


texts: Making a tagged Part-of-Speech corpus with the help of a lexicon

texts:
I have a part-of-speech lexicon that has two columns, words and part-of-speech tags, inside a Pandas's Dataframe.

Also, I have a list of tokens (words) in another Dataframe.

I want to take each token in the untagged corpus and search it inside the entire lexicon. If a token is matched in the lexicon, then take that token's tag and add it to another column in the untagged Dataframe. If the token is not found then return 'X'.
Here how I did it:
lexicon_rows = lexicon.iloc[:,:].values

def add_tag(untagged_row):
    tag = lexicon_rows[:, [1]][lexicon_rows[:, 0] == untagged_row['word']]
    if tag.size == 1:
        return str(tag[0][0])
    else:
        return 'X'

untagged['tag'] = untagged.apply(add_tag, axis=1)

I am not sure whether each word in the untagged corpus is searched against the entire lexicon or not.
My question is: Am I doing it right? If so, what is a better approach to accomplish this task? If not, could you please provide me with an answer?
Thank you.


texts: Why use embeddings when data about the categories are abundant?

texts:
Suppose I'm making a movie recommendation engine and I have tons of data about the movies like genre, ratings, release info, earnings, etc. It makes sense to create an embedding layer for the users since I have no additional info on them and the embeddings will learn the geometry of the users based on their watch preferences. My gut still tells me to make an embedding layer for the movie index as well and concatenate it with the movie-related numerical features. Why though?
Can't we just pass in the numerical features of the movies for the model to differentiate? Are the embeddings supposed to capture some additional latent information perhaps not captured in the raw metadata?
I know the embeddings get us the nice feature of being able to compare similar movies but I could just do the same this with my numerical features right?
This isn't specific to recommendations but generally, why would you create an embedding vector for categorical features when you already have a lot of descriptive data about each item in that categorical feature?


texts: Calculating cosine similarity between 3D arrays using Python

texts:
I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value?
The matrices are the embeddings of two words each, obtained from BERT.


texts:
I need to benchmark a NER tool -recognising dates, places and organisation- ideally using chatbot data, or conversational data. Would you know a dataset, with an existing baseline, to perform such task?
Thanks for your help!


texts: Why Maximum Likelihood Estimation for normal distribution?

texts: Since we can compute the mean and the standard deviation of a set of random variables, why do we use  Maximum Likelihood Estimation to estimate these parameters?


texts: Reduction of multiple answers to single variable

texts:
The questionnaire for the data is here

The first question takes multiple entry for the same question, I want to reduce this to a single variable. How do I do it?

The clean data is available here.
NB: The Column CompuPlat has missing values.

part of dataset

CMFam CMHobb  CMNone  CMOther CMPol   CMProf  CMRel
0   0   1   0   0   0   0
0   0   0   0   0   0   0
1   1   0   0   0   1   0
0   0   0   1   0   0   0
0   0   0   0   1   1   0
1   0   0   0   0   1   1
Community Membership_Family
Community Membership_Hobbies
Community Membership_None
Community Membership_Other
Community Membership_Political
Community Membership_Professional
Community Membership_Religious
Community Membership_Support

I want to club all of them in a variable CM


texts: How to build a recurrent neural net in Keras where each input goes through a layer first?

texts:
I'm trying to build an neural net in Keras that would look like this:

Where $x_1$, $x_2$, ... are input vectors that undergo the same transformation $f$. $f$ is itself a layer whose parameters must be learned. The sequence length $n$ is variable across instances.

I'm having trouble understanding two things here:

What should the input look like?
I'm thinking of a 2D tensor with shape (number_of_x_inputs, x_dimension), where x_dimension is the length of a single vector $x$. Can such 2D tensor have a variable shape? I know tensors can have variable shapes for batch processing, but I don't know if that helps me here.
How do I pass each input vector through the same transformation before feeding it to the RNN layer?
Is there a way to sort of extend for example a GRU so that an $f$ layer is added before going through the actual GRU cell?


texts: Support Vector Machine (SVM) for classification problem based on Earth Mover's Distance (EMD)

texts:
I would like to run SVM for my classification problem using the Earth Mover's Distance (EMD) as a distance measurement. As I understood the documentation for Python scikit-learn (https://scikit-learn.org/stable/modules/svm.html#svm-kernels) it is possible to use custom kernel functions:

import numpy as np
from sklearn import svm

def my_kernel(X, Y):
    return np.dot(X, Y.T)

clf = svm.SVC(kernel=my_kernel)

Also there is a package with EMD implemented (https://pypi.org/project/pyemd/). I tried to run it similar as in example using my own data (below). I have distributions of eigenvalues. But there is an error and can't figure out what does it mean.

from pyemd import emd
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

# Construct train and test data
X = np.concatenate((eigs_ctrl_fmri, eigs_pat_fmri))
y = np.concatenate(([0] * eigs_ctrl_fmri.shape[0], [1] * eigs_pat_fmri.shape[0]))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42) # 70% training and 30% test

model_svm = SVC(kernel=emd)
model_svm.fit(X_train, y_train)

TypeError: emd() takes at least 3 positional arguments (2 given)

So, it means that for emd function we need one more argument. Example in documentation says that this is distance matrix:

from pyemd import emd
import numpy as np
first_histogram = np.array([0.0, 1.0])
second_histogram = np.array([5.0, 3.0])
distance_matrix = np.array([[0.0, 0.5],
                             [0.5, 0.0]])
emd(first_histogram, second_histogram, distance_matrix)

3.5

But how I should construct distance matrix for SVM model - in advance calculate the distance between all samples in my data? If it is, I don't really understand how I should provide an input to SVM model.

And one thing about EMD - why do I need the distance matrix to calculate the earth mover's distance? For me it looks strange that I need to calculate the distance and provide it as an input to the function which calculating the distance. I undertstand that I missing something in basic understanding the EMD principle and how to use it for SVM. Can somebody help me please with that?

Thank you for any info to boost my understanding!


texts: Explaining feature_importances_ in Scikit Learn RandomForestRegressor

texts:
For a project, I used the feature_importances_ attributes from the RandomForestRegressor. Everything works well but I don't know how to explain why one feature is more important than another. I mean I know that the higher the score  is the higher the importance, but I don't understand how it is calculated.

For exemple, if a variable as a score of 0.35 what does it mean?

I would appreciate if someone could explain me how it works!

Thanks!


texts: Where can I find datasets with large number of features (100000) but less samples(100-1000)?

texts:
I am working on a dimensionality reduction algorithm for datasets with high feature to sample ratio. I need datasets with around 100000 features or more and 100-1000 samples with class labels to evaluate my algorithm's performance.

Moreover the features should be real valued and not binary like in the Dorothea dataset.


texts: How to use Variational Autoencoder's μ and σ with user-generated z?

texts:
My understanding of VAE is that unlike Autoencoders, it does not directly give you a discrete encoding (latent code vectors n-dim) instead, it gives you both mu and sigma (n- dim mean vectors and n-dim standard deviation vectors). Then you have epsilon which you use to sample from a normal distribution with mu and sigma to create z. When combining mu, sigma and epsilon, you get z which is the one decoded by the VAE's decoder. z is basically the main encoding.

Say my z, mu, sigma are of n-dimension like 10 (10-dim z, mu, sigma). I enable the user to have a free picking/giving me numbers 10 vectors [-a, a], say a = 5. So the user is free to pick 10 vectors between -5, 5.

This becomes my z that is decoded by my decoder to generate a new image.

[Main problem]

My VAE is trained on a dataset of apparel. Now, if I run my VAE's encoder on each of the data in the dataset, I'd get a mu and sigma for each (not sure if this is still correct).

Using the z given by the user, how do I find the most similar from the dataset using VAE's encoding of only mu and sigma?

My thinking is to generate z using mu and sigma generated by VAE'S encoder but in order to generate z, I still need to sample from a distribution using epsilon in which makes it non-discrete w.r.t user-generated z. This adds randomness to it so I am not sure how would I use the encoded z to find the nearest to user-generated z.


texts: What is the difference between C and lambda in the context of an SVM?

texts:
I don't understand the difference between the parameter $C$ and $\lambda$ in terms of the SVM. It seems to me that they are both involved in regulating over-fitting of the data.

What difference between $C$ and $\lambda$?


texts:
I had the following data-cleaning question in an interview test that I struggled on (I've changed the details to anonymise it and protect the company's interview process)

  Given the following dataframe df, return a new series with day as the index, and a single column with the set of meals consumed by everyone who ate that day (i.e, both Alice and Bob on days 1 and 3, but only Alice on day 2). Do not use for loops or list comprehensions, only method chaining and a single lambda function that accepts only a single argument.

df = pd.DataFrame({'day':[1, 2, 3, 1, 3]*3,
                   'person':['Alice', 'Alice', 'Alice', 'Bob', 'Bob']*3,
                   'meal':['breakfast', 'breakfast', 'breakfast', 'breakfast', 'breakfast']+
                          ['lunch', 'brunch', 'brunch', 'lunch', 'lunch']+
                          ['dessert', 'dinner', 'snack', 'beer', 'dessert']
                  })

In other words, the goal is to obtain the following dataframe:

goal = pd.DataFrame({'day':[1, 2, 3],
                     'meal':[{'breakfast', 'lunch'},
                             {'breakfast', 'brunch', 'dinner'},
                             {'breakfast'}]
                    }).set_index('day')

Does anyone know how to do this? Thanks!


texts: Why does Siamese neural networks use tied weights and how do they work?

texts:
Reading this paper on one-shot learning "Siamese Neural Networks for One-shot Image Recognition" I was introduced to the idea of Siamese Neural Networks.

What I did not fully grasp was what they meant by this line:

  This objective is combined with standard
  backpropagation algorithm, where the gradient is additive
  across the twin networks due to the tied weights.

Firstly, how exactly are they tied? Bellow, I believe I've provided the formula by which they compute the gradient. T is the epoch, $\mu_j$ is the momentum, $\lambda_j$ the regularization, $\eta_j$ the learning rate, $w_{kj}$ I believe to be the weight between neuron k and in one layer and j in another but correct me if I'm wrong.

\begin{equation}\begin{array}{c}
\mathbf{w}_{k j}^{(T)}\left(x_{1}^{(i)}, x_{2}^{(i)}\right)=\mathbf{w}_{k j}^{(T)}+\Delta \mathbf{w}_{k j}^{(T)}\left(x_{1}^{(i)}, x_{2}^{(i)}\right)+2 \lambda_{j}\left|\mathbf{w}_{k j}\right| \\
\Delta \mathbf{w}_{k j}^{(T)}\left(x_{1}^{(i)}, x_{2}^{(i)}\right)=-\eta_{j} \nabla w_{k j}^{(T)}+\mu_{j} \Delta \mathbf{w}_{k j}^{(T-1)}
\end{array}\end{equation}

My other question is why this is even desirable?
Why not just reuse the same network twice?
Or perhaps the two networks will be identical after training?
If the networks are identical after training, why would you set it up like this?
What benefits does it have?


texts: Applying a robust scaler to column 0 to 6 of my data frame?

texts:
I have a dataframe X and would like to scale columns 0 to 6 using robust scaler, how do i achieve this please? the following code only returns the scaled section of the data frame without the remaining columns, but i want the whole columns returned with 0-6 scaled only.
from sklearn.preprocessing import RobustScaler

scaler=RobustScaler()
X=pd.DataFrame(scaler.fit_transform(X),columns([['Administrative',
'Informational','ProductRelated','BounceRates','PageValues','SpecialDay']]))

X


texts: How can i tell if my model is overfitting from the distribution of predicted probabilities?

texts:
all,
i am training light gradient boosting and have used all of the necessary parameters to help in over fitting.i plot the predicted probabilities (i..e probabililty has cancer) distribution from the model (after calibrating using calibrated classifier) i.e. their histogram or kde. as you can see from below the probabilities for my class 1 are concentrated on the upper and lower end.
i have tried playing around with bandwith too to smooth this a little and it doesn't smooth the bumps too much. what do you think this shows about my model? isn't it a good thing that the model for class 1 (which is has cancer) is assigning a greater probability for this class?
i am unsure how to interpret this or where i could be going wrong

the red curve is positive class (has cancer) and the blue curve is hasn't. below is plot used to generate.
results = df[['label','predicted_prob']]

colors = ['b', 'r']

for label in [0, 1]:
    results[results['label'] == label]['predicted_prob'].plot.kde(bw_method=0.35,color=colors[label])
plt.xlim(0,1)


texts: Spyder 4: changed behavior or "run cell" / run selected code

texts:
I'm a user of spyder. This weekend I updated to spyder 4, which seems to have received many usefull improvements, however I have a problem with running selected code. The logic seems to have changed.

Unfortunately for me it is very important, that I can select code lines and run them ad hoc, without copying them to the shell each time. For older versions of spyder this could be easily done by selecting the code and pressing [Ctrl]+[Enter], but now this seems to execute not just the code I selected. To be honest, I don't even know, what it executes.

The same applies if I select "run cell" from the menu after selecting code.

Can anybody shed some light into this? how can I execute selected code in spyder 4?

For me this functionality is so important, that I really think about downgrading to a lower version, even though I would lose big improvements in the editor.


texts: Default value of learning rate in adam optimizer - Keras

texts:
I am working on a image classification problem using Transfer Learning.

Parameters used given below:
Adam optimizer with learning rate - 0.0001

adamOpti = Adam(lr = 0.0001)
model.compile(optimizer = adamOpti, loss = "categorical_crossentropy, metrics = ["accuracy"])

For testing I used adam optimizer without explicitly specifying any parameter (default value lr = 0.001).

With the default value of learning rate the accuracy of training and validation got stuck at around 50%.
And when I use learning rate = 0.0001 in the first epoch itself I could see that the accuracy is going to 90%.

Could you please help me understand
1. why with lower value of learning rate the accuracy is increasing rapidly?
2. Also which of the above used learning rate is the better?
3. How could I make use of decaying learning rate in Keras?

Thank you


texts:
I created a deep learning (CNN) model, I used data augmentation and two dropout layers (0.5).

How to know if the deep learning model is overfitting or not?


texts:
R supports a wide range of OOP designs like s3,s4,RC and others via packages,and it's a bit overwhelming to decide on which to use and a more basic question that I have is when and where do you use OOP  while doing  machine learning or data analytics ,can someone answer this from a data scientist/ML practitioner point of view .

I'm aware of how OOP works in R at a superficial level ,but

  should I invest time in learning OOP in R ?
  how practical is it from a data science point of view  ?


texts:
I have to calculate precision and recall for a university project to measure the quality of the classification output (with sklearn).
Say this would be my results:

y_true = [0, 1, 2, 1, 1]
y_pred = [0, 2, 1, 2, 1]

confusion matrix:
[1 0 0]
[0 1 2]
[0 1 0]

I have read about it and the definition makes sense for me in a binary setting but with 3 labels I find it hard to interpret precision/recall.
If I use sklearn.metrics.precision/recall_score it gives me 0.4 for both (average = micro)

Now for the precision this makes somewhat sense because 2 out of 5 are correctly classified. But I am having problems interpreting the 0.4 result for recall.


texts: How many times is backprop used in epoch?

texts:
As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found.

So one epoch means that the forward-backprop (and updating weights with gradient descent) is done only once and in order to find the minimum we need to run the algorithm more times?

Or is forward-backprop combination done multiple times (like for every input of data that is passed to the input layer) in one epoch until the minimum for that epoch is found?


texts: Why does Naive Forecast,Moving average,Simple Exponential Smothing model predictions flatten?

texts:
I have applied simple forecasting models such as Naive Forecast, Moving Average, Simple Exponential Smoothing, Holts Linear Trend Model on 2018 sales data of a salesperson.

All the model resulted in flatten or prediction line flattens at zero. Could be it be an issue with data? as most of the data is flatten at zero.

model = ARIMA(train_log, order=(0, 1, 2))
output = model.fit(disp=-1)

#convert fitted values in to series
output_series=pd.Series(output.fittedvalues, copy=True)
print(output_series.head())

#Calc Cumm sum
output_series_cumsum= output_series.cumsum()
print(output_series.head())

#Add cumsum values
output_tr_log=pd.Series(train_log.ix[0],index=train_log.index)
output_tr_log=output_tr_log.add(output_series_cumsum,fill_value=0)
output_tr_log.head()

#convert to predicted ARIMA vlaues to original format
convert_output = np.exp(output_tr_log)
plt.title('RMSE: %.4f'% (np.sqrt(np.dot(convert_output, train_log))/len(train_log))

Date       Sales
----       -----
2018-01-27  1
2018-01-30  60
2018-01-31  22
2018-02-01  490
2018-02-04  53
2018-02-05  30
2018-02-06  204
2018-02-07  234
2018-02-08  64
2018-02-10  70
2018-02-11  81
2018-02-12  10
2018-06-01  40
2018-06-02  669
2018-06-06  1188
2018-06-07  1250
2018-06-10  3861
2018-06-14  40
2018-06-21  44


texts: Splitting train and test in multilabel classification to avoid missing data in the train set

texts:
I have a dataset (600 rows) composed of two columns:

-Summary: which contains the text of a document

-Keywords: which contains the keywords that identify that document.

                                              Summary                     KeyWords_in_Array_wo_insurance
0    court sanction scheme transfer insur reinsur b...                                       [insolvency]
1    immedi custodi sentenc month week impos direct...  [administration of justice, civil evidence, se...
2    motorist injur hit run collis car identifi dri...                         [negligence, road traffic]
3    claimant given permiss continu claim compani a...                      [insolvency, civil procedure]
4    court gave guidanc approach taken applic relea...                           [civil procedure, costs]
5    plaintiff solicitor entitl declar life critic ...                                           [trusts]
6    claimant insur establish requir standard road ...                           [personal injury, torts]
7    minimum indemn requir institut charter account...  [arbitration, civil procedure, costs, accounta...
8    applic secur cost court could take account eve...          [civil procedure, insolvency, cpr, costs]

I want to classify a summary in specific keywords. The keywords are not mutually exclusive.

My code is:

X_train, X_test, y_train, y_test = train_test_split(df_final["Summary"], df_final["KeyWords_in_Array_wo_insurance"], test_size=0.20, random_state=42)

mlb = MultiLabelBinarizer()
y_train_mlb = mlb.fit_transform(y_train)

classifier = Pipeline([
    ('vectorizer', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf', OneVsRestClassifier(LinearSVC()))])

classifier.fit(X_train, y_train_mlb)
y_predicted = classifier.predict(X_test)
all_labels = mlb.inverse_transform(y_predicted)

y_test_mlb = mlb.transform(y_test)

print("Accuracy = ", accuracy_score(y_test_mlb,y_predicted))

I am getting a low accuracy score: 20%

Therefore I am thinking that my classification is not good enough.

The reason it might be that some keywords are used only once. For example, the keywords "animal" or "partnership" or "succession" are used only in 1 row. (Meaning, they are assigned to only one summary)

I think, therefore, that when I split the dataset in training and test, some "lonely keywords" enter the test dataset but not the training one. Thus, the model will never be trained on them.

Is this the reason why my accuracy is so low?

Or am I doing something else wrong?


texts: How to take advantage of variables whose values are available in the past but not in the future?

texts: Example: weather data. You know the location data, but you don't know the previous days'/weeks' temperatures and other weather conditions. How can you exploit these variables in your past data when you build a predictive model that attempts to forecast pretty far out into the future?


texts: Why this single layer network does'nt work

texts:
I am trying following code (modified from https://www.kdnuggets.com/2017/10/seven-steps-deep-learning-keras.html ):

def single_layer(input_shape, nb_classes):
    print("input shape:", input_shape)
    print("print nb_classes:", nb_classes)

    from keras.models import Sequential
    from keras.layers import Dense, Activation

    model = Sequential()
    model.add(Dense(nb_classes, input_shape=input_shape, activation='softmax'))

    model.compile(optimizer='sgd', loss='categorical_crossentropy')
    model.summary()
    return model

However, when I try to fit this model with an X_train of dimensions 64,64,3 and 17 classes, following is the output with error:

input shape: (64, 64, 3)
print nb_classes: 17
Using TensorFlow backend.
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_1 (Dense)              (None, 64, 64, 17)        68
=================================================================
Total params: 68
Trainable params: 68
Non-trainable params: 0
_________________________________________________________________
Traceback (most recent call last):
....
....
  File "/home/abcd/.local/lib/python3.5/site-packages/keras/engine/training.py", line 950, in fit
    batch_size=batch_size)
  File "/home/abcd/.local/lib/python3.5/site-packages/keras/engine/training.py", line 787, in _standardize_user_data
    exception_prefix='target')
  File "/home/abcd/.local/lib/python3.5/site-packages/keras/engine/training_utils.py", line 127, in standardize_input_data
    'with shape ' + str(data_shape))
ValueError: Error when checking target: expected dense_1 to have 4 dimensions, but got array with shape (10396, 17)

Why this code is not working and how should it be modified to make it work?


texts: Different probability distributions for each number of a MultiDiscrete action space

texts:
I have made a custom gym environment, and I have a question regarding the actions. I use a MultiDiscrete action space, that is, it provides a list of integer numbers.
In my case, the first number defines the type of action to take, while a second number defines other things that are related to that specific action.
For example, imagine a 2D map where an agent walks up, down, left, and right. The first number defines the direction (e.g., up), and the second number defines the number of steps to walk on that direction.
The problem is that I want to sample all possible types of actions (in the above example, the direction) from a uniform distribution, that is, in each step all directions have the same probability to be chosen. Then, for each specific type of action, learn a policy for choosing the best action/second number (in the above example, the number of steps).
More formally, this means that in each state of the MDP, we allow the agent to choose an action only from a subset of all possible actions. This subset of actions is defined by the first number of the MultiDiscrete action space (i.e., the direction).
One solution that I was thinking of is to incorporate the direction of movement into the states. So, in each step we can sample a distribution (in my case I need a uniform distribution), update the state, and then apply the given action from the (smaller) action space which is now defined only by the second number (i.e., in the above example, the number of steps to walk).


texts: Help understanding the Tensorboard histogram names and meaning in an LSTM Model

texts:
Can someone please help me understand what the names and shapes of the following tensorboard histogram outputs mean about an LSTM model I coded? Thank you!

I understand the terms in the names like bias, gradients, LSTM etc. I do not understand what they are saying about the LSTM model with 3 cells. The LSTM model trains really well with no underfitting or overfitting.

Happy to provide any additional information, please ask!

Model Summary:

Layer (type)                 Output Shape              Param #

lstm_2 (LSTM)                (None, 3)                 408

LSTM Model in keras:
model = Sequential()
model.add(LSTM(3, input_shape=(1,30)))
model.add(Dense(1, activation='sigmoid'))

Tensorboard output of histograms:

Any help is much appreciated.


texts:
I am studying for my Machine Learning exam. In sample exam questions, there is a specific one that I could not understand:

And the question is: "What is the error rate for random guessing?"

I could not understand the question, can anyone explain it to me?


texts: Weights shared by different parts of a transformer model

texts: Which parts of a transformer share weights, like, do all the encoders share the same weight or all the decoders share the same weights?


texts: Manual install vs conda install tensorflow-gpu

texts:
Asking mainly for Ubuntu 16.04/18.04.
What's the difference between installing CUDA and cuDNN together with tensorflow-gpu in conda (conda install tensorflow-gpu), and installing it all by hand and then using pip?
Does it mean CUDA and cuDNN are only available in the environment where conda install tensorflow-gpu was invoked?
Also, why do some people use the -c flag when installing it?


texts: Clustering with categorical as well as numerical features

texts:
I have dataset consisting of house prices for example. The dataset contains features such as: house size, monthly rent, house colour, location, year the house was built.
I wanted to group these all attributes into clusters. The problem is how to represent the categorical features, such as colour. And how does the clustering algorithm work with these categorical variables?
Another question is: what happens if let’s say monthly rent 0, how does this affect clustering?


texts: Modeling buying behaviour - where to start?

texts:
I'd like to model purchase decisions but I am at a bit of a loss where to start, as the standard ML textbooks don't seem to describe this specific situation which is admittedly a bit odd and complex.
In this situation, sellers present consumers with menus. Consumers are required to purchase and eat at least 1 item whenever a menu is presented to them. The cost of the menu items are normally distributed, and the number of items on the menu approximate a logarithmic PMF, i.e. ~60% of menus have 1 item, ~20% have 2 items, ~10% have 3 items, and so on. The consumers have an annual minimum quota of purchases they have to make to stay alive, and they are naturally limited by their budget and current appetite. Sellers are naturally limited by wanting to keep their consumers alive and not overwhelming them with food at any one point in time. The data I have shows that consumers usually select 1 item, but it is not uncommon to purchase 2 or more items on a menu if they look particularly delicious.
I want to predict which item(s) the consumer will pick when they are presented with a menu. A multiclass logistic regression to find the most likely mandatory purchase seems obvious. But what about possible 2nd or 3rd purchases from the same menu? Then I consider a separate model that looks at the 2nd item choice as a dependent variable given the first item choice, and so on. However, something tells me that setting up a chain of classifiers feeding classifiers like this is not ideal. Is there a method of considering the whole menu of choices together? I am not aware of what class of problems and related ML algorithms this would fall under.
Can anybody point me in the right direction please?


texts: If you already had a recording of two separate sound sources mixed, where one source was a person singing and the second source was a piano playing. Could ICA be used to separate the recording into 2 distinct sound sources?  If so, how would ICA be used to do this?


texts:
I use spectrogram as input to a Convolutional Neural Network I have created with tensorflow.keras in Python.

Its shape is (time, frequency, 1).

The input's shape of the CNN is (None, time, frequency, n_channels) where n_channels=1 and the first layer is a Conv2D. In between every Convolutional layer I use a BatchNormalization layer before an Activation layer.
The default value for BatchNormalization is "axis=-1".
Should I leave it as it is or should I make it with "axis=2" which corresponds to the "frequency" axis?

The thought behind this is that the features of a spectrogram are represented in the frequency axis.


texts: Suitable modelling approach to incorporate different signals

texts:
I am working with data in the form of content that is being reviewed to decide if it violates the policy standards or not. In the review process there are several methods to increase the chances of making the correct decision. Content might be sent for multi-review in which case it is reviewed additional times to help get to a correct decision. We can also check the historical accuracy scores of the humans reviewing the content, higher accuracy scores indicate a higher likelihood of a correct decision. Similarly, we can check some other factors such as length of review time, the type of the original content etc.
There is a separate audit process which evaluates a subset of decisions and decides on the policy correct decision in that instance. We can compare these policy correct decisions to the original decisions to evaluate the marginal lift in accuracy from the multi-review system, reviewer accuracy etc. The goal is to develop a system which incorporates these signals and outputs a score that represents our confidence that the review decision is correct.
I have done analysis comparing the marginal improvement in accuracy based on the individual signals, I have also calculated the margin of error for these scores depending on the number of samples we have. For example, when content gets multi-reviewed it is x% more like to be the correct decision with a margin of error of 0.5%. Some signals have a higher margin of error since we had less samples.
I'm trying to figure out the best way to incorporate all of the signals into a single score. I am not sure if regular classifiers will apply since only a portion of posts get multi-reviewed and we only have historical reviewer accuracy scores for a portion.
The way I have been thinking about it in my head is in the form of a decision tree, if the content has been multi-reviewed yes/no, if we have an accuracy score yes/no. Perhaps a decision tree / random forest model would be suitable. It would be great if the model was interpretable and the output could be explained. I would also be open to a more straight forward solution without the need for complex modelling.
Thanks for any advice you can provide on this :)


texts: Clustering (unsupervised learning) for uneven classes

texts:
I am looking for an unsupervised method that can see also the points that start to look different from the majority. Which clustering techniques (I use python) can be used for such data sets?

I have tried k-means but as I was expecting it has failed considerably to see such peaks.


texts: Measure correlation for categorical vs continous variable

texts:
Given a variable which is categorical that depends on continuous variables, I would like to know how to check wether these continous variable explain the categorical one.
So:
Y = cagetorical
X1 = continous
X2 = continous
X3 = continous

I'd start with a correlation but which? I've seen How to get correlation between two categorical variable and a categorical variable and continuous variable? but there it is explained wether there is a difference in categorical variables explaining a continous variable, so I think it's another topic?
I'm fine with tool advices in R and python as well.
edit: I'm not sure wether cateogrical is correct here. The values of $ Y $ are $ 0, 1, 2, 3 $ but I could also use $ A, B, C, D $. They represent a classification of the measure of cleanliness of a room.


texts: How to prepare masks for multiclass semantic segmentation?

texts:
It's very straightforward for binary semantic segmentation: black color (0s) is responsible for background, whereas white color (1s) is responsible for objects of interest.

But what about multiclass semantic segmentation? As far as I understand, these masks must be RGB images since we use more than two colors. Is it correct? Or should I have a separate binary mask for every class?

If I can use RGB images with multiple colors as masks, should I use some specific colors for masking? If not, should I specify colors I chose somewhere in a network as class parameters? Or will any CNN automatically detect any number of different colors in my masks?

These questions may seem naive and primitive, but I was unable to find any clear explanantions of thus aspect of multiclass semantic segmentation.


texts: Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?

texts:
I have seen other posts in this forum but didn't find any convincing answer.

Random Forest has an another way of tuning hyperparameter via OOB by design. OOB and CV are not the same as OOB error is calculated based on a portion of trees in Forest rather by full Forest.

So what are the advantages and disadvantages of using OOB instead of a CV?

Is getting to train on more data by using OOB correct to say?