texts: Input explanatory categorical variables along with time series into neural network

texts:
I want an advise on the ways to enter time series along with additional variables into convolutional neural network.

Story first: I have a dataset of time series with daily energy consumption data (measurements every 30 min). Also, it's important to mention that often some days are skipped and sometimes there are two different measurements for one date - it means, that dates are not strictly consequential. Also there's a number of additional explanatory variables for each day, like is it a holiday (0-1), sunrise/sunset time (h), outside temperature (deg, not categorical), etc.

So my dataset looks like the following:

h1 h2 h3 h4 ... h48 v1 v2 v3 v4 v5 v6 .. vn
h1 h2 h3 h4 ... h48 v1 v2 v3 v4 v5 v6 .. vn
...

where h_i are 30-min measurements and v_i are explanatory variables. I am using some moving window with N rows.

One another point of this setup is that I am trying to use convolutional network with sometimes 2D convolutions and I hope to look into the filters later to analyze patterns in energy consumption (including cross-day ones) which might refer to different devices. So no RNN at this stage.

Here comes the problem. Parsing dates as one-hot is impossible in terms of RAM, as it creates a huge sparse table of floats (366 columns for day, 12 for month, 24 for sunrise, etc). So I ended up feeding hourly consumption table into CNN subnetwork separately, and feeding embeddings of explanatory variables into separate Dense network, and then concatenating results of these networks inside my flow. In pseudocode:

variables_to_embed_input = Lambda( lambda z: z[:,:,48:,:])(inputs)
hourly_consumption = Lambda( lambda z: z[:,:,:48,:])(inputs)

x = my_CNN(hourly_consumption)

embedding_outs = my_embeddings_NN(variables_to_embed_input)

additional_vars = Concatenate()(embedding_outs)
additional_vars = Dense(512, activation='linear')(additional_vars)

merged = Concatenate()([x, additional_vars] )
...

The problem is that network where time series and additional variables are fed separately, performs poorly - just a little bit better than only CNN on consumption data without dates and close to simple regression built on daily consumption.

I think that network must somehow "see" that specific row of explanatory variables corresponds to a specific row of energy consumption series, but now they are fed separately and joined only near the top of the network. Is there any better way to input those explanatory data along with time series?

EDIT: I am currently working on feeding variables as channels, but I am uncertain that I will be able to extract meaningful information from filters afterwards, even if it will start working.


texts: Which kind of model is better for keyword-set classification?

texts:
There exists a similar task that is named text classification.

But I want to find a kind of model that the inputs are keyword set. And the keyword set is not from a sentence.

For example:

input ["apple", "pear", "water melon"] --> target class "fruit"
input ["tomato", "potato"] --> target class "vegetable"

Another example:

input ["apple", "Peking", "in summer"]  -->  target class "Chinese fruit"
input ["tomato", "New York", "in winter"]  -->  target class "American vegetable"
input ["apple", "Peking", "in winter"]  -->  target class "Chinese fruit"
input ["tomato", "Peking", "in winter"]  -->  target class "Chinese vegetable"

Thank you.


texts: does xgb multi-class require one-hot encoding?

texts:
I was trying an xgboost from python with a multiclass single-label problem and assumed the label can be an integer indicating my class (as opposed to eg one-hot) .

params = {'eta': 0.1,
#          'objective': 'binary:logistic',
          'objective': 'multi:softmax',
          'scale_pos_weight':9,
           'eval_metric': 'auc',
           'nthread':25,
         'num_class':6}

dtrain = xgb.DMatrix(df_train_x,label= df_train_y)
dvalid = xgb.DMatrix(df_val_x,label= df_val_y)
watchlist = [(dtrain, 'train'), (dvalid, 'valid')]
model = xgb.train(params, dtrain, 500,watchlist, maximize=True, verbose_eval=50,early_stopping_rounds=20)

However I hit an error

(1353150 vs. 225525) label size predict size not match

and I note that my sample size is 225525 , number of classes is 6 , and 6*225525 is 1353150 so it appears that xgb is looking for one-hot ... however when i use one hot I get an error hinting that one-hot can't be used -

dtrain = xgb.DMatrix(df_train_x,label= df_train_y)
ValueError: DataFrame for label cannot have multiple columns

!!!


texts: Classifying Live Images From Camera Feed with bunch of objects in Background?

texts:
I can successfully run image classification if I feed examples from Google image to mobilenet model on Raspberry pi with Google Coral Edge TPU.
However, if I feed live images from camera in my living room, the result is pretty bad. It almost never guess the correct object.
Is it because live image from camera naturally has bunch of objects in the background, and the model is trying to recognize entire picture including background as one object? Is this a reasonable assumption?
If so, do image classification models expect you to separate background/foreground as preprocessing and feed only foreground?
If so, what's a good Python module to separate foreground from background?
Is there a way to make a image classification model to pick out a foreground object from an image with multiple objects in the background and classify only that portion without retraining or transfer learning?
If I feed live images to an object detection model (mobilenet_ssd) based on coco dataset, the result is pretty good. However, it only recognizes 90 classes, and I don't need object location , so it's overkill.


texts: Getting 0 accuracy and NaN mae for all epochs training my NN

texts:
Background:
I made a simple game using python library 'Turtle' in which there is a long plank with a ball balanced on top of it. I can press right or left arrow keys to rotate the plank (either clockwise or anticlockwise) which makes the ball roll to the either side.
Issue:
I wanted to train a neural network to make this whole process automated(to balance the ball on plank). So I myself played the game and recorded the states(conditions) and my corresponding actions(buttons pressed) for a period of 2 minutes in the form of a csv. At the end, I got a dataset of 1300 conditions( eg. velocity of ball, direction of velocity, inclination of plank, etc) and the corresponding 1300 actions i took to balance the ball on the plank. Now when I try to train my neural network on this data, I get 0 accuracy and 'nan' mae for all the epochs. How do I resolve this?
I've attached the code below
#all necessary imports

df=pd.read_csv('train.txt')

model= Sequential([
               Dense(units=16,activation='relu',input_shape=(5,)),
               Dense(units=32,activation='relu'),
               Dense(units=10,activation='relu'),
               Dense(units=1,activation='softmax')
            ])

model.compile(optimizer=Adam(learning_rate=0.0001),loss='sparse_categorical_crossentropy',metrics=['accuracy','mae'])

X_train=df[['x','y','theta','u','v']]
Y_train=df['action']

X_train= X_train.astype('float32')/10
Y_train= Y_train.astype('float32')/10

model.fit(X_train,Y_train,batch_size=40,epochs=20)

And this is what I'm getting for all the epochs :
loss: nan - accuracy: 0.0000e+00 - mae: nan


texts: Processing XML via StreamXmlRecordReader and Pig in parallel

texts:
I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR.

<records>
  <record>
    <id>1</id>
    <name>Jane</name>
  </record>
  <record>
    <id>2</id>
    <name>John</name>
  </record>
</records>

I can read in a small XML and loop with Pig program step in EMR:

DEFINE XPath org.apache.pig.piggybank.evaluation.xml.XPath();

A = LOAD '$INPUT' using org.apache.pig.piggybank.storage.XMLLoader('record') as (x:chararray);
B = FOREACH A GENERATE XPath(x, 'record/id'),XPath(x,'record/name');
dump B;

I can also read in the file, broken into records, using a Streaming program step with -inputreader StreamXmlRecordReader,begin=<record>,end=</record> as an argument.

Currently the Pig script is breaking with the large XML, it works fine on a smaller sample. What I really need is to break this large XML into single records, then parse each record in parallel using a chain of Pig (for the XML/XPath) => further custom steps to parse each subsequent output in parallel, how would I do this? How do you organise AWS EMR to run one step, then split into many steps?


texts: How can I use my laptop to implement analysis in Python but use a different fast server for computation of analysis?

texts:
I am working in a company but I am new in the field. We have a central server which is faster than my laptop (of course). So my goal is that I still use my laptop to do python analysis for machine learning but actual computation is happening in the central server so that the computing power is stronger. I am not considering any cloud service as we have a fast server computer.

But I am clueless in this yet. Can anyone advise me how to do this?

Thank you very much in advance.


texts: What's the best way to predict weekly selling data?

texts:
I am trying to create a model to predict the units that will be sold for different grocery items say in the next week.

I am structuring the problem in a three-step procedure.

Group together the selling data from a macro-category (e.g. if I want to predict "egg box XYZ" I will consider all the items from category "eggs") summing the selling data from all items of that category for each day. Then I use prophet to predict the selling data for next week for the whole category. I am doing this step as the "whole category" data have more history and should be more robust compared to single items data.
Implement a prophet model on the specific items. I expect the error to be higher since the data are less consistent.
Implement a regressor (could be GradientBoost, RandomForest etc.) on the single item using as input variable for a specific day: the previous week selling data for that item, the forecasted sold units for the whole category, the forecasted units (again with prophet) for the specific item, whether we are in a particular time of the year (e.g. boxing day), promotions applied etc. The idea behind this regressor would be to dampen the error of the second step. However I cannot lower the error from the second step.

How could I approach this problem?


texts: When can it be called transfer learning?

texts:
A common definition of transfer learning is:

"Transfer learning is the improvement of learning in a new task
through the transfer of knowledge from a related task that has already
been learned."

— Chapter 11: Transfer Learning, Handbook of Research on Machine Learning Applications, 2009.
This raises the question, when a task can be termed "related".
Let's assume a neural networks is trained to estimate house prices for american houses.
Could it be called transfer learning, if I retrain/finetune the model to estimate european house prices?
Is it still the same task or can it be considered a separate task?


texts: Cosine similarity versus dot product as distance metrics

texts: It looks like the cosine similarity of two features is just their dot product scaled by the product of their magnitudes. When does cosine similarity make a better distance metric than the dot product? I.e. do the dot product and cosine similarity have different strengths or weaknesses in different situations?


texts: How can I split a dataset and use cross validation?

texts:
How can I split my dataset by using Greedy layer-Wise pre-training in Deep Learning Neural Networks?

As an example, if I have a dataset with 60000 samples:

(? % for unsupervised pretraining)
(? % for supervised fine tuning)
(? %validation)
(? % test)

How can I use cross-validation for hyperparameter optimization?


texts: What is the best metric to evaluate highly imbalanaced binary classifiction? (such as fraud detection in credit card)

texts:
What is the best metric to evaluate highly imbalanaced binary classifiction? (such as fraud detection in credit card?

I have examining several metrics precision recall F1 lassification Report (macro avg,weighted avg), ROC, AUC,.. but I do not know what is more acceptable to evaulate highly imbalanced binary classification such as credit card fraud detection https://www.kaggle.com/mlg-ulb/creditcardfraud


texts: How does SHAP values help us to determine importance of a feature for a model trained by gradient boost?

texts:
I've read http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf and https://medium.com/@gabrieltseng/interpreting-complex-models-with-shap-values-1c187db6ec83 which is like a summary of the first link.

In general I didn't understand anything about how does SHAP values helps and how it helps us determine importance of features from the first paper. At the second article he has a very simple decision tree and calculates shapely value for a feature for a specific training example. It doesn't say that at the end which value determines its importance(i.e mean of a feature for every training example. I don't know). Or why this works.

And there is a confusion between them. First the first article uses SHAP values which defined as "Shapley values of a conditional expectation function of the original model" at the second article it just uses shapley values.

I read several academic papers and website articles but I couldn't address to my question. Most of websites deal with its framework application anyways. If you can explain or give a useful resource I would be appreciated.


texts: Should there be 'total derivative' symbol in the mathematical representation of back-propagation algorithm's formula?

texts:
I curiously want to know why in many references [1][2] on back propagation, authors use the partial derivative symbol, i.e $\frac{\partial}{\partial W}$, instead of correctly alternating between it and total derivative symbol, i.e $\frac{\text{d}}{\text{dW}}$?

Doesn't it make the back-propagation formula's derivation context more ambiguous?

[1]: Wikipedia - Back propagation - https://en.wikipedia.org/wiki/Backpropagation

[2]: Stanford Wiki - UFLDL- http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm


texts:
I'm reading the book Fundamentals of Machine Learning for Predictive Data Analytics by Kelleher, et al. I've come across something that I think as an error but I want to check to be sure. When explaining precision and recall the authors write:

  Email classification is a good application scenario in which the different information provided by precision and recall is useful. The precision value tells us how likely it is that a genuine ham email could be marked as spam and, presumably, deleted: 25% (1 − precision). Recall, on the other hand, tells us how likely it is that a spam email will be missed by the system and end up in our inbox: 33.333% (1 − recall).

Precision is defined as: $TP \over {TP + FP}$. Thus:
$$1 - precision = 1 - {TP \over TP+FP} = {FP \over TP + FP} = P(\textrm{prediction incorrect}|\textrm{prediction positive})$$
So this should give us the probability that an email marked as ham (positive prediction) is actually spam. So precision and recall in the quote above should be switched?


texts:
I have been taught to check correlation matrix before going for any algorithm.
I have a few questions around the same:

Pearson Correlation is for numerical variables only.

What if we have to check the correlation between a continuous and categorical variable?
I read some answer where Peter Flom mentioned that there can be scenarios where correlation is not significant but two variables can be multi-collinear?
Removing the variable is the only solution?

I was asked in an interview if we are removing one of the correlated variables, then how this multi-collinearity come?

since pandas.corr() checks correlation between all variables.
How is multi-collinearity is different from correlation?


texts: How to Setup Multivariate Time Series Dataset for Classification

texts:
First post on StackExchange. I’m fairly new to ML, with about 1 year of experience so please pardon any ignorance or misuse of terms.

I have a multivariate time series dataset where I would like to predict the likelihood of an outcome of 1/0 (think of this as a Conversion) in the next 3 time periods for unique ObjectID’s. I have a table that takes snapshots of ObjectID over time, and I want to use current data to predict if the ObjectID will convert (1 or 0) in the coming 3 time periods. Once an ObjectID reaches conversion, it will stay as such.

There’s a twist: some of the X variables are static, they do not change with time. I call these attributes, there are 8 in my dataset. These are essentially characteristics ObjectID in question.
I have 2 variables that change with time, Age (in months) and a categorical variable with 7 levels through which the ObjectID progresses. Here’s how the data looks:

ObjID   Age  Time      Attr1    Attr2   Att3    CurrCat Conversion
id1234  0   1/1/2019    ABC      XYZ    HIJ        A       0
id1234  1   1/2/2019    ABC      XYZ    HIJ        B       0
id1234  2   1/3/2019    ABC      XYZ    HIJ        A       0
id1234  3   1/4/2019    ABC      XYZ    HIJ        D       0 <-- current time
id6789  0   1/1/2019    CBA      ZYX    JIH        C       0
id6789  1   1/2/2019    CBA      ZYX    JIH        C       0
id6789  2   1/3/2019    CBA      ZYX    JIH        D       1
id6789  3   1/4/2019    CBA      ZYX    JIH        A       1

How can I setup this dataset for a classification or decision tree model?

I'll be building the model in Python, so any suggested packages would be helpful too.


texts:
I am developing an application for research purposes that uses randomly generated tweets to detect emerging conceptual topics. I need to collect a large amount of them (let's say, beyond 5 million) and due to the fact that I want the application to detect all kinds of topics, I need to generate a set of the most general or abstract search terms I can find in order to query the stream. The problem is that I can't figure out how or where should I search for these search terms. Let's put it that way: if you had a dataset which consisted of the entire knowledge of the human population and you want to generate a representative subset of it, with which search terms would you query this dataset?
Is it going to be something like this:

Technology,
People,
Politics,
...

Is there a set of the most abstract concepts which I can study?

P.S.: Of course I know that Twitter hasn't got the entire knowledge of humans, but you get the picture ;)


texts: Size of folds in k-fold cross-validation

texts:
When evaluating results using cross-validation, several strategies can be adopted, as using 5 or 10 folds, or doing leave one out cross-validation, as well as doing a 80/20 split.

Under which are general conditions should I try one or another?


texts: ValueError: ('Error when checking model input: expected no data, but got:', array)

texts:
I'm trying to build an inception and resnet model with my own image data. The dataset is 8000 images in total and has 6 labels. Everything goes fine while building the model. But the mentioned error occurs in the model.fit().
I'm really not sure what the problem after spending 14 hours.

I tried the following

Changing the image dimension ordering
Making changes to keras.json
changing the input_tensor shape in the model

Image of the error :

inception_model = InceptionV3(input_tensor = inception_model.input, include_top = True, weights = 'imagenet')
inception_last_layer = inception_model.get_layer('predictions').output
inception_out = Dense(num_classes, activation='softmax', name='output')(inception_last_layer)
custom_inception = Model(inception_model.input, inception_out)

for layer in custom_inception.layers[:-3]:
        layer.trainable = False

custom_inception.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy', 'mse', 'mae', 'mape'])
train_inception = custom_inception.fit(X_train, y_train, batch_size=8, epochs=2)

I'm currently using keras 2.2.0 which I downgraded from latest version after going through some keras issues in github. It did solve some initial hiccups. I'm currently using inception and resnet from their respective python files which I made some changes include_top=include_top to
require_flatten=include_top from this

Here are the inputs shapes
(1690, 220, 220, 1)  is the X_train shape
(1690, 6)  is the y_train
(423, 220, 220, 1)  is the X_test shape
(423, 6)  is the y_test


texts: Normalized 2D tensor values are not in range 0-1

texts:
Below function takes in 2D tensor and normalizes it using broadcasting .The issue is except all values to be in range 0-1 but the result has values outside this range . How to get all values in 2D tensor in range 0-1
def torch_normalize(tensor_list):
    means = tensor_list.mean(dim=1, keepdim=True)
    stds = tensor_list.std(dim=1, keepdim=True)
    normalized_data = (tensor_list - means) / stds
    return normalized_data

INPUT
tensor_list=tensor([[-5.6839, -7.5829, -7.2277, -6.5066, -8.4702, -7.9844, -5.6841,  1.8570,
          1.6170, -3.7592, -4.4140, -0.4981,  0.2501,  5.8463,  1.8897, -1.3968,
         -5.5402, -2.4561, -5.6819]])

Normalized result

tensor([[-0.5981, -1.0615, -0.9748, -0.7988, -1.2780, -1.1594, -0.5981,  1.2420,
          1.1835, -0.1284, -0.2882,  0.6673,  0.8499,  2.2155,  1.2500,  0.4480,
         -0.5630,  0.1896, -0.5976]])


texts: LSTM Sequential Model question re: ValueError: non-broadcastable output operand with shape doesn't match broadcast shape

texts:
This is probably a very simplistic question but I have not been able to find resources that directly address this. I know I must be understanding this incorrectly; I'm not quite sure how.
I've noticed that if the number of units in the last Dense output layer of my LSTM sequential model does not equal the number of features (columns), I get an error.
If you wanted to output 1 feature in the output (Dense) layer, and you had several input features, how would you:

Do that without errors
identify which feature is being outputted, or does Keras provide outputs for each feature and you have to identify which one you want?

I want to train the model with multiple features, but I'm only interested in one feature's prediction.
Example:  I have data with 'open', 'low', 'high' and 'close' and 'volume columns (5 features). If I set the number of units in the last Dense layer to anything other than 5, I get a broadcast error telling me I have inconsistent shapes in the model.  If I put 5 units in the last Dense output layer, I get no errors.
Example:
def create_model(self, epochs, batch_size):

    model = Sequential()

    # Adding the first LSTM layer and some Dropout regularisation
    model.add(LSTM(units=128, return_sequences=True,
                   batch_size=batch_size, input_shape=(TIME_STEP, self.X_train.shape[2])))

    # Adding a second LSTM layer and some Dropout regularisation
    model.add(LSTM(units=128, return_sequences=True))
    model.add(Dropout(DROPOUT))

    # Adding a third LSTM layer and some Dropout regularisation
    model.add(LSTM(units=128, return_sequences=True))
    model.add(Dropout(DROPOUT))

    # Adding a fourth LSTM layer and some Dropout regularisation
    model.add(LSTM(units=128, return_sequences=False))
    model.add(Dropout(DROPOUT))

    # Adding the output layer
    model.add(Dense(units=5))
    model.summary()

    # compile model
    adam = optimizers.Adam(lr=LR)
    model.compile(optimizer=adam, loss='mae')
    model.fit(self.X_train, self.y_train, epochs=EPOCHS, batch_size=BATCH_SIZE)

If I enter 'units=1' in the Dense layer, I get the following error:

ValueError: non-broadcastable output operand with shape (11784,1) doesn't match the broadcast shape (11784,5)


texts: Slowly decreasing validation / training cost and their abnormal values

texts:
I have a dataset of size ~100,000 of images, I'm training a CNN model on them for regression.

optimizer: Adam
batch_size: 64
Number of epochs: 50

When I set the learning rate for Adam optimizer for $1e-2$ and $1e-3$, training and validation costs were normal and soon converged to good and predictable values. But now I've set the lr for $1e-1$ to just test it but training and validation costs are strange values, first $80k$ ! and after many epochs now they are reached to $600$. What's wrong with my hyperparameters, etc?

What about the decay in this situation? what should it be? its current value is $\frac{1e-1}{80}$

I really appreciate any help.


texts: Processed data cannot build Naive Bayes Classifier model

texts: Having preprocessed data using weka supervised cfssubseteval forward and backward selection, the decision tree model builds and makes prediction well, but naive Bayes do not recognize the same Arff file of preprocessed data with 11 nominal attributes and 74 instances


texts: Hi..Can anyone help me resolve the error with following piece of code below?

texts:
from sklearn.svm import SVR

svr = SVR(kernel = "rbf")

svr.fit(X_train,Y_train)

Y_pred = sc_X.inverse_transform(svr.predict(X_test))

pd.DataFrame(Y_pred, columns = ['Price']).to_excel("Final_Pred.xlsx", index = False)

ValueError: operands could not be broadcast together with shapes (2671,) (12,) (2671,)


texts: Is there any working alternatives to Netvizz?

texts: I'd like to download my friends data as a graph/network from Facebook. In Feb 2015, Facebook banned the awesome application Netvizz. Do you know any alternative application for my purpose?


texts: What does pandas describe() percentiles values tell about our data?


texts: Reading accuracy in Keras from the saved model

texts: Yesterday I trained the model in Keras all night, but when I tried to see the results, my computer was not working. When I turned it ON, then I saw 'model.h5' file built already, but I couldn't see the results of accuracy of my training and testing. I saved the model with 'model.save', but how can I now see the accuracy of the model only from that saved model?


texts: Generating Similar Words (or Synonyms) with Word Embeddings (Word2Vec)

texts:
We have a search engine, and when users type in Tacos, we also want to search for similar words, such as Chilis or Burritos.

However, it is also possible that the user search with multiple keywords. Such as Tacos Mexican Restaurants, and we also want to find similar word such as Chilis or Burritos.

What we do is to add all the vectors together for each word. This sometimes works, but with more keywords the vectors tend to be in a place where there are no neighbors.

Is there an approach where we can use not only one word, but multiple word, and still gives us similar results? We are using pre-trained glove vectors from Stanford, would it help if we train on articles that are food related, and specifically use that type of word embeddings for this task?


texts: How to perform polynomial landmark detection with deep learning

texts:
I am trying to build a system to segment vehicles using a deep convolutional neural network. I am familiar with predicting a set amount of points (i.e. ending a neural architecture with a Dense layer with 4 neurons to predict 2 points(x,y) coords for both). However, vehicles come in many different shapes and sizes and one vehicle may require more segmentation points than another. How can I create a neural network that can have different amounts of output values? I imagine I could use a RNN of some sort but would like a little guidance. Thank you

For example, in the following image the two vehicles have a different number of labeled keypoints.


texts: Is it wrong to use residuals from one model as data in another model?

texts:
Why is it wrong to use the residuals from one model to fit another? Is it wrong? Is the difference down to how the residuals are used? For example, the residuals are useful features for extending a model's predictive performance, but fitting a model on residuals from another does not explain casual or statistical relationships within the dataset.
I don't know if this is a debate between the differences between machine learning/data science and statistics. Still, I came across this opinion in both the accompanying lecture (linked timestamp for his comments) and the textbook from Richard McElreath [1] (excerpt below, maintaining the original emphasis and cited reference).

Rethinking: Residuals are parameters, not data. There is a tradition, especially in parts of biology, of using residuals from one model as data in another model. For example, a biologist might regress brain size on body size and then use the brain size residuals as data in another model. This procedure is always a mistake. Residuals are not known. They are parameters, variables with unobserved values. Treating them as known values throws away uncertainty. The right way to adjust for body size is to include it in the same model,[83] preferably a model designed in light of an explicit causal model.

I have seen examples of using residuals from one model to fit another within a machine learning context, such as Regression-Enhanced Random Forests [2]. I think the spirit is generally applied for time series analysis when you detrend the line [3].
References
[1] McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. CRC press. Page 137.
[2] Zhang, H., Nettleton, D., & Zhu, Z. (2019). Regression-enhanced random forests. arXiv preprint arXiv:1904.10416. Link
[3] Modelling currently infected cases of covid-19 using h2o driverless ai. (2020, March 30). Retrieved March 02, 2021, from https://www.h2o.ai/blog/modelling-currently-infected-cases-of-covid-19-using-h2o-driverless-ai/
[83] Freckleton, R. P. (2002). On the misuse of residuals in ecology: regression of residuals vs. multiple regression. Journal of Animal Ecology, 71:542–545. Link


texts: Understanding image size changes in DCGAN

texts:
I have been studying and trying to implement Generative Adversarial Networks using PyTorch. More precisely I tried to replicate the DCGAN PyTorch Tutorial tutorial using some custom dataset.
My code worked well and I was able to get some nice results, but when I look at the architecture of both the generator and the discriminator I struggle to understand how the image sizes change as it goes through the different convolutional layers. To make my question a bit more clear, I do not understand the process to go from the original noise vector of size 100 to the 64X64X3 output image in the below architecture:
class Generator(nn.Module):
    def __init__(self, ngpu):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)

I checked some online resources as well as the PyTorch documentation, and I found some different formulas to calculate the output size of convolutional layers. However none of them were enough to explain the big transformations that occur in this particular architecture.
I hope someone here can help me understand this!


texts: Kaggle Titanic submission score is higher than local accuracy score

texts:
This is the starter challenge, Titanic. The original question I posted on Kaggle is here. However, nobody really gives any insightful advice so I am turning to the powerful Stackoverflow community.

Based on this Notebook, we can download the ground truth for this challenge and get a perfect score.

I tested it and it does give me 100% on LB for the purpose of confirming it is the ground truth as it claims. (side question here: how do I remove this perfect submission because now it shows I have 100% on this challenge but I want to show my real score, which is roughly 80% and I will keep improving)

Sometimes submission on Kaggle takes several minutes to get back the score so I used the ground truth locally to test my different models to save time. However, they always give me different results. See the following:

These are the code I use, what's wrong? You can use my code to try your submission and do you also have the same problem?

def mark(pred):
    solution = os.path.join(dirname, './output/solution.csv')
    submission = os.path.join(dirname, './output/'+pred)
    solution = pd.read_csv(solution)
    submission = pd.read_csv(submission)

    solution.columns = ['PassengerId', 'Sol']
    submission.columns = ['PassengerId', 'Pred']

    df = pd.concat([solution[['Sol']], submission[['Pred']]], axis=1)
    num_row = df.shape[0]
    print(pred[:-4], '==', (df[(df['Sol'] == df['Pred'])]).shape[0] / num_row)

if __name__== "__main__":
    mark('achieve_99_dtree_rfe.csv')
    mark('advanced_feature_with_stacking_5_fold.csv')


texts:
I want to create a ship detection classifier from a dataset that is formed by 4000 photos(3072*2048).

But the dataset  that i currently have is not labeled  so i can feed it to a cnn.So i want to cluster this dataset to 2 labels(or 2 directories) ship and no_ship.I tried running k-means but the results were dissapointing.Is some other more functional  way to do this?


texts: How to summarize multiple time series like dataset

texts:
I am running an experiment where I am sampling CPU utilization every second.

A snippet of the data from each experiment run is something like this:

timestamp   cpu1  cpu2  cpu3  cpu4
1.581015e+09    0.0   0.0   0.0   0.0
1.581015e+09   41.6  57.4  54.5  47.0
1.581015e+09   84.3  72.5  46.6  82.8
1.581015e+09    1.0  83.7   1.0   2.0
1.581015e+09   33.0  43.3  44.4  37.8

I am using pandas to explore the data.

To ensure validity of the experiment, I repeat the experiment 20 times.

Now I have 20 of each of the dataset above.

My question now, is what is the best strategy to:

Summarize the data.
Graph a summary of the data

If I only had one run, I would like to graph the data in a line chat, but now that I have 20, not sure how to proceed.

Find the average at each second across all the 20 dataset? Or is there any recommended approach to handling this?


texts: Confused between optimizer and loss function

texts:
I always thought the SGD was a loss function then I read this on a notebook
model.compile(loss="sparse_categorical_crossentropy", optimizer=keras.optimizers.SGD(lr=1e-3), metrics=["accuracy"])

now I am confused what's the difference between loss and optimizer ? are they both used at the output layer to calculate the loss?
or is the optimizer something used in each layer?


texts: Anomaly detection in multiple parameters

texts: I am a newbie to data science with a typical problem. I have a data set with metric1, metric2 and metric3. All these metrics are interdependent on each other. I want to detect anomalies in metric3. Currently, I am using Nupic from numenta.org for my analysis and it doesn't seem to be effective. Is there any ML library which can detect anomalies in multiple parameters?


texts: How to chose a Machine Learning algorithm?

texts:
I was wondering, are their any guidelines or any rules of the thumb as to which algorithms perform best for each task?

What I'm looking for is something along the lines of:

  NLP tasks are usually accompanied highly-dimensional and sparse data. Algorithm X performs well under these circumstances.

etc.


texts: Which checkpoint file to be used as starting checkpoint for retraining the model in Machine Learning?

texts:
I kept a Tensorflow Object Detection model on training to identify diamonds in an image. While training, checkpoint files are saved in Trainings folder. Checkpoints are saved every 1000 steps. I trained the model for ~2000 Epochs and for deployment I exported to saved_model format which creates a folder called Inference Graph with
-------------TRAINING FOLDER-------------
checkpoint(file)
ckpt-128.index
ckpt-128.data-00000-of-00001

ckpt-127.index
ckpt-127.data-00000-of-00001
...

-------------INFERENCE GRAPH FOLDER-------------

1. checkpoint folder
    - checkpoint
    - ckpt-0.data-00000-of-00001
    - ckpt-0.index

2. saved_model folder
    - assests(folder)
    - variables(folder)
    - saved_model.pb

3. pipeline.config

Now, if I want to retrain for extra 2000 Epochs on top of the previously trained model, which checkpoint file should I used for retraining, the one in Trainings folder (i.e ckpt-128) or the one in the Inference Graph folder (i.e ckpt-0)?
Please help


texts: SMOTE caused my total nrows in train to fall to a very small proportion

texts:
I have a highly skewed dataset with minority class in target being just about 4%. I decided to apply SMOTE using library DMwR in R. Here is my target:

>table(data$target)
 0      1
9908 222661

>prop.table(table(data$target))
     0          1
0.04260241 0.95739759

Next I Split my data into train and test with a 70:30 split and then run SMOTE() on the train set.

> smoted <- DMwR::SMOTE(target~ ., train, perc.over = 100)
>
> nrow(smoted)
[1] 27640
> nrow(train)
[1] 162799

I am unable to understand why the total sample size of smoted data reduced from train. Is there a way to achieve balanced dataset without having to reduce the size of the train set?
Also, I was worried if it will affect my model if I end up training it over such small train set? (I am running a Random Forest and GBM to help classify the target).


texts: What is major difference between different dimensionality reduction algorithms?

texts:
I find many algorithms are used for dimensionality reduction. The more commonly used ones (e.g. on this page ) are:
Principal component analysis (PCA).
Factor Analysis
Independent component analysis (ICA).
Non-negative matrix factorization (NMF or NNMF)
Latent Dirichlet Allocation (LDA)

What are major differences between these? How does one choose one over the other? Thanks for your insight.


texts: Feature extraction from sequence of images with Siamese Neural Network

texts:
I am trying to train a neural network to recognize certain actions in short movies.
Each such movie consists of a fixed number of frames, each frame - the image is of course the same size, after preliminary preprocessing.
And now I'd like to do some feature extraction of each of these images using the Siamese Neural Network (SNN). I found articles somewhere that SNN might be great for this, but without implementation details.
These articles show that they take a pair of two vectors together (in my case it will be a picture, but it does not matter if it is 1d or 2d) and then they use this SNN model which return to them how much the two vectors are similar to each other.
To quote the article:

We employ the contrastive loss function [29] while training our
models. We choose this formulation over a standard classification loss
function like cross entropy since our objective is to differentiate
between two audio frames. Let X = (X1, X2) be the pair of inputs X1
and X2, W be the set of parameters to be learnt and Y be the target
binary label (Y = 0 if they match and 1 if otherwise).

The article itself here: https://www.eecs.qmul.ac.uk/~simond/pub/2020/AgrawalDixon-EUSIPCO2020.pdf
Well, in general, I do not really understand how to feed this SNN with data, since I have these 100 movies, each has 35 frames, and each of these 100 movies is assigned y categories ranging from 0-10 (10 frames per action). And the SNN itself operates on a single frame, how do I train this SNN? I have to choose randomly a pair of two frames, regardless of what action they come from and whether they are from the same or not calculate this vector as close if from the same and far if from another?
Will such a model be useful at all? The first thing that came to my mind was not to compare single frames with each other in this SNN, but whole sequences and on this basis calculate the similarity of the entire sequence to another.
How do you think which approach is better and what exactly should be done
in this approach with comparing single frames to each other?
Thank you in advance
EDIT:
To be precise, I have 2 questions. First, which solution could work better, SNN operating on the entire sequence of 35 frames or maybe SNN operating on a single frame.
Second, for a single-frame approach, how exactly is that model supposed to work. Should these single two frames be chosen randomly? For example, frame 5 of 35 in Action X and frame 25 out of 35 in Action Y, will this even work? Or maybe selecting the i-th frame but from random actions, e.g. 5 frame from action X and 5 frame from action Y.


texts: Comparing accuracy of models in ordinal regression / classification

texts:
I am looking into creating a model to predict whether an item is "Very Good", "Good", "Bad" or "Very Bad".

After I fit the training data to the models, comparing the accuracy of the models during test stump me: should it matter if a model misclassified a G to VG while the other G to VB? What about a model that has two misclassifications of one level away versus another model with only one misclassification but three levels away (eg VG to VB)?

Any guideline on what is the common approach? Also, my thinking at the moment is that this should be a regression problem, but I'm happy to be corrected if I should approach this labeling of datasets more as a classification problem.


texts: How can I get the diameter of each community

texts:
I am trying to calculate the diameter of each community in my dataset, Zachary's karate club using Jupyter. I created a loop to iterate through, but it gives me the diameter of the whole network rather than of each community.

import pandas as pd
data = pd.read_csv('zachary.txt',sep =" ", header = None)
data_values = data.values
g = Graph()
new_data = data_values.tolist()
data_graph = g.Adjacency(new_data, mode = 'undirected')
s = data_graph.community_infomap()
print(s)
s_List = list(s)
print(s_List)
for ic in s_List:
    y = data_graph.diameter(ic)
    print(y)

I expect the output to be like "$1,2,2$" or "$1,3,1$" but the actual output is "$5,5,5$", which is the diameter of the whole community.


texts: how to make a file name as a column in panda?

texts: i have a CSV file which name is data.CSV and i want to make column from file name  for example data is column and value must be data as well  how can i do this?


texts: Why DDPG's policy target is Q-value itself?

texts:
Could someone explain why the target of the DDPG's policy is $Q(s,\mu(s))$?

My understanding of DDPG is like this. Since it is intractable to calculate the $argmax_a Q(s,a)$ in a continuous space, DDPG uses an universal function estimator (Neural Network) to learn and predict the best action that realize $maxQ(s,a)$ output.

So, my question is that what is the actual target when DDPG trains $\mu(s)$? I thought that it should be an actual action that gives the highest Q-value at given state $s$ ($argmax_aQ(s,a)$). However, in OpenAI spinning up, it says it can make approximation that $max_aQ(s,a)\approx Q(s,\mu(s))$, and says the loss of the policy is $E[Q(s, \mu(s))]$.

What does this mean? Are they targeting the Q-value, instead of action-value? How can this policy can learn the best action?


texts:
I have asked this question in Mathematics Stackexchange, thought however that it might be more fit for here:
I am currently taking a Data-Analysis course and I learned about both the terms LDA (Linear Discriminant Analysis) and FDA (Fisher's Discriminant Analysis). I almost have the feeling that they are used as somewhat of synonyms in some places, which obviously is not true.
Can someone explain me how those approaches are related? Since LDA's aim is to reduce dimensionality while preserving information from those dimensions and FDA suggests one approach how to achieve this, can I say Fisher's approach is just somewhat of a subtopic of LDA?
Or as additional question I might add, can FDA and LDA be a synonym, hence be equivalent under any given circumstances?
When would I use LDA, when FDA?
I have already found somewhat of an answer on Wikipedia:

The terms Fisher's linear discriminant and LDA are often used interchangeably, although Fisher's original article actually describes a slightly different discriminant, which does not make some of the assumptions of LDA such as normally distributed classes or equal class covariances.

Could someone still go into a bit more detail, please?


texts: What could be the problem leading to the result that a model can never perfectly overfit?

texts:
I tried to fit my model on a small batch of 128 samples for binary classification. The model should be powerful enough as it has hundreds of thousands of parameters. It should be able to overfit to 100% accuracy. However, it only fts to 96% for the best. It is about the same as when I train it on 30,000 samples. So, I tried the following but all failed:
use a smaller batch of 16 samples, it still cannot overfit
use different optimizers, including Adam, SGD, Adagrad, even reset the optimizer every 1,000 epochs, not working
every epoch, only train on the samples that are misclassified, not working.
The problem should be with this network since another more basic neural network can 100% fit. This one can only 99.2% fit. The top layer is indeed sigmoid.
Anyone got any idea what could be the problem?


texts: NLP : variations of a text without modifying it's meaning

texts:
I am currently working on the automation of recurring reports (weekly 30-50 pages reports for around 100 districts). Those reports have a mostly fixed form : maps, graphs, data tables and small zone of text.

Apart for some discussion around colors and legends, it isn't difficult to automate the production of maps / graphs / tables. (I work with Rmarkdown if you want to know)

However, for the text, a simple approach like writing 'r value' in markdown to produce a variable value inside of the text feel 'too automated'. The reports end up having ten sentences like 'During the last quarter (QX 201X) total result was XXX (a +X% growth compared to the same quarter the previous year).'

I'd like to get automatic variations of that phrase without modyfiying it's meaning. I've ended up writing half a dozen variations myself. But (1) it still feels repetitive and unnatural, and (2) doing it for every phrase of the report may take a lot of time.

We have seen a lot of extraordinary things in transfering things for visual representation (see : https://en.wikipedia.org/wiki/Neural_Style_Transfer). So I was wondering if we have similar things for NLP, that would allow a text to be rewritten using a different 'style' (a neutral style -or an absence of style- in my case), keeping it's main content. The main paper I found on the subject is titled 'What is wrong with style transfer for texts?' and shows why style transfer doesn't really work for texts. Given (1) the constraint (keeping the same meaning) and (2) it's formalism (I know which number should be shown), I feel like the problem may be simpler than the whole style transfert.

Any idea where to start to automatically write variations of a text while keeping it's meaning constant ?


texts: What are the standard parameters values for SMOTE technique?

texts:
I'm working on an imbalanced class data set (200 samples) with 2 classes, first class has 50 sample and second has 150 sample.

My questions:

When I use SMOTE technique on my data set my total dataset samples
will be greater than 200 or smaller?
Is there standard parameters value about SMOTE technique or not?


texts: Layman's Interpretation of XGBoost Importance

texts: I'm trying to come up with a good way to explain the 3 importance metrics (Gain, Cover, Frequency) to a layman with only a basic understanding of XGBoost and trees in general.  How best would you frame this explanation?


texts: Machine learning methods on 1 feature dataset

texts:
Assuming I have following dataset but much longer. Can I use any machine learning methods having only one feature? Giving Name and predicting Fullname. I'm newbie and think that doesn't make any sense cause it applies just basic statistics computations as mode. But maybe is something better? If some methods can be used, which ones?

Name   | Fullname
------------------
John   | Novak
Adam   | Johnson
Sophia | Loren
Dave   | Miller
Anna   | Smith
Dave   | Smith
...


texts:
Here is my xgboost hyperparameters

xgb = XGBClassifier(
     bagging_fraction= 0.8,
     boosting= 'gbdt',
    colsample_bytree= 0.7,
    feature_fraction= 0.9,
    learning_rate= 0.05,
    max_bin= 32,
    max_depth= 10,
    min_child_weight= 11,
    missing= -999,
    n_estimators= 400,
    nthread= 4,
    num_leaves= 80,
    predictor= 'gpu_predictor',
    seed= 1337,
    silent= 1,
    subsample= 0.8,
    tree_method= 'gpu_hist',
    metric = 'logloss',
    verbose= True
)

my result is not very stable
which parameter should I add to make it stable?


texts: Prediction error without having a true value

texts:
Quick summary about the problem: we are trying to deploy our regression model, where the clients require "individual prediction error". Since we're predicting something unknown in advance, we can't measure the error the standard way of y_true - y_predicted.

I have done some research already, but the problems with existing methods are this:

Since we're using boosting algorithms (xgboost, catboost), we can't
rely on normality assumptions and generate standard confidence
intervals.
A solution proposed here is to train multiple models and get the
average of it, but then it's not viable in the production level as it
would become at least 3 times as slow to train / predict.
Another way is to create a quantile regression as stated here,
but this would impact our accuracy, which we can't sacrifice.
Finally, we have tried to train a model using our validation set error and try to predict the test set error, but the accuracy is very very low.

So my question is this: Is there a way to estimate an individual prediction error, without knowing it's true value in advance, that wouldn't affect the model accuracy or speed?


texts: How to pass features extracted using CNN into RNN?

texts:
I have word images as below:

Let's say it's a 256x64 image. My aim is to extract the text from the image as 73791096754314441539 which is basically what an OCR does.
I am trying to build model which can recognise word from images.
When I am saying word it can be any of the follwing:

Any dictionary word, non-dictionary word
a-z,A-Z, special characters including spaces

I have built a model (snippet because of company policies) in tensorflow as below:

inputs = tf.placeholder(tf.float32, [common.BATCH_SIZE, common.OUTPUT_SHAPE[1], common.OUTPUT_SHAPE[0], 1])
# Here we use sparse_placeholder that will generate a
# SparseTensor required by ctc_loss op.
targets = tf.sparse_placeholder(tf.int32)

# 1d array of size [batch_size]
seq_len = tf.placeholder(tf.int32, [common.BATCH_SIZE])

model = tf.layers.conv2d(inputs, 64, (3,3),strides=(1, 1), padding='same', name='c1')
model = tf.layers.max_pooling2d(model, (3,3), strides=(2,2), padding='same', name='m1')
model = tf.layers.conv2d(model, 128,(3,3), strides=(1, 1), padding='same', name='c2')
model = tf.layers.max_pooling2d(model, (3,3),strides=(2,2), padding='same', name='m2')
model = tf.transpose(model, [3,0,1,2])
shape = model.get_shape().as_list()
model = tf.reshape(model, [shape[0],-1,shape[2]*shape[3]])

cell = tf.nn.rnn_cell.LSTMCell(common.num_hidden, state_is_tuple=True)
cell = tf.nn.rnn_cell.DropoutWrapper(cell, input_keep_prob=0.5, output_keep_prob=0.5)
stack = tf.nn.rnn_cell.MultiRNNCell([cell]*common.num_layers, state_is_tuple=True)

outputs, _ = tf.nn.dynamic_rnn(cell, model, seq_len, dtype=tf.float32,time_major=True)

My current approach is to use take input a word image pass it through a CNN extract high level image features, convert the image features to sequential data as below  [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3]] -> [[a1,a2,a3],[b1,b2,b3],[c1,c2,c3]]
then pass it through a RNN(LSTM or BiLSTM), then use CTC (Connectionist Temporal Loss) to find the loss and train network.
I am not getting results as expected, I wanted to know if:

There is some other better way to do this task
If I am converting features to sequence correctly
Any research paper where something like this is done.


texts: What are the limits of capping ratios to [0-1]?

texts:
I work with some financial ratios. By construction those variables should be between 0 and 1. However, for some reasons I end up with values <0 and >1. These reasons include :

reporting problems (fat finger errors, unit problems...)
discrepencies between databases
discrepencies due to accounting reassessment

What would be the reasons for me to not cap my ratios to [0,1] ?


texts: How to obtain and load a good initial data set for object localization?

texts:
I'm looking for a good data set for training a CNN based network to do object localization (i.e. a data set with class labels and bounding box data).

What is a good initial data set to use?

How can I obtain and load all the data (images, labels, bounding boxes)?

I'm using python, so a python-friendly data source is ideal.


texts:
I have a pandas dataframe with a column CAS_BRM_IDA of type category (even if its values seem of foat type ... but they are not meaningful per se)

I built the following serie from that dataframe :

p = df.groupby( [ 'CAS_BRM_IDA' ] ).mean()[ 'TOP_FRD']
print( p )

Result displayed :

CAS_BRM_IDA
10.0       0.001131
13.0       0.000000
15.0       0.002038
17.0       0.000000
20.0       0.003802
...
missing    0.019549

I then tried to build a dictionary using that serie :

mydic = dict( [ ( i , p[i] ) for i in p.index ] )

But I got the following error message :

TypeError: cannot do label indexing on
  <class 'pandas.core.indexes.category.CategoricalIndex'>
   with these indexers [10.0] of <class 'float'>

Some details about the p index :

CategoricalIndex(
    [10.0, 13.0, 15.0, 17.0, 20.0, 21.0, 30.0, 31.0, 40.0, 43.0,
     50.0, 51.0, 56.0, 'missing'],
     categories=[10.0, 13.0, 15.0, 17.0, 20.0, 21.0, 30.0, 31.0, ...],
     ordered=False, name='CAS_BRM_IDA', dtype='category')

I can't figure out where is the problem.  '10.0' value seems to have been interpreted as float which I thought it's not.

How can I build a dict from a Series with a Categorical Index which contains floats?


texts:
I'm trying to classify images. I want to run each image through a pretrained CNN to apply convolution and pooling and end up with a smaller picture/matrix where the value of each pixel is a feature. Then I want to pass that to an SVM for classification.

Can someone provide code to get started with feature extraction with CNN and SVM ?


texts: When to Perform non-linear dimension reduction

texts: When do we feel need to go through non-linear transformation like kernel PCA ? Please share an example


texts: I have few points in $S^n$, i.e., the $n$-dimensional unit sphere embedded in $\mathbb{R}^{n+1}$, and I would like to project them down to $S^2$, i.e., the 2-dimensional unit sphere (embedded in $\mathbb{R}^3$) to visualize it with the constraint that neighboring points should be close by. I spent some time playing with t-sne but of course, the points no longer lie on $S^2$. I normalized the projections but that introduces weird distortions, for instance, if the variance of one dataset is very small in $S^n$ as compared to other, I expect the same to hold in their $S^2$ projections; that is not the case upon normalizing t-sne. Any ideas? I would really like something that makes the previous statement hold.


texts: Using ontology to infer labels for process model

texts:
I'm trying to implement a specific type of process mining, that has been presented in this thesis [link]. It is based on HMMs and generates a process model in form of a directed graph, where:

Nodes are called intentions and correspond to hidden states
Edges are called strategies and consist of different activities
These activities correspond to the HMM's observable emissions
Intentions can be fulfilled using different strategies

A user event log consisting of user IDs, timestamps and activities is used as input. The image below is an example of such a process model. The highlighted nodes and edges resemble the path that has been predicted using the Viterbi algorithm.

You can see that the graph's nodes and edges only carry numeric labels, which allow to distinguish between the different strategies and intentions. In order to make these labels more meaningful to the human reader, I'd like to infer some suitable labels.

My idea is to use an ontology to obtain those labels. After some research I figured out that I probably needed to do something that is generally referred to as "ontology learning". For this I would need to create some axioms in RDF/OWL format and then use these as input for a reasoner, that would infer an ontology.

Is this approach correct and reasonable to achieve my goal?

If this is the way to go, I will need some tool to generate axioms in an automated way. So far I couldn't find any tool that would do that completely out-of-the-box. Based on what I've seen so far I conclude that I would need to define some kind of mapping between the original data and the desired axioms. I took a closer look at protégé, which offers a plugin for spreadsheets. It seems to be based on the MappingMasterDSL project [link].

I've also found an interesting paper [link] on ontology learning where an RNN-based model is trained in a end-to-end fashion to translate definitory sentences into OWL formulae. BUT: My user event log data does not contain any natural sentences. Its activities are defined by tokens derived from HTML elements of the user interface. Therefore the RNN-based approach does not seem to be applicable here. (For the interested reader, the related project can be found here [link])

Isn't there really any easier way than hand-crafting the axioms' schema(ta)?

Assuming that I have created my axioms and inferred an ontology, I would like to use the strategies' (edges') observable activities (emissions) to infer a suitable label. I guess I would need to query my ontology somehow. I could use the activity names as parameters for my query and look for some related entities that reveal the desired label. I'm expecting something like:

  "I have a strategy with ID=3, that strategy can be executed with
  actions a, b and c, give me all entities of the ontology, that
  have these actions as property value and show and give me all related
  labels for those entities"

But where would the data for the labels actually come from?

I think I'm missing some important step during the process of ontology learning. Where do I find an additional data source for the labels and how do I relate this data to my ontology's entities?

Also I'm wondering if there is a way to incorporate the inherent knowledge of the process model's topology into my ontology.


texts: What are some good models to test the speed of a data science machine?

texts:
I'm writing a battery of tests (in Python) for the purpose of measuring the speed of my company's different computational instances. The goal is to see how fast different AWS EC2 instances are at running different ML models or common data science tasks.

I'm reaching out to ask if anyone knows of any time-consuming (but realistic) models that can be built using only the standard Anaconda packages and either the built in datasets, or a popular dataset known in the industry?

The goal would be to give end users a sense of how much computational power they need, and I think using some popular data sets or models to compare relative time to completion would be the best way for them to choose the right amount of power for their needs.


texts: Compare ranking across trials with pandas

texts:
I am carrying out several runs of an experiment with different configurations on many different inputs.  The experiment has a random element and I'm trying to investigate whether this randomness affects the "best" configuration choice.  Here's a simplified version of my data frame:
>>> d.tail(20)
       trial  runtime input config
29561      1    17.98  ss8a   te_p
29562      2    20.79  ss8a   te_p
29563      3    19.93  ss8a   te_p
29564      4    19.88  ss8a   te_p
29565      5    17.73  ss8a   te_p
29566      1    10.80  ss8a   te_d
29567      2    14.13  ss8a   te_d
29568      3    12.53  ss8a   te_d
29569      4    13.61  ss8a   te_d
29570      5    12.20  ss8a   te_d
29571      1     8.55  ss8a   te_w
29572      2    10.24  ss8a   te_w
29573      3     9.86  ss8a   te_w
29574      4    10.52  ss8a   te_w
29575      5    10.18  ss8a   te_w
29576      1    19.33  ss8a   te_r
29577      2    18.99  ss8a   te_r
29578      3    14.96  ss8a   te_r
29579      4    18.88  ss8a   te_r
29580      5    18.31  ss8a   te_r

Each input is run with 25 configurations, 5 times over.
I guess I'm asking both a statistics and a python pandas question... How can I tell whether the ranking is consistent across the different runs?  Of course I can take the median time as representative and report on the best configuration for each input, but I'm interested in how solid this verdict would be.  I know that there are big variations in runtime for a significant minority of inputs and configurations.
One idea I had (I am not a trained statistician) was to repeatedly draw random samples from the trials for each input and configuration combination and plot (or use a statistical measure to compare) the rankings of the different configurations.  One problem is that I have hundreds of inputs to do this for, so how could I visualise or summarise effectively?


texts: How to visualize change in a distribution with a few outliers that account for a very large percent of the total?

texts:
I'm working on an edtech product where some of our traffic lands on webpages about textbooks.

Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "whales" - individual books that account for a large percent (~20%) of total subject traffic.

Year over year, these whales grow or shrink (sometimes books are replaced in a school, or schools drop textbooks altogether). This change in whale traffic contributes to a big change in overall subject traffic.

I'm trying to figure out how to visualize this change, given a dataset that looks something like the table attached below (it shows only 2 months traffic per book, but I have access to all the months).

I've tried overlapping histograms (and boxplots), where each histogram is a month. But this visualization doesn't indicate how huge my whales (outliers) are, and how much influence they have.

Any help on chart types or how to otherwise tell this story would be much appreciated.


texts: Transformer seq2seq model and loading embeddings from XLM-RoBERTa

texts:
Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. Output (sign language sentence): Food he sells). But I have a very small dataset of sentence pairs - around 1000. And the language is a low resource language.

I am a new researcher on the field of deep learning. Please help with your valuable advice.


texts: setting class weights for imbalanced dataset in tensorflow

texts:
I have 24 classes in my dataset and I am trying to set class weights in the following way:

ty = np.argmax(trainy, axis=1)
class_weights = class_weight.compute_class_weight('balanced', np.unique(ty), ty)
class_weights = dict(enumerate(class_weights))
model.fit(trainX, trainy, epochs=epochs, batch_size=batch_size,  class_weight = class_weights)

However, I get this error in Colab and do not know where 12 comes from:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
 58     ctx.ensure_initialized()
 59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
 61   except core._NotOkStatusException as e:
 62     if name is not None:

InvalidArgumentError:  indices[0] = 15 is not in [0, 12)
 [[{{node GatherV2}}]]
 [[IteratorGetNext]] [Op:__inference_train_function_69536]

Function call stack:train_function


texts: Confidence score for all observations is between 0.50 - 0.55

texts:
Hello Data Science Stack Exchange Community,
This question will appear to be open-ended, however any answers or thought will be much appreciated. I am trying to go-through a pre-trained Random Model Classifier with minimum documentation like what was the confusion matrix, ROC-AUC curve of the classification problem when the model was developed. I only have the pickle file and the data set on which it needs to run. When I ran the model I observed for most of the cases the prediction score or confidence score is between 0.50 to 0.55
I would like to know is it right to say that "model might have learn the pattern even tough all the confidence score is between 0.50 and 0.55"
Because I would like to explain that since the all the predictions score are concentrated around a single value it is not right to say that the model learnt and probably if some one would have tried to plot AUC it would have been straight line.
Again any thoughts and answers will be much appreciated.


texts: What could explain a much higher F1 score in comparision to accuracy score?

texts:
I am building a binary classifier, which classifies numerical data, using Keras.

I have 6992 datapoints in my dataset. Test set is 30% of the data. And validation set is 30% of the training set.

When evaluating the model, I get these values:

recall:  0.8914240755310779
precision:  0.7006802721088435
f1_score:  0.7846260387811634
accuracy_score:  0.7035271816800843

How come is the accuracy_score so about 10% lower than the F1-score?

Here is the code I'm using to evaluate the model:

print('recall: ', recall_score(Y_test, y_pred))
print('precision: ', precision_score(Y_test, y_pred))
print('f1_score: ', f1_score(Y_test, y_pred))
print('accuracy_score: ', model.score(X_test, Y_test, verbose=0))

And here is my model:

def create_model(neurons=23):

    model = Sequential()
    model.add(Dense(neurons, input_dim=37, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

    # Compile model
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', precision, recall])

    return model

model = KerasClassifier(build_fn=create_model, epochs=500, batch_size=5, verbose=1)

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=seed)

transformer = Normalizer().fit(X_train)
X_train = transformer.transform(X_train)
transformer = Normalizer().fit(X_test)
X_test = transformer.transform(X_test)

tensorboard = TensorBoard(log_dir="logs/{}".format(time.time()))
time_callback = TimeHistory()
es = EarlyStopping(monitor='val_acc', min_delta=0, patience=20, verbose=0, mode='auto',restore_best_weights=True)

# Fit the model
history = model.fit(X_train, Y_train, validation_split=0.3, epochs=200, batch_size=5, verbose=1, callbacks=[tensorboard, time_callback])


texts: How can I increase the number of iterations per epoch in MATLAB?

texts:
I am training a deep learning network using MATLAB and would like to increase the number of iterations per epoch.

Using trainingOptions

options = trainingOptions('sgdm', ...
    'Momentum',0.9,...
    'MaxEpochs',100, ...
    'ValidationData',{XValidation,YValidation}, ...
    'MiniBatchSize', 64,...
    'Verbose',false, ...
    'Plots','training-progress');

I get the following, where the number of iterations per epoch is set automatically to 1. Therefore 1 pass of 64 training examples is needed for one epoch. How can I increase this to, for example 10 iterations per epoch?

According the the docs:

  An iteration corresponds to a mini-batch.

As I understand, the number of iterations is the number of passes, each pass using 64 (batch size) number of examples. So in the above, I have 1 pass of 64 training examples for 100 epochs.

Does this mean the iterations per epoch are set automatically according to how many training examples there are present? Can I alter this parameter?


texts: Batch normalization variance calculation

texts:
In batch  normalization the variance calculation during the training phase is done by ($x_i$ are the individual elements in the training batch of size $m$)

$\sigma_B^2 = \frac 1m \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during the test time we calculate the population statistics for the same as

$E[x] = E_B[\mu_B]$ and $Var[x] = \frac{m}{m-1}E_B[\sigma_B^2]$ (as you can see the unbiased estimate of population variance is calculated during the test time as we are building the model to predict the population distribution)

Would it not seem more logical that bias correction be done during training phase itself and subsequently its expected value can be used without the $\frac {m}{m-1}$ correction factor. i.e

During training:

$\sigma_B^2 = \frac 1{m-1} \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during test time:

$E[x] = E_B[\mu_B]$ and $Var[x] = E_B[\sigma_B^2]$


texts: ROC curve for different hyperparameters of `RandomForestClassifier`?

texts:
I'm currently trying to train a RandomForestClassifier on a dataset consisting of 5000 instances with 12 (now) encoded features and a binary target label. Through GridSearchCV I found out, that

best_parameters = {
    'criterion':    'gini',
    'max_depth':    12,
    'max_features': 'log2',
    'n_estimators': 300
}

works best out of

hyperparameters = {
    "n_estimators": [9,10,20,30,40,50,60,100,150,200,300,1000],
    "max_depth":    [3,6,9,12,20],
    "criterion":    ["gini", "entropy"],
    "max_features": ["log2", "auto"]
}

And gives a mean_test_score of 0.8546 which is quite good already, as I think.

Now I would like to get some kind of visual interpretation like a ROC curve for each parameter. But does it actually make sense in the case of a RandomForestClassifier to create a ROC curve for each hyperparameter? Or are there other ways to tune my classifier?


texts:
I am having a hard time evaluating my model of imputation.
I used an iterative imputer model to fill in the missing values in all four columns.
For the model on the iterative imputer, I am using a Random forest model, here is my code for imputing:
imp_mean = IterativeImputer(estimator=RandomForestRegressor(), random_state=0)
imp_mean.fit(my_data)
my_data_filled=  pd.DataFrame(imp_mean.transform(my_data))
my_data_filled.head()

My problem is how can I evaluate my model. How can I know if the filled values are right?
I used a describe function before and after filling in the missing values it gives me nearly the same mean and std. Also, the correlation between variables stayed nearly the same with slight changes.


texts: Difference between impurity and misclassificaton

texts:
I am reading the gini index definition for decision tree:

Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.

This seems to be the same as misclassification. Is Gini index just a fancy name for misclassification? Or is there really some subtle difference? Thanks!


texts: Create User Table from Google Firebase Analytics and Cloud Firestore Collections

texts:
I am exporting my snowplow collections collected on Firebase using the extention
Export Collections to BigQuery
firestore-bigquery-export
v0.1.10.
Whenever a document is created, updated, imported, or deleted in the specified collection, this extension sends that update to BigQuery. The columns of my interest are as follows.
['etl_tstamp','event','event_id',
 'user_id','geo_country','geo_region','geo_zipcode','geo_latitude','geo_longitude','geo_region_name','page_url','page_referrer','page_urlhost','contexts','br_lang','os_timezone','geo_timezone','derived_tstamp']

I also have my events table on BigQuery which is created from the Integrations in Firebase and exports raw event data from Google Analytics from my app.
My Aim is to create a user_table with user details by joining these 2 tables in an ETL job which has Demographic Information and etc..
I am confused with the approach I need to take for creating this table.


texts: What machine learning technique should I use in medical problem

texts:
What are the best machine learning techniques to classify responders to a medicine if I have:

Clinical data with ~200 features (age, education, marital status etc.)
Gene data with around 250K features (genom data (snips) taken from the patient (DNA analysis))
Number of obs. ~ 4K (the data is taken from a study on 4000 patients).

Please advise.


texts: Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

texts:
I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it based on DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter paper? This is published in March 2020. I am looking for links between this paper and Sentence-BERT (sentence-transformers). Original sentence-bert paper is published in Aug 2019. I wanted to try pre-trained model using S-BERT model and hence tried distilbert-base-nli-mean-tokens model. After implementation I found out that it's much faster than other pre-trained models available on sentence-transformer website. While studying it's paper I realised original paper do not mention this pre-trained model.
I found Making Monolingual Sentence Embeddings Multilingual using
Knowledge Distillation this paper published by same author which do mention DistilmBERT but not DistilBert Can someone please help me solve this mystery?


texts: Is there an existing way to find out if there is an association between two words in a sentence using NLP

texts: So, I was working on word2vec , playing with it and although it's a great tool for finding similarities between two words,But when we come to finding a semantic association between two words it cannot be used. It would be interesting to know if there's an existing way, if yes how accurate is it and what principles does it work on?


texts:
Background

I would like to build a face recognition model for registration and login for some kind of service. For example, using this approach (CNN + SVM).

When a new user wants to register a service, the image of his/her face is recorded and the machine learning model is trained using these images. Then, when a person requests for the service, the model recognises if this person is a member or not.

Question

But when there is new user comes in for registration, the machine learning model has to be retrained and scans through all the previous images (or feature vectors). It seems that this approach has a scalability issue when the the number of users is large.

I have read through this post but my situation is not quite the same as the suggested answer, because my machine learning model aims to distinguish member vs non-member. Does anyone know how to tackle this scalability issue? Thanks.


texts: Handling a large dataset consisting of npy files

texts:
I have a high number of npy files (448 files) each consisting of around 12k frames (150x150 RGB images) which together make the input to my neural network (X). However, since it is impossible to load all of the files into a single array, and the fact that it is necessary to shuffle all of the samples to avoid bias, how do I create the input and feed it to the network? Someone suggested creating a dummy array to represent indices, shuffle that, create chunks based on the array size and the indices and then feed the chunks to the neural network. However, I was wondering if there is another simpler method.
So in a word, I would like to do this step but with a high number of large npy files:

X_train_filenames, X_val_filenames, y_train, y_val = train_test_split(...)

Note1: Some suggested using TFRecords but I could not find out how to convert and use them.


texts:
In my problem I want to distinguish people from other shapes in images e.g I want to accurately know how many people are in specific region of image (at least for small number of people, for crowded places it is reasonable to get worse results).

In OpenCV there are three methods with pretrained datasets. Two basic HOGs + SVM algorithms (one trained on inria and the other on daimler dataset) and DPM method that I used also with inria dataset.
Models were trained on images of pedestrians of size 64x128(inria) and 48x96(daimler).

I've been doing my tests on a video file with two people in it, but those people are standing or sitting.
From my observations I can tell that:

DPM is the best algorithm, but it is really slow. It can detect human in sitting position.
HOGs are very dependent on the scale e.g. svm trained on daimler dataset works better with small resolution(180p in my case, because camera was close to the people), but for inria results are worse
HOGs works better when full human shape is clearly visible
Daimler gives a lot of false positives

The results coincide with expectations, because models where trained on standing pedestrians, but even when people were standing the accuracy is really bad.
Basically I need to train my own models, but I am concerned about such a strong dependence on the scale and doubt in the sense of using these methods.

I was thinking about using CNN's, but my target device has ARM cpu (https://www.mediatek.com/products/homeNetworking/mt7623n-a).
I don't need this detection to work in realtime and I'll be feeding the network with only a part of image (smaller than 480p).

Do you think that such a network can work with decent performance on such cpu?
Do you have any suggestions on type of the network, library (c++) that I could try?

UPDATE

I am experimenting with OpenCV's DNN module and yolov3-tiny. It processes whole frame in ~2s and doesn't use a lot of ram (on my target device). I am really satisfied with predictions. Of course, full yolo is better than tiny version, but much slower and consumes a lot of ram.

I am confused about number of models that exists and datasets they were trained on. Caffe Zoo has a lot of them, but I found that e.g GoogleNet doesn't have person class, so this is way I chose yolo for tests. But Yolo also is available for COCO or VOC and I just need to classify people not dogs etc.

I still have to experiment with input parameters in opencv to see how they affect performance and accuracy.

I am also thinking about retraining yolo for humans only maybe in this way I'll reach accuracy of full yolo and performance of the tiny one. What do you think? In general it is better to use COCO or VOC dataset?


texts: Establishing the ground truth for difficult cases when it comes to images

texts:
I need to create some labels with labelImg for specific objects, but there are two main factors, which make the process of creating bounding boxes a difficult task.

The objects of interest are small compared to the size of the image and stacked together, think like two or more rectangles stacked together.
The quality of the image is not very good, which means that if I zoom in a very high level, the quality of the image worsens, thus I am not even able to distinguish their boundaries.

Has anyone encountered a similar problem?


texts: Is LSTM a proper implementation for sequence classification

texts:
I have 20k samples in two classes namely positive and negative. Samples are formed of 10 digits and within each sample a digit is used exactly once. For instance; 0123456789 and 9876543102 are proper samples as each digit is used once. Order of the digits are what makes samples positive or negative. A classification statement for instance, positive samples can have 2 after 5, but negatives can't. Or a more intricate one like, 90% percent of positive samples have leading 0's and none of negatives have this. For the remanining 10% percent, 512 and 315 are patterns that makes samples positive.
There is a total of 10!=3.6m possible samples. So, in order to carry out a binary classification, what should be the proper ML approach? Can LSTM's handle this? Maybe a primitive Neural Network with 10 inputs and a binary classification? Or I should look for a different ML approach? Thanks.


texts: Training on data with inherently non-applicable data cells

texts:
I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense.

Chemical Dataset

Contains thousands of rows of chemical mixtures with many columns of properties. Example properties: bromine content, density.

Inherently non-applicable data

The chemicals can be gas, liquid or solid but some properties are only applicable to samples of a certain state. An example could be viscosity in liquids, bond type (ionic, molecular, covalent) in solids or density in gas.

So far...

...all research has pointed towards methods of fixing "missing values" via column means, data imputation or something similar. There doesn't seem to be any sense in imputing the freezing point of a gas. A gas mixture does not have a freezing point. I am still in the process of data preparation and unsure how to proceed.

I am working in python and missing data is stored as NaN values. Perhaps there are some models that can deal with such NaN-values.

Side-Note:

The majority of the dataset is comprised of distillation curve datapoints (sequential data describing what percentage of a chemical sample evaporates as temperature is increased). This data is present for all samples.

Follow-up 1: Is there a model that will give me NaN values for the freezing point when I give it something that resembles a gas?

Follow-up 2: Can this be compared to image object detection where the object is partially obscured? or part of the image is corrupt?


texts: Don't understand the simple ResNet from 'Deep Implicit Layers' Tutorial

texts:
The Deep Implicit Layers Tutorial is a nice resource that dives into Neural ODEs, Deep Equilibrium Models etc., using the JAX library. In chapter 3 (out of 5), [Link: http://implicit-layers-tutorial.org/neural_odes/] there is the following example ResNet, which I am confused about:
import jax.numpy as jnp

def mlp(params, inputs):
  # A multi-layer perceptron, i.e. a fully-connected neural network.
  for w, b in params:
    outputs = jnp.dot(inputs, w) + b  # Linear transform
    inputs = jnp.tanh(outputs)        # Nonlinearity
  return outputs

def resnet(params, inputs, depth):
  for i in range(depth):
    outputs = mlp(params, inputs) + inputs
  return outputs

What is particularly confusing to me is that in the 'resnet' function, the for loop over the depth seems COMPLETELY redundant.
The inputs of previous layers are not being fed to later layers, and the 'i' index is not being used.
Am I missing some fundamental information about ResNets? Is it a mistake? Can someone explain the point of the for loop?


texts: How to handle overfitting in the following classification case

texts:
The confusion matrix is as below :-

[[  0   0   5   1   0   0]

[  0   0  19  14   0   0]

[  0   0 217 151   0   0]

[  0   0  84 282   0   0]

[  0   0   6 111   0   0]

[  0   0   0  10   0   0]]

np.random.seed(0)
classifier = Sequential()
classifier.add(Dense(300,input_dim=11))
classifier.add(Dropout(0.5))
classifier.add(keras.layers.LeakyReLU(alpha=0.86))
classifier.add(Dense(250,activation='tanh',kernel_regularizer=keras.regularizers.l2(0.12)))
classifier.add(Dropout(0.5))
classifier.add(Dense(100,activation='relu'))
classifier.add(Dropout(0.4))
classifier.add(Dense(6,activation='softmax'))

classifier.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])

Thanks for the help


texts: What is the difference between SGD classifier and the Logisitc regression?

texts:
To my understanding, the SGD classifier, and Logistic regression seems similar.
An SGD classifier with loss = 'log' implements Logistic regression and loss = 'hinge' implements Linear SVM. I also understand that logistic regression uses gradient descent as the optimization function and SGD uses Stochastic gradient descent which converges much faster. But which of the two algorithms to use in which scenarios? Also, how are SGD and Logistic regression similar and how are they different?


texts: I have a non-linear dataset. How can I do regression in orange? Orange does not have polynomial regression widget.


texts: How to input LSTM output to MLP with concatenate?

texts:
I am having a training data set for a time-series dataset like below where my target variable is var1(t) which is the value of var 1 at time=t.
import numpy as np
import pandas as pd
train_df = pd.DataFrame(np.random.randint(0,100,size=(100, 16)))
train_df.columns=['var1(t-3)','var2(t-3)','var3(t-3)','var4(t-3)','var1(t-2)','var2(t-2)','var3(t-2)','var4(t-2)','var1(t-1)','var2(t-1)','var3(t-1)','var4(t-1)','var1(t)','var2(t)','var3(t)','var4(t)']
train_X=train_df.drop(['var1(t)'],axis=1)
train_y=train_df[['var1(t)']]

I am inputting the LSTM network with the past 3 timesteps (t-3) to (t-1) of all 4 variables and then feed the output of the LSTM with the current timestep value of the var2,var3,var4 with an MLP with functional API in Keras.
So I prepared the inputs for the LSTM and MLP like below :
#subset the 3 previous timesteps of the 4 variables for the time sries part
train_X_LSTM=train_X[train_X.columns[:12]].reset_index(drop=True).values
#target is always var1(t)
train_y_LSTM=train_y.values
#take the current timestep fatures which are var2,var3,var4 which are realised at t=t
train_X_MLP=train_X[train_X.columns[-3:]].reset_index(drop=True).values
#target is always var1(t)
train_y_MLP=train_y.values

Then I tried the below network :
#lstm input shape
lstm_input = Input(shape=(train_X_LSTM.shape[0],train_X_LSTM.shape[1]))
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[0],train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = concatenate([lstm_output, mlp_input], axis=-1)
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit(x_train, y_train, batch_size=64, epochs=10, validation_split=0.2)

This throws the error of

ValueError: A Concatenate layer requires inputs with matching shapes
except for the concat axis. Got inputs shapes: [(None, 1), (None, 100,
3)]

which shows I am not combining them correctly. Any help is appreciated!


texts: Spoken utterance classification on RAVDESS using MFCC

texts:
I am planning to classify two audio files in which different sentences are spoken. Don't want to do speech to text as on prem speech to text conversion models are not good, and don't want to go to cloud. So planned to use RAVDESS dataset, which is basically for emotion detection. There are two sentences spoken in RAVDESS dataset - 01 = "Kids are talking by the door", 02 = "Dogs are sitting by the door"
The approach I took is to convert label 0 and 1 for these two sentences audios, extract MFCC signals and then do a binary classification. However, I am not getting accuracy more than 70%. Can someone please let me know what can be the reason. I suspect any of following:

Is the approach itself is wrong?
MFCC signals can not be used for spoken text classification?
Am I missing some feature engineering in MFCC before feeding to classifier?
There can be problem with my code or classifier? I have pasted code below
Any other suggestion?

    # Generate data for statement_type
    import time,os
    import librosa, numpy as np

    # path = '/content/drive/My Drive/Ravdess/'
    path = '/content/RAVDESS-emotions-speech-audio-only/Audio_Speech_Actors_01-24'
    lst = []

    start_time = time.time()

    for subdir, dirs, files in os.walk(path):
      for file in files:
        # print(file)
          try:
            #Load librosa array, obtain mfcss, store the file and the mcss information in a new array
            X, sample_rate = librosa.load(os.path.join(subdir,file), res_type='kaiser_fast')
            mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=40).T,axis=0)
            # This is because our predictor needs to start from 0 otherwise it will try to predict also 0.
            # file = int(file[7:8]) - 1
            # file = int(file[13:14]) - 1
            label = np.array([1,0] if int(file[13:14]) - 1 == 0 else [0,1])
            # print(label)
            arr = mfccs, label
            lst.append(arr)
          # If the file is not valid, skip it
          except ValueError:
            print("error at : " , file)
            continue

# Model and training
model.add(Input(shape=(40,1)))
model.add(LSTM(512, activation="relu", return_sequences=True))
model.add(Dropout(0.3))

model.add(Flatten())
model.add(Dense(2))
model.add(Activation('sigmoid'))
opt = keras.optimizers.Adam(lr=0.0001)
model.summary()

model.compile(loss=keras.losses.BinaryCrossentropy(), optimizer=opt, metrics=['accuracy'])

trainhistory=model.fit(x_traincnn, y_train, batch_size=16, epochs=300, validation_data=(x_testcnn, y_test))


texts: Fit Decision Tree to Gradient Boosted Trees for Interpretability

texts:
I was wondering if there is literature on or someone could explain how to fit a decision tree to a gradient boosted trees classifier in order to derive more interpretable results.
This is apparently the approach that Turi uses in their explain function which outputs something like this:
Turi's explain function:

from their page here.
I know that for random forests you can average the contribution of each feature in every tree as seen in the TreeInterpreter python package, but this seems to be a different method since it is focused on exact splits and one decision tree.
Does anyone know anything more about this method for interpreting gradient boosted trees?


texts: What is divergence exactly in machine learning?

texts: I know about KL divergence, JS Divergence and clearly know that it is different from the divergence in calculus. I have an intutive feeling of divergence as it roughly compares the closeness of two distributions. I know that it is not a distance(maybe due to non-symmetricity like KL). But I don't know what it is exactly and how it is defined mathematically? I am a machine learning beginner So Pardon me for mistakes.


texts: How Calculate Effect (percentage) label of the input variables on the output variable by DecisionTreeClassifier

texts:
a description problem below.
I have 10 words like X1 , X2 , X3 , ... , X10

and three Label like short , long , hold.

My problem is that how calculate Effect (percentage) label of the input variables
by DecisionTreeClassifier Algorithm.
DT=DecisionTreeClassifier()
DT.fit(X_train, y_train)

and how calculate Effect (percentage) label of the input variables


texts: How to improve a LSTM self-attention model given the absence of overfitting

texts:
I am doing a binary classification on time series data.
Class 0 is a single class but class 1 is actually a combination of 7 different classes.
My objective to classify class 0 from other classes.
The training set has a shape of (4917,100,136) as (batches, timesteps, features).
In the beginning, the model shows significant overfitting problem.
I managed to minimize the overfitting problem through:

Reduced layer units.
Increased or decreased number of layers( did not affect the model much)
Applied regularization through l2 and dropout.

I have also applied class_weight to the model to mitigate the unbalanced issue.
I tried different Adam and RMSprop, not much difference.
    Dense_unit = 32
    LSTM_unit = 32

    dense_reg = 0.003
    attention_param = LSTM_unit*2
    attention_init_value = 1.0/attention_param

    u_train = np.full((X_train.shape[0], attention_param),
                      attention_init_value, dtype=np.float32)
    u_val = np.full((X_val.shape[0],attention_param),
                     attention_init_value, dtype=np.float32)
    u_test = np.full((X_test.shape[0],attention_param),
                     attention_init_value, dtype=np.float32)

    with keras.backend.name_scope('BLSTMLayer'):
        # Bi-directional Long Short-Term Memory for learning the temporal aggregation
        input_feature = Input(shape=(X_train.shape[1],X_train.shape[2]))
        x = Masking(mask_value=0)(input_feature)

        x = Dense(Dense_unit,kernel_regularizer=l2(dense_reg), activation='relu')(x)

        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(dense_reg),activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(dense_reg),activation='relu')(x)
        x = Dropout(dropout)(x)

        y = Bidirectional(LSTM(LSTM_unit,activity_regularizer=l2(0.0029),kernel_regularizer=l2(0.002),recurrent_regularizer=l2(0.002),return_sequences=True, dropout=dropout))(x)
        y = Bidirectional(LSTM(LSTM_unit,activity_regularizer=l2(0.0029),kernel_regularizer=l2(0.002),recurrent_regularizer=l2(0.002),return_sequences=True, dropout=dropout))(y)

    with keras.backend.name_scope('AttentionLayer'):
        # Logistic regression for learning the attention parameters with a standalone feature as input
        input_attention = Input(shape=(LSTM_unit * 2,))
        u = Dense(LSTM_unit * 2, activation='softmax')(input_attention)

        # To compute the final weights for the frames which sum to unity
        alpha = dot([u, y], axes=-1)  # inner prod.
        alpha = Activation('softmax')(alpha)

    with keras.backend.name_scope('WeightedPooling'):
        # Weighted pooling to get the utterance-level representation
        z = dot([alpha, y], axes=1)

    # Get posterior probability for each emotional class
    output = Dense(num_classes, activation='softmax')(z)

    model = Model(inputs=[input_attention, input_feature], outputs=output)

    optimizer = opt_select(optimizer,learning_rate)

    model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=optimizer)

    print(model.summary())

    hist = model.fit([u_train, X_train],
                     y_train,
                     shuffle=False,
                     batch_size=batch_size,
                     epochs=epochs,
                     class_weight = {0:1.75607143,1:0.69903327},
                     validation_data=([u_val, X_val], y_val)
                    )

The training accuracy is very close to the validation accuracy(orange line) as shown in the screenshot. Further regularization didn't improve validation accuracy.
Questions:
How can one improve the model under these situations?
The self-attention actually did not make the prediction better than models contains only pure LSTM.
Can anyone suggest how to apply more self-attention layers into the model?


texts: XGBoost: Quantifying Feature Importances

texts:
I need to quantify the importance of the features in my model. However, when I use XGBoost to do this, I get completely different results depending on whether I use the variable importance plot or the feature importances.

For example, if I use model.feature_importances_ versus xgb.plot_importance(model) I get values that do not align. Presumably the feature importance plot uses the feature importances, bu the numpy array feature_importances do not directly correspond to the indexes that are returned from the plot_importance function.

Here is what the plot looks like:

But this is the output of model.feature_importances_ gives entirely different values:

array([ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.00568182,  0.        ,  0.        ,  0.        ,
        0.13636364,  0.        ,  0.        ,  0.        ,  0.01136364,
        0.        ,  0.        ,  0.        ,  0.        ,  0.07386363,
        0.03409091,  0.        ,  0.00568182,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.00568182,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.00568182,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.01704546,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.05681818,  0.15909091,  0.0625    ,  0.        ,
        0.        ,  0.        ,  0.10227273,  0.        ,  0.07386363,
        0.01704546,  0.05113636,  0.00568182,  0.        ,  0.        ,
        0.02272727,  0.        ,  0.01136364,  0.        ,  0.        ,
        0.11363637,  0.        ,  0.01704546,  0.01136364,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ], dtype=float32)

If I just try to grab Feature 81 (model.feature_importances_[81]), I get:0.051136363. However model.feature_importances_.argmax() returns 72.

So the values do not correspond to each other and I am unsure about what to make of this.

Does anyone know why these values are not concordant?