texts: Understanding multi agent learning in OpenAI gym and stable-baselines

texts:
I was trying out developing multiagent reinforcement learning model using OpenAI stable baselines and gym as explained in this article.
I am confused about how do we specify opponent agents.
It seems that opponents are passed to environment, as in case of agent2 below:
class ConnectFourGym:
    def __init__(self, agent2="random"):
        ks_env = make("connectx", debug=True)
        self.env = ks_env.train([None, agent2])

The ks_env.train() method seems to be the one from kaggle_environments.Environment:
def train(self, agents=[]):
    """
    Setup a lightweight training environment for a single agent.
    Note: This is designed to be a lightweight starting point which can
          be integrated with other frameworks (i.e. gym, stable-baselines).
          The reward returned by the "step" function here is a diff between the
          current and the previous step.
    Example:
        env = make("tictactoe")
        # Training agent in first position (player 1) against the default random agent.
        trainer = env.train([None, "random"])

Q1. However I got confused. Why does environment contains train() method? That is why environment should do training? I feel, train() should be part of the model: above article uses PPO algorithm which contains train() method. This PPO.train() gets called when we call PPO.learn() which makes sense.
Q2. But then, reading PPO.learn()'s code, I dont see how it trains current agent against multiple opponent agents. Shouldnt model algo do this? Am reading it wrong? Or model is unaware of number of agents, its just known to environment and thats why environment contains train()? In that case, why does we have explicit Environment.train() method? Environment will return rewards as per multiple agent behavior and model will learn from that.
Or am completely screwed in basic concepts? Can somoene help me out?


texts: How to implement various SVR types in python?

texts:
I am new to the Machine learning world and have created some models on wind speed prediction using SVR and RF.
Now, I wish to implement and compare these results with LSSVR (Least square support vector regression) and TSVR(Twin support vector regression).
However, the problem is that I am unaware of existing libraries for the same. If anyone can help me out a way or suggest that where I can access the required regression models would be great!
PS: I am using jupyter notebook (python 3)


texts: Data aggregation and split train test samples

texts:
I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e.g., temperature, humidity, wind, rain, sun radiation).

Some of my weather features (temperature, humidity, wind) are in hourly intervals and others in daily intervals (rain, sun radiation).
My target (daily electricity consumption) is also in hourly intervals.

My goal is to predict daily consumption so I need to aggregate my input data at a daily interval (sum of 24 values for consumption, average of 24 temperatures, ....)

My question is: Do I have to aggregate the data before split train/test of after ?

If I do aggregation first, I will consider only those days with 24 values (one for each hour) and drop others in order to not introduce bias
Then I will split train/test. So basically, I will clean my data before splitting.

Am I missing something?


texts: Use of Correlation Map in Machine Learning

texts: I would like to know the use of correaltion map in machine leraning. For example, if there are 2 features with high correaltion, should either of the features be removed before appying the algorithm or it depends on every data set. Any explanation would be highly helpful. Thanks in advance.


texts: How to fill df with data from another df - Python

texts:
I am trying to fill df with values form a different one.
The first df is:

The second df is:

I want to add the column to first df having values from the second one to receive output:

number
trans
date
value

HF01
one
sty
1

HF01
one
lut
2

-
-
-
-

HF05
five
paz
50

I came up with code:
found = []

for index, row in one.iterrows():
    for ind, rw in two.iterrows():
        if ind not in found:
            if (one.loc[index, 'number'] == two.loc[ind, 'number']) & \
               (one.loc[index, 'trans'] == two.loc[ind, 'trans']) & \
                    one.loc[index,'value'] =  two.loc[ind, one.loc[index, 'date']]
                    found.append(int(ind))
                    break

It works but it is extremely slow. How it should be done properly?


texts:
I'm doing research where a part of the collected data is of Ordinal type. I will implement ANN with Logistic Regression function in the Activation function. What I have learnt from documents of other websites as well as an answer in https://datascience.stackexchange.com/, the target value is of ordinal type while the independent variables are of ratio or interval type. But my independent data is of Ordinal type and the target data will be label (say Like or Unlike).
How should I build a function for ANN if I'm not wrong in my understanding?


texts: Calculate the number of params of a neural network


texts:
In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this:

1) Can I mix and match these two approaches? e.g. Feature Scale x1 and Mean Normalize x2?

2) How do you determine which of these options to apply? It seems that either could accomplish the task of increasing your convergence rates... I suppose you just need to know your data set to understand which will reliably reduce your values while leaving as few outliers as possible?


texts: Acurracy = 0 using tensorflow on tabular data

texts:
I am new to machine learning and trying to apply this tutorial to my tabular data. My input is a pandas dataframe containing the features (I encoded the categorical columns as floats) and a pandas series containing the labels, which are ints.
This is my code:

def df_to_dataset(dataframe, labels, shuffle=True, batch_size=32):
   ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
   if shuffle:
      ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds

feature_columns = []
# numeric cols
for header in list(X_train):
  feature_columns.append(feature_column.numeric_column(header))
feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
batch_size = 32
train_ds = df_to_dataset(X_train, y_train, batch_size=batch_size)
val_ds = df_to_dataset(X_val, y_val, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(X_test,y_test, shuffle=False,
  batch_size=batch_size)

model = tf.keras.Sequential([
   feature_layer,
   layers.Dense(128, activation='relu'),
   layers.Dense(128, activation='relu'),
   layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
          loss='binary_crossentropy',
          metrics=['accuracy'],)
          #run_eagerly=True)

model.fit(train_ds,
      validation_data=val_ds,
      epochs=5)
loss, accuracy = model.evaluate(test_ds)
print("Accuracy", accuracy)

First of all, when I print the shape of the datasets, all feature shapes are shown as (?,). I looked that up, but it didn't really get, what that meant, so I moved on.
When executing the model, I got the following output:

W0625 16:28:50.013361 140172694484864 deprecation.py:323] From
/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180:
add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops)
is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Epoch 1/5
2437/2437 [==============================] - 12s 5ms/step - loss:
-368933040.3744 - acc: 0.0000e+00 - val_loss: -1374389959.0878 - val_acc:
0.0000e+00
Epoch 2/5
2437/2437 [==============================] - 11s 4ms/step - loss:
-4239125012.7993 - acc: 0.0000e+00 - val_loss: -8055676778.8942 - val_acc:
0.0000e+00
Epoch 3/5
2437/2437 [==============================] - 11s 4ms/step - loss:
-14449097654.0468 - acc: 0.0000e+00 - val_loss: -21844830544.8532 - val_acc:
0.0000e+00
Epoch 4/5
2437/2437 [==============================] - 11s 4ms/step - loss:
-32560744568.1740 - acc: 0.0000e+00 - val_loss: -44181551604.6596 - val_acc:
0.0000e+00
Epoch 5/5
2437/2437 [==============================] - 11s 4ms/step - loss:
-60235093753.8022 - acc: 0.0000e+00 - val_loss: -76823729189.8015 - val_acc:
0.0000e+00
1219/1219 [==============================] - 3s 2ms/step - loss:
-78553874677.2896 - acc: 0.0000e+00
Accuracy 0.0

Obviously something is wrong, but I don't even know where to begin debugging this, so any tips are welcome!
Edit: the labels I am trying to predict are point scores from 80 to 100. Because of this, I switched from binary_crossentropy to mean_squared error. Now the loss is still very large, but positive and the accuracy is still 0


texts: How should I format input and output for text generation with LSTMs

texts:
I'm attempting to generate a response to an input line of text using an LSTM. I've considered various forms of input, including one-hot encoding each character in the line and passing each input line as a vector of one-hot encoded vectors. I've also considered using a dictionary and one-hot encoding each word in the sentence based on its alphabetical position.

However, I'm not sure about any of this, as I am new to natural language processing in machine learning. What would be the best way to format my input (and my output) for this problem?


texts: Finding lookalike for large number of users

texts:
We have a large user-base, within which we want to find lookalikes, around 25-30 Million, for the users, we have data such as show liked by the user, genre liked etc.

What we need to do is extrapolate our custom list (ex: find lookalikes for a list of 100,000 and extend it to 300,000) from our final result of lookalike_job_data, when we need to use the IDs somewhere.

This is written in Java and we use Threadpool for parallelizing:

Our current approach is, we have all our data in Apache Solr, as our data is updated incrementally, we query for 'term-frequency' of a shows_watched by a user, sort them as per 'term-frequency', we don't take into consideration the shows which have 'term-frequency' as 1
-> regard them as new users

or users which only have 2-3 shows in their list.

Then we query for users who have watched the same shows in pairs.
For ex. if we get 5 shows which have 'term-frequency'
such as sh1-> 7, sh2->4, sh3->5, sh4->6, sh5->14

We make pairs of 3 among them which are in 'AND' with 'OR' over all of the pairs as if our query will be as:

Find me users, that have watched :

(sh1 AND sh2 AND sh3) OR (sh2 AND sh3 sh4) and so on

Our query also uses 'frange'( a feature in 'SOLR' to limit a value) which limits the results to a distance from the base user.

It is applied to 'dist' (feature in 'SOLR') function, which gets the distance of other's shows to the base user
(same shows for which we the term-frequency).

So this will be our cluster for that user, and we ignore the rest of the user's in this cluster but still, this approach is slow for our usage, We'll probably run this job for full data once/twice a week and incrementally assign the new user's to existing clusters b/w them.

This process is still slow, can someone recommend a better approach, I am open to anything.

For 2 months our data size is around 140GBs.

Our SOLR runs on an AWS i3.2xlarge(8 core, 60GB ram, 2TB instance store SSD) machine,
and we fire queries from 4core, 16 GB ram machine

We initially have our data in S3.


texts: Comparing SMOTE to down sampling the majority class in imbalanced binary classification

texts:
I have a binary classification task with imbalance between the two classes.
I want to compare SMOTE vs down sizing the majority class to the size of the minority class.

I trained the classifier with 3-fold validation using the two methodologies:

SMOTE to increase the size of the minority class to the majority class size
Downsizing the majority class to the minority class size with a random subsampling

To test which methodology works better I trained my classifier (Random Forests) with a 3-fold Cross-Validation.

The confusion matrices I get from 3-fold CV seems to promote the use of SMOTE (better classification performance for the two classes). I assume that this CV can be used to choose the best methodology.

However, when I test the classifier on a real testing set (which was kept out and not used for training or validation) I don't see a real superiority of the SMOTE algorithm w.r.t. random subsample of the majority class. The minority class is better classified, but at the expense of the majority class performance.

Is this a limitation of SMOTE algorithm or my model selection methodology (using 3-fold CV) has some flaws?


texts: transfer learning with sentiment analysis?

texts:
The question is how good and what are some things to keep in mind when sentiment analysis models are tested on different datasets than they are trained on.

Say the task is to perform sentiment analysis on product reviews (unlabeled datset) - to classify positive, negative or neural. Because the data is unlabelled, a model can be trained (perhaps using logistic regression or NN) on a similar labeled dataset (say movie reviews, or product reviews) and tested on the original unlabelled dataset.

Will something like this work? Because the words of product names that occur in the unlabelled dataset will not be words that the model was exposed to during training, during test time will these words possibly throw off the model?


texts: Grouping by 2 Columns in R to Sum, Count, Percentage, Weighted mean, mode

texts:
I am trying to use dplyr or data.table to calculate sums, percentages, counts, and weighted means for various sub groups.

Each Name can have subgroups High Medium and low, and potentially others, and there are many observations for each name. For each name, i want to calculate the count of highs, mediums, lows; the sums of highs, mediums, and lows, the % of total distribution to high/medium/low; the weighted mean price of each rating category for each name, and the mode of type (most commonly occurring).

Hopefully not too confusing but i have attached a screenshot of sample data. Ultimately it is a few hundred thousand rows of data with observations in the rows and variables in columns. Any input is greatly appreciated!


texts: In general, when does TF-IDF reduce accuracy?

texts: I'm training a corpus consisting of 200000 reviews into positive and negative reviews using a Naive Bayes model, and I noticed that performing TF-IDF actually reduced the accuracy (while testing on test set of 50000 reviews) by about 2%. So I was wondering if TF-IDF has any underlying assumptions on the data or model that it works with, i.e. any cases where accuracy is reduced by the use of it?


texts: Using K-fold cross-validation in Keras on the data of my model

texts:
I would like to use K-fold cross-validation on my data of my model.

My codes in Keras is :

a = np.array(
[[283, 95, 72, 65],
[290, 100, 80, 72],
[120,170,130,122],
[100,230,110,200],
[300,100,200,500]]
)
X = a[:,0:2]
Y = a[:,3]

from sklearn.model_selection import KFold, cross_val_score
k_fold = KFold(n_splits=3)
model = models.Sequential()
model.add(Dense(12, input_shape=(3,)))
model.add(LeakyReLU())
model.summary()

cross_val_score(model,X,Y)

But, It makes this error:

  If no scoring is specified, the estimator passed should have a 'score' method. The estimator  does not.

And when I select a scoring parameter as:

cross_val_score(model,X,Y, scoring= 'accuracy')

It makes another error:

  TypeError: Cannot clone object '' (type ): it does not seem to be a scikit-learn estimator as it does not implement a 'get_params' methods.

How can I use K-fold cross-validation on this model?

Thank you


texts:
I am seeking theoretical suggestions more than else. This is my first actual practical work, and I am kinda stuck now that I have done a few things: I don't know in which direction to look in order to improve my model.

What kind of data I have

I have a dataset of information about a population. Each entry is a percentage of people belonging to a certain category (my features) over the whole number of inhabitants. I have 52 observation and 35 features, so a 52x35 input matrix. So my target is a 1D-vector of 52 elements, and these are percentages too.

       male  female  students  employees  freelancers  unemployed  catholics...
city1  ##%  ##%     ##%     ##%...
city2
city3
...
city52

and my target vector is the results of the winning candidate of the last presidential election in each city.

What I did

I thought that, having such a small dataset, it would be better to stick to simple models, staying away from polynomial models etc. in order to avoid overfitting. Some engineering work to do on the features, too.

Feature selection

I removed redundant information and I agglomerated variables, for example by summing up "irrelevant" percentage (compared to other values of the same conceptual categories) and filing them under new "other" features.

Model selection

As I said, I decided to keep it simple, start with some simple linear regression and then checking how other regression models performed compared to linreg.

Dataset was splitted into train $(30%)$ and test set. For each model I selected the related parameter which maximized the score. I then did Random Feature Elimination (with cross validation) for each model, using the parameters selected in the previous step, leaving just a few most relevant features (since I know I have few observations compared to the number of features), and took this as final output.

I focused mainly on linear and ridge regression, with the latter performing better. My best R2 is around 0.7xx.

I see a bit of $overfitting$, but no clear path in the residuals. The target points with the worst predictions actually make sense in the context of the work (they are all belonging to a certain category).

After this I tried with some trees and randomforest regressors, but they performed worse than ridge regression.

What to do next

Well, this is where I lose my path. I think I did the right things until now, considering the kind of data I have (but please feel free to correct me), but.. is that it? Am I done or do I have something more to do, or maybe do again something in a different way? When can I call my self satisfied?

This is my very first ML work so any suggestion is very welcome.


texts:
Can TF-IDF be used a sole method for Topic Modeling ? (I know there are better methods like LDA , LSA etc)
I just want to understand if TF-IDF alone can help us in Topic modeling . If yes , can someone explain how that simple framework works ?
I want to understand the application and capabilities of TF-IDF as a sole method for Topic Modeling. I could not find this anywhere else in the internet .


texts: After running K-means on 12 features, I get an array containing clusters for each row. What is the next step after this?

texts:
So I used the elbow method to identify the optimal number of clusters, i.e. 4 in this case. After running K-means on dataset with 12 features, I get an array with the cluster number each row belongs to, like this:

Feature1    Feature2  .... Feature12   Clusters
<data>      <data>         <data>      2
<data>      <data>         <data>      0
<data>      <data>         <data>      0
<data>      <data>         <data>      3
<data>      <data>         <data>      1     and so on.

What should be the next step now? Should I individually check the differences between each cluster to identify the similarities and differences? If yes, how exactly?


texts: SGDClassifier fit and partial_fit functions

texts: I wanted to know what is the correct way to train the SGDClassier model on new data observations? Should I use the fit function or the partial_fit function? How are they different?


texts:
Currently, to train a model, you need to collect a huge blob of data.

Are there feasible concepts of decentralized machine learning? Like, feed the model somehow from isolated data sources, or merge smaller models?


texts:
I have $3$ datasets I'd like to run a logistic regression on split into

$X_1, y_1$
$X_2, y_2$
$X_3, y_3$

How can I run a loop so that I can run an automated logistic regression using the X_train, X_test split function from SkLearn and in doing so printing 3 separate accuracy results for each dataset?

To run on one data set of X, y is as follows:

X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.25, shuffle=False)

logreg = LogisticRegression()
logreg.fit(X_train,y_train)
y_pred = logreg.predict(X_test)

print('Accuracy:',metrics.accuracy_score(y_test, y_pred))


texts: What are recommended methods for multi-task prediction?

texts:
Currently, we are working on a school project which is trying to predict the number of crimes in some area/neighbourhood.

There are 8 different categories for crimes and we've tried to find the correlation among those categories and now we only have 4 left.
Instead of building a model for each category, we want to predict these 4 categories simultaneously by some multi-output algorithm.

Our sample size is around 27,000 for 6 years (from 2011 to 2016, 4000+ for each year). We are going to use (maybe) cross-validation to build/test our model.

Would you please list 2-3 algorithms which already have fully or partially implemented library in Python (preferred) or R you would recommend to use with our dataset scale?
I only found scikit-learn with this algorithm. But it's for classification rather then prediction numbers.

This is a intro-level ML course project, the group is not very experienced in the field and the time is limited so we don't want to implement an algorithm from scratch.


texts: Specifying number of threads using XGBoost.train

texts:
When using the xgboost.train() function, all the threads are used. I would like to use a specific amount. Unfortunately, this function does not accept the parameters nthread nor n_jobs. How can I control the number of threads being used?

Thanks.

// Edit

It seems that I found a solution. In contrast with the method, how one provides the nthread (or n_jobs) parameter to XGBClassifier of XGBRegressor, by adding this parameter directly to the brackets as xgb.XGBRegressor(nthread=n) then as indicated on xgboost document (page 46), I added an additional parameter parameters["nthread"] = number_of_threads to the parameters (a dictionary) I am using. After testing with different numbers, the number of threads being used reported in htop was the same as the number_of_threads parameter provided. Can anyone confirm this to be the right method?


texts: Which latent variable model is better to find hidden variable?

texts:
Currently, I am exploring the concept of latent variable for regression type datasets.
I have gone through literature of few of the methods and models used in finding latent variable, like: EM algorithms, Partial least square regression, Latent semantic analysis, Mixed Effect models (linear-nonlinear), HMM, and there are many more!!
For Example:
volume DataFrame head is
    length     width      volume
0  1.395702  4.822958    40.821677
1  5.761620  9.912682   242.571731
2  3.444930  2.111199    18.904144
3  6.236642  7.609429   425.838818
4  7.270517  1.106117    39.883937

In the above mentioned dataset, height, type-of-object, units and other parameters are missing. As mentioned by people previously, it is difficult to regenerate them as such but there is a possibility of acknowledging them using latent variable.
Hence, I would like to ask which method or model will be more suited for such kind of dataset?
Thanks!!


texts: What would be the best way to impute data?

texts:
Other than just filling in with the mean of a feature, what other methods are there which can work well? I am trying to decide whether or not to use a denoising-autoencoder or just impute with the mean or any other method which can perform well enough which is quickly implementable.

Also is it necessary to make any assumptions about the underlying distribution? The denosing-autoencoder seems attractive here because you do not need to make any explicit assumptions.


texts: Predefined Neural Networks instead of fine tuning?

texts: I usually try to form my ANNs with classic fine-tuning approach but I recently learned that there are different "predefined" networks specially for certain tasks. Is there a good summary about these? Are they really perform better than home-made ones?


texts: How does the ULM subword tokenization avoid just splitting every word into single characters?

texts:
According to the paper:

The probability of a subword sequence X = (x_1, x_2...) is formulated as the product of the subword occurence probabilities p(x_i)

sum_i p(x_i) = 1

The most probable segmentation x* for the input W is then given by x* = argmax(X from S(W) ) P(X)

S(W) is set of different possible tokenizations of word W

Now imagine you have the word "is", with two possible tokenizations:

"i", "s"
"is"

$p(i)*p(s) > p(is)$ always! Because in your unigram corpus there will be words that contain "i" and/or "s" but not "is".
Therefore, no matter what other training criteria they use, according to their definition for finding the best tokenization, and by setting $p(x) = cnt(x) / sum_i cnt(x_i)$, "is" would always be tokenized to ("i", "s"). Same logic follows for all other words.


texts: Randomforest parameter tuning R(caret) and Python(scikit learn)?

texts:
Question: is it possible, or even necessary, to perform a cross-validation check to tune the parameters of a Python randomforest implementation (eg scikit learn) when training a new model, as can be done in R's caret?
Background R: When using R's caret's randomforest library, one can tune the parameters by performing a n-fold cross validation, e.g.
train_control <- trainControl(method = "repeatedcv",
                          number = 10,
                          repeats = 5,
                          verboseIter = TRUE,
                          allowParallel = TRUE,
                          summaryFunction = multiClassSummary)
rf1 <- train(Class ~ .,
         data = train_transformed,
         method = "rf",
         metric = "Accuracy",
         tuneGrid = my_grid1,
         trControl = train_control)

This outputs -
## Aggregating results
## Selecting tuning parameters
## Fitting mtry = 1 on full training set

or whatever mtry is the best for a dataset.
Background Python:  Using scikit learn, one can instantiate a randomforest regressor and can perform a cross-validation check on that regressor
RFRegModel = RandomForestRegressor(random_state=42)
cv = cross_validate(RFRegModel,X,y,cv=5,verbose=1)
print(cv['test_score'])

the difference here is that the cross validation doesn't appear to influence the tuning of the randomforest parameters, as I think it does in R. I think all that is happening here is that we are taking random subsamples of the training data set and testing the implementation of the
Question again: is it possible to train the model in python in a similar manner as is done in caret, i.e. by forcing an n-folds cross-validation
parameter tuning?  or, is this even necessary? Is the caret implementation overconstrained by it's methodolgy?


texts:
I have a huge collection of objects from which only a tiny fraction are in a class of interest. The collection is initially unlabelled, but labels can be added using an expensive operation (for example, by human).

Currently I use the simple generic machine learning strategy:

Use hand-crafted rules to select a smaller subset of objects (thus leaving out a fraction of interesting ones).
Label part of the smaller subset, and use these for training and choosing a classification algorithm and its parameters.
Classify the remaining objects in the smaller set (and also perhaps in the big set).

This has two drawbacks:

The labeller still needs to see a huge number of uninteresting objects, and therefore is able to label only a very small fraction of interesting ones.
The objects not in the smaller set are completely ignored in the learning phase, resulting in a loss of some information (the classification algorithm might not work well on this complement).

It seems that it would be better to use online learning: i.e., select the objects to show to the labeller based on the previous labels. But then it becomes no longer obvious that the result of classification algorithm retains the nice theoretical properties (i.e., statistical consistency).

Is there a general framework for active object detection which works either theoretically or practically (or both)? I could not get the complete picture from the Wikipedia article active learning.


texts:
I am pretty new to neural networks and I would appreciate some guidance... maybe books or articles on the following topic:

I am an airfoil designer. At fixed flow conditions, the pressure on the airfoil wall will be different depending on its geometry, e.g. thickness.  For different geometries, I have a large data set of pressure distributions along the airfoil (they can be treated as images). My idea is to use those "pictures" as training data of a neural network (input), in which the output will be the geometry of airfoil. Then, I would like to target a pressure on the wall (picture) and get a nice airfoil design...

Please let me know if you are aware of similar studies and if there is any book to start with!


texts: Integrate SageMaker notebook instances into development flow

texts:
I often get stuck by the calculation time when I work on a model locally. Even sampling quite heavily sometimes still let the running times very long. Ideally, these times must be of the order of the second. Otherwise it makes experimenting a bit tedious.

SageMaker provides with quite a handy way to fire up a familiar model development environment by mean of its notebook instances. To switch from your local to the online env, it recommends using github or its own git platform.
I guess the idea is just to pull your branch where you left it locally before you decided to switch.

Problem is, how do you do when your git platform does not integrate with AWS (company constraints) ?
Or is there another smooth way to do that ? (scp ?)

Just to be clear, when I talk about developing a model, I do not just talk about training and scoring a model. Most of the time I need to work and analyse in-depth the impact of a pre processing step.


texts: I remember reading an article comparing predicted probabilities of boosted, stacked encoders, random forests, and neural network algorithms. One important point I got from the article was that gbm pushes predicted probabilities towards 1 and zero. Does anyone know an algorithm that produces predicted probabilities that are close to 0.5?


texts: Multiple time series sequence prediction for multiple multivariate time series

texts:
My question is somehow similar to this question, but not satisfied with the answer. I have 100 samples, each sample consists of 6 time-series, (let's say [X1, X2, X3, X4, X5, Y]) of length 200. Each sample is independent of each other. I want to build a time-series model, which will take one sample (300, 5) as input and predict Y (300, 1) as output.

LSTM relates the various windows (in my case sample) while making the model, which is undesirable in my case. Though I tried I tried it and but could not achieve good results. This is the actual and predicted value of some of the samples. We observe that, prediction is happening in a narrow band between 35000 to 50000 for most of the cases. My intuition is LSTM has learnt only the latest sample passed and making prediction based on that.
Further more, I thought of using ARIMAX, but same problem will arise there as well. I tried CNN as well which was predicting better than LSTM but not close to actual values.

So my question is: what will be best strategy to train such model where it takes a 2D time series data as input, learn the temporal dependency of the X and Y and predict time series sequence of Y.


texts: Pandas dataframe groupby and then sum multi-columns sperately

texts:
I have a pandas dataframe as following:

df = pd.DataFrame({
    'Date': ['2017-1-1', '2017-1-1', '2017-1-2', '2017-1-2', '2017-1-3'],
    'Groups': ['one', 'one', 'one', 'two', 'two'],
    'data1': range(1, 6),
    'data2': range(10, 15})

Output:

    Date      Groups     data1    data2
0  2017-1-1    one       1        10
1  2017-1-1    one       2        11
2  2017-1-2    one       3        12
3  2017-1-2    two       4        13
4  2017-1-3    two       5        15

I would like the output to look like this:

    Date      Groups     sum of data1    sum of data2
0  2017-1-1    one       6                33
1  2017-1-2    two       9                28

I can groupby "Group" and agg.(sum) either data columns, but couldn't do 2 simultaneously. How can I do this?

Thanks.


texts: How does bagging help reduce the variance

texts: I learned that bagging helps reduce variance by averaging but I couldn't understand this. Can someone explain this intuitively?


texts: Space partitioning to estimate product weight

texts:
Looking for some vocabulary to help me refine my research so I can tackle this problem.

Here's an overview of the problem statement I'm working on.

At my company we manufacture various products, each product can have a number of different configurations, let's call each unique configuration of a product a PIN (this set up pretty much allows for unlimited customization).

Each product belongs to one of n discrete product classes.

We don't actually know the weights of any particular PIN, however we have years of shipping data and when things get shipped we have a weight associated with the entire shipment.

A shipment can contain several cartons and each carton can contain many different PINs (coming from any one of the n discrete product classes).

I have a nice, clean dataset which contains Shipment ID, Carton number, Product Class,  PIN, Number of PINs in Carton, Number of items in Carton, and Total Weight of Carton.

Using this data my job is to estimate the weight of any given PIN.

I don't have a robust enough vocabulary to start my search, and I am hoping someone can help me with either some keywords, or relevant articles that I can look into that might help me make progress against this.

Cheers


texts: Suggestions for building Localisation algorithm for 5 classes

texts: I'm currently working on a project for object detection using a very light CNN model , but till now I was able to classify all 5 classes with ~94% accuracy with 5 layered CNN using Keras and now I'm using sliding window algorithm to localise the 5 class objects ,but the accuracy of this algorithm is very poor i.e ~40 % and few bounding boxes are at random , before this I tried YOLO v3 tiny model and tensorflow model as well , but I need to reduce the time taken to predict each frame. can anyone suggest any approaches to attempt this problem


texts: Best practices to store Python machine learning models

texts:
What are the best practices to save, store, and share machine learning models?

In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ~100Mo large. Also, joblib can save one model to multiple files unless you set compress=1 (https://stackoverflow.com/questions/33497314/sklearn-dumping-model-using-joblib-dumps-multiple-files-which-one-is-the-corre).

But then, if you want to control access rights to models, and be able to use models from different machines, what's the best way to store them?

I have a few choices:

Store them as files, and then put them in a repository using Git LFS
Store them in an SQL database as binary files:

For instance in Postgresql https://wiki.postgresql.org/wiki/BinaryFilesInDB
This is also the method recommended by the SQL Server team:

https://docs.microsoft.com/en-us/sql/advanced-analytics/tutorials/walkthrough-build-and-save-the-model
https://microsoft.github.io/sql-ml-tutorials/python/rentalprediction/step/3.html
https://blogs.technet.microsoft.com/dataplatforminsider/2016/10/17/sql-server-as-a-machine-learning-model-management-system

HDFS


texts: Hyper-parameter search for time series LSTM-RNN using tensorflow

texts:
This question is about tuning the hyper-parameters of a LSTM-RNN on tensorflow. I would like to know about an approach to finding the best hyper-parameters for your RNN.

I am working on a dataset to predict monsoon patterns in central India.

My time-series plot looks like this.

My goal is to predict the value of rainfall of a day 30 days in advance.

Currently I have the following hyper-parameters.

num_days = 30 #number of days used to make next prediction
lstm_size=256
num_layers=2
keep_prob=0.8
batch_size = 50
init_learning_rate = 0.03
learning_rate_decay = 0.99
init_epoch = 5
max_epoch = 1000

Based on the time-series plot what is a good choice for my given hyper-parameters?


texts: Tensor Mean of values greater than a threshold

texts:
I have a tensor of shape = [a,b,c]

The tensor mean along dim=1 will give me an output of shape = [a,c]

My goal is to compute the mean of values along dim=1 greater than a threshold.
How is this possible in TensorFlow?

I am aware of tf.where() and tf.boolean_mask() functions but not sure exactly how to use them for this task.


texts: Regression performance with Feature Selection

texts:
I would like to ask you a theoretical question. In my project I am trying to get a better performance from my regression model by feature selection methods, especially with CatBoost feature importances.
I would like to ask:
1- I know the term "Garbage in Garbage out", so more features do not always mean better performance; moreover it decreases the performance. But can we get a better evaluation score like MSE, RMSE by eliminating less important features from the model? In my project It wasn't the case; MSE increased gradually while I am removing features step by step.
Should I expect a better model and higher predictive performance by eliminating unnecessary features? Or when should I expect that?
2- R2 is another metric, but I think it won't increase by eliminating a feature. In my opinion an extra feature may increase R2 value or do not effect at all. Am I right or an unnecessary feature may decrease R2?


texts: How effective is Moore Penrose for solving regression problems with overdetermined system of equations?

texts:
For regression problems with #Predictors > #observations, I recently read about Moore Penrose (pseudo inverse method) which solves the problem of non invertible matrix in OLS for regression problems.

How well does this 'generally' perform as compared to some of the other alternatives such as Ridge/Lasso, Partial least squares, Principle component regression?

Also, while datasets with predictors > observations, is there a method among the above listed that always performs better than the rest (purely in terms of prediction accuracy)?


texts: How can I fix regression model interpretation of feature?

texts:
I'm building a regression model to predict the values of a feature $Y$ given a set of other features $X_{1}, X_{2}, X_{3}..X_{n}$.

Onde of these other features, let's say $X_1$, is known to be inversely proportional to $Y$ based on domain's knowledge. The problem is my model is interpreting his coefficient as positive, letting it directly proportional to $Y$. I've tried plenty of different models to verify if I could get better interpretation, such as OLS, Linear Regression, and Logistic Regression, but every model I tried failed to interpret the $X_1$ coefficient.

What can I do to get a regression that better reflects the real-world behavior of this coefficient?


texts:
I've trained a few models to classify between two categories of text. Logistic regression was the best. Now how can i test it on unseen data?
I tried this:

def train_model():
 classifier.fit(feature_vector_train, label)
 predictions = classifier.predict(feature_vector_valid)
 joblib.dump(classifier, url+name)
...

load_model =joblib.load('my_model.pkl)
result = load_model.score('testx')

It tells me i need a y input. However, if it's new i don't have the label. WHat am i missing?


texts: Predict event likelihood given time of other events

texts:
On a daily basis there is a sequence of events.

Each event may or may not occur on a given day.

Given the times of the preceding events for the current day I want to be able to say whether there is a high likelihood of an event occurring right now.

There are around 20 events that may occur in a day.  I have captured only a small number of days worth of observations (say 20-40) to start but will be building up as time goes by.

What kind of model should I use to predict this?

Here is picture of the data to help clarify:

UPDATE/CLARIFICATION

As each day time passes during the day I always want to know if each event is likely to happen at that moment of time. By the nature of the data it is possible that on a given day only a single event happens, but when it does it will be likely to happen at around the same time it has happened in the past.


texts: How to anonymize (de-identify) data in Python?

texts:
I have tried a simple algorithm to anonymize the data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values of strings and integers. The data sample is available here

This is what i have tried.

import pandas as pd
import uuid as u
import datetime as dt
 # generate a pseudo-identifier sequesnce using python random number generator library uudi.

    def uudi_generator(length):

    uudi_list= list()
    i=0
    while i < length:
        uudi_list.append(u.uuid4())
    i+=1
    return uudi_list

#import original originaL dataset
dataset = pd.read_csv('bankcredit-data.csv')

# pseudo identifier
sLength = len(dataset['housing'])
dataset.insert(0, 'uuid', pd.Series(uudi_generator(sLength), index=dataset.index))

# Transaction record attached to the original
dataset.insert(0, 'transaction_date', pd.Series([dt.datetime.now]*sLength, index=dataset.index))

#transcation record is attached to originaL data file
dataset.to_csv('bankcredit-data.csv')

#delete identifiabLe record from dataset
del dataset['firstnamme']
del dataset['lastname']

# export  de-identified dataset as csv to be shared with the user
dataset.to_csv('deidentified-data.csv')


texts: Statistic that is responsive to changes in time series, yet not too volatile

texts:
Let's say I have a fairly volatile time series $X_t$ - it doesn't have any reason to show an upward / downward trend, but it does show drops and spikes from time to time. It can also change level (e.g. for a few weeks it will be at one level, but it will quickly drop down to a new level a few % points down and stay there).
I'd like to have a statistic (basically a time series Y_t) that signals a change in $X_t$ quickly. It should also converge to the same (or close to the same) level as $X_t$ with as little delay as possible. Finally, $Y_t$ should have as little volatility as possible (I'm aware there's a tradeoff between low volatility and responsiveness).
I've tried the rolling mean - if there's a short duration (say 1 period) drop in $X_t$, then the rolling mean reacts quickly to the drop, but shows a delay in reacting to the recovery in $X_t$ (i.e. when $X_t$ is restored to normal values). The delay is the same as the window size minus the drop duration.
At this stage, I'm considering two statistics. The exponential moving average with a window of $5$:
$$E_t=\frac{\sum_{i=0}^t(1-\alpha)^iX_{t-i}}{\sum_{i=0}^t(1-\alpha)^i}$$
where $$\alpha=\frac{2}{\text{window}+1}$$
The other statistic I'm considering is a combination of EMA and median. Let $M_t$ denote the rolling median with a window of $9$. Then
$$C_t=aE_t+(1-a)M_t$$
To test their response to a drop, I did a quick simulation as follows:
a = pd.Series(np.hstack((np.ones(10)*0.81, np.ones(4)*0.77,
              np.ones(4)*0.72, np.ones(5)*0.77, np.ones(10)*0.81)))
x = 0.5
b = a.ewm(span=5).mean()
c = x*a.ewm(span=4).mean() + (1-x)*a.rolling(7).median()

figure(num=None, figsize=(27, 9))
plt.plot(a, marker="o")
plt.plot(b, marker=".")
plt.plot(c, marker=".")

By the time $C_t$ converges to the same values as $X_t$, the difference between $C_t$ and $E_t$ is really small (less than a percentage point).
$C_t$ has the feature that it won't drop as sharply as $E_t$ in response to a drop in $X_t$, but after a few periods it will abruptly drop down to the level of $X_t$. Similarly when $X_t$ recovers, $C_t$ initially shows a slower rise than $E_t$, but then suddenly shoots up, overtaking $E_t$ and catching up with $X_t$.
Now intuitively it seems that overall the EMA $E_t$ would be a better statistic, but I'm not sure. Is there any concrete, rigorous justification for the superiority of any one statistic over the other? For example, is it possible to show that maybe $C_t$ reacts similarly to two different scenarios, and is therefore unsuitable to differentiate between those scenarios?
Finally, given the requirements of quick response, quick convergence and low volatility, is there any other statistic I may have missed that's better than these two?


texts: Passing reduced/different feature data to LimeTabularExplainer compared to the original model

texts:
I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not interpretable.
I want to pass a subset of oringal features (which are interpretable) to the Lime explainer, so that my resultant explainations are also interpretable.
In sections 3.1 and 3.3 of original paper, the authors talk about this https://arxiv.org/abs/1602.04938
rf = sklearn.ensemble.RandomForestClassifier(n_estimators=500)
rf.fit(train, labels_train)

explainer = lime.lime_tabular.LimeTabularExplainer(train,
                                                   feature_names=feature_names,
                                                   class_names=target_names,
                                                   discretize_continuous=True)
exp = explainer.explain_instance(test[i], rf.predict_proba, num_features=2, top_labels=1)

Suppose I need to pass test[i][:2], only the first two features to the surrogate model. Is there a way for this in LIME or even in SHAP?


texts: The actual results and results from pickle files are not matching in pandas for DBSCAN clustering

texts:
I've built a DBSCAN clustering model. The output result and the result after using the pickle files are not matching.

Based on HD and MC column, I am clustering WT column.

data = HD,MC
Target = WT

Below, for 1st record the cluster is 0.

But after running it from 'pkl' file, it is showing predicted result as [-1]

Dataframe:

      HD         MC             WT         Cluster
      200        Other          4.5        0
      150        Pep            5.6        0
      100        Pla            35         -1
      50         Same           15         0

Code:

 le = preprocessing.LabelEncoder()
 df['MC encoded'] = le.fit_transform(df['MC'])

 col_1 = ['HD','MC encoded']
 data = df[col_1]
 col_2 = ['WT']
 target = df[col_2]
 data = data.fillna(value=0)

 model = DBSCAN(eps=1, min_samples=20).fit(data)
 outliers_df = pd.DataFrame(data)
 print(Counter(model.labels_))

 x = model.fit_predict(target)
 print(Counter(x))

Result:

  Counter({-1: 604, 0: 142, 1: 83, 9: 36, 2: 27, 7: 26, 10: 26, 8: 24, 4: 23, 5: 23, 3: 22, 11: 21, 6: 20, 12: 20, 13: 20})
  Counter({0: 1093, -1: 24})

Code:

  df["Cluster"] = x

  filename1 = '/model.pkl'
  model_df = open(filename1, 'wb')
  pickle.dump(model,model_df)
  model_df.close()

  output = open('/MC.pkl', 'wb')
  pickle.dump(le, output)
  output.close()

  with open('model.pkl', 'rb') as file:
     pickle_model = pickle.load(file)

  pkl_file = open('MC.pkl', 'rb')
  le_mc = pickle.load(pkl_file)
  pkl_file.close()

 def testing(HD,MC,WT):
     test = {'HD':[HD],'MC':[MC], 'WT':[WT]}
     test = pd.DataFrame(test)
     test['MC_encoded'] = le_mc.transform(test['MC'])
     pred_val = pickle_model.fit_predict(test[['HD','MC_encoded']])
     print(pred_val)
     return(pred_val)

      pred_val = testing(200,'Other',4.5)

Result:

    [-1]


texts:
I am an beginner in text generation and and deep-learing but I like to get in touch with it. Currently I am learing about LSTM networks and VAEs for text generation. I would like to read a sequence at once and output another sequence. What I learned from this post is that the input shape should be 3D. Since my text data is 2D I want to reshape it. But this is now the confusing part. As I have training data with the shape of (nb_sequences, max_seq_lenth) = (274, 95). As the post pointed out, it would be important to reshape the data to a format of input = (nb_sequence, nb_timestep, nb_feature) with nb_sequence being the number of sequences (274), nb_timestep the length of the sequence (95) and nb_feature the number of features describing a sequence at a timestep, right?
I have two questions here:
1). How to reshape properly? My vocab size is 359 which I imagined (obviously wrong) to be the number of the features. --> input = (274, 95, 359). This is not possible since an array of 274*95 = 26030 can't be reshaped into (274, 95, 359).
If I take the number of diffrent words my sequences are containing, wouldn't it lead to some erros since sentences do not consist out of the same words naturally? Meaning that I would have a variable feature number with each sequence?
2). How would one realize the problem I am facing? I read about one-hot encoding the sequences. This might solve the problem I have regarding question 1), but as far as I understand this would lead me to a quite inefficient way of predicting sequences as I want to handle more data. Is there any suggestion how to do it in a effient way?


texts: EfficientNet function composition or Hadamard

texts:
In the page 3 of the paper of EfficientNet, there is a equation
$$\mathcal{N} = \bigodot_{i=1...s} \mathcal{F}_{i}^{L_i} \big(X_{\langle H_i, W_i, C_i \rangle}\big)$$
where $\mathcal{N}$ is the conv net and each $\mathcal{F}_i^{L_i}$ is the $i$th-stage layer operator that has length $L_i$.
What I don't understand is, what is this $\odot$ in this equation? Does the author refer to the Hadamard product or does he refer to the function composition? He previously mentioned that $\mathcal{N} = \mathcal{F}_k \odot ... \odot \mathcal{F}_1 (X_1)$, where k is the depth of the net. So I thought it means that $\odot$ is just function composition. But EfficientNet has skip connection. In the keras implementation it uses layers.merge.Multiply() so it can also means that the input data $X$ is multiplied with the transformed $\mathcal{F}(X)$, and $\odot$ maybe means the Hadamard product.
Does anyone knows the answer? Thanks.


texts:
I am using the HDBSCAN algorithm so as to perform unsupervised clustering and detect outliers. Based on the documentation there are two outputs from the clustering process that can give insight on which points are outliers.

The GLOSH outlier detection algorithm that gives a degree of certainty of whether a point is an outlier or not.
The HDBSCAN labels that if an element in not part of a cluster is considered as noise and has the corresponding label.

I have been working on some data, and I have noticed that these two approaches do not give the same results. That means that there are points in my dataset that are labeled as not part of a cluster but are not detected by the GLOSH outlier algorithm. Should I consider the union of these two approaches as my outliers or I am missing something in their interpretation?


texts: What kind of analysis can I do in this dataset?

texts:
I have a dataset of clients (their city, name, age, gender, number of children) and another dataset about the products that they have bought. i have been asked to do:

extract knowledge about client profiles.

I don't know what knowledge should I extract.

we are studying clustering and classification so they should connect to the question. what i thought about is to make clusters of clients. but i don't know what criterisas should I depend on.

should I just use a clustering algorithm like k means and let it give me the clusters ?

any suggestion would be appreciated

Update

These are the dataset that I have:

Card: CardID, City, Region, PostalCode, CardStartDate, Gender, DateOfBirth, MaritalStatus, HasChildren, NumChildren, YoungestChild
Item: ItemCode, ItemDescription, CategoryCode, SubCategoryCode, BrandCode, UpmarketFlag
Transaction: Store, Date, Time, TransactionID, CardID, PaymentMethod
Category: CategoryCode, CategDescription
Transaction_Item: Store, Date, Time, TransactionID, ItemNumber, ItemCode, Amount
SubCategory: SubCategoryCode, SubCategDescrip

The teacher said that we should categories the clients and then indicate which products best suit for each customer.


texts:
I recently discovered this article that details moving audio classification into the image domain and I was intrigued enough to dive in and take a closer look.
A good portion of the article details how to pre-process a dataset prior to training. I've already spent a couple of evenings working on this and have a small application that scans through a directory and produces the spectrograms. In the article audio is standardised to summed mono and converted.
The issue is since I'm so new to this I'm unsure that I'm producing these spectograms in a state where they are really fit for use. Hence the question.
I've run my tool over my input data already, trained an MS.NET ResNet model over a tiny dataset I've prepped and all things considered, it actually got most of tests I ran on it correctly. It's clearly not ready but the fact it got as many right as it did has already surprised me so I know I'm not a million miles away.
The code in the article is all Python so I've had to find .NET replacements and get as close to the articles methodology as I can get.
His code for generating spectrograms is:
melspectogram_transform = torchaudio.transforms.MelSpectrogram(
  sample_rate=fixed_sample_rate, n_mels=128)
melspectogram_db_transform = torchaudio.transforms.AmplitudeToDB()

melspectogram = melspectogram_transform(audio_mono)
plt.figure()
plt.imshow(melspectogram.squeeze().numpy(), cmap='hot')

melspectogram_db=melspectogram_db_transform(melspectogram)
plt.figure()
plt.imshow(melspectogram_db.squeeze().numpy(), cmap='hot')

Which produces graphs like this:

After some experimentation I arrived at this code:
(double[] audio, int sampleRate) = ReadWav(_sourceFile);
var sg = new SpectrogramGenerator(sampleRate, fftSize: 8192, stepSize: 500, maxFreq: 22050, fixedWidth: 500);
sg.SetColormap(Colormap.Magma);
sg.Add(audio);

// Create a Mel Spectrogram
var bmp = sg.GetBitmapMel(250, intensity:5, dB: true);

bmp.Save(Path.Combine(_targetDirectory, $"{_fileName}.png"), ImageFormat.Png);

Which produces graphs like this:

Which to my untrained eye looks like it should be good to go and, from my initial testing, looks like given a decent sized dataset, should work.
The library I'm using is this one swharden/Spectrogram
A potential issue is that whilst I am pre-processing the training audio files are not standardised in length (hence the fixed width parameter). This seems to be working as some of my graphs look like this (padded):

When I was testing I tested quite a few files where this was the case and the identification worked so I don't think this is a major problem but again, I'm very new to this and most likely not in a position to really critique and assess it as well as the folks here.
Is there anything I can or should do to produce better input images or does this already seem like a reasonable approach?
I clearly need a much larger data set but want to ensure I have a fighting chance of success before leaving my machine number crunching for days!
UPDATE
I have now built a working pipeline to standardise all audio files to 16bit 44.1Khz mono files either clipped or padded to 4 seconds. These are then converted to mels and look like this:

I'm sure there is a significant problem with these mels as it looks like the image is rolling vertically.
Thanks for reading.


texts:
I am working on Sentiment Analysis model.
The dataset I have has three labels POSITIVE , NEGATIVE and NEUTRAL

But the problem is the data is not equal for labels.
Say out of 100K , 75 K are neutral, 15K positive and 10K negative.

I wanted to know whether it is necessary to chose equal distribution of labels while training or I can go ahead with unequal data and if so till what extent.
Are there any ways to deal with such problem.


texts: Passing a word cloud choice to a corpus viewer

texts:
Can I pass a selected word (via a click) in a word cloud to the RedExp filter in a corpus viewer?
Is:

Desired:

ron_despain@hotmail.com...I'll join up next...Thanks for any help


texts: How to perform node classification using Graph Neural Networks

texts:
I'm am trying to perform node classification using graph neural network methods. My initial plan was to convert my graphs to adjacency matrices and train my network on that, with the node features being my target.

However, my graphs all have a different number of nodes, so I believe adjacency matrices will not work. I then found information about node embeddings and applications in biology (see here). It infers here that by embedding your nodes it no longer matters about graph size.

What I am wondering, is if using embeddings is actually the solution to my problem? Then, how, practically, do I build a Graph Neural Network out of this method? The link (in the slides for part 2) still says I need to train my network to learn the embeddings - but now I have my original problem, how can I train the network on many different sized graphs?

Doing node classification on small, differently sized graphs seems to be somewhat commonly done. What I am slightly confused by is the step from theory to practise.

Could anyone please offer any advice? Or links to more material that could solve some of my queries?


texts:
I am doing a course on Udemy on which the instructor applied OLS (Ordinary least square) on a housing dataset. The curve he got was linear,with parameters [10^5,239].
Now When I tried to repeat the same using gradient descent as described by AndrewNg on youtube my algorithm didn't converge.
When I used other data gradient descent was converging,which means my code was correct.
So is it possible that same data do not converge on GD but converge on OLS?


texts: How to make sense of the input and output for a speech-generation model like WaveNet?

texts:
I am currently studying this model speech generation known as WaveNet model by Google. https://arxiv.org/pdf/1609.03499.pdf using the linked original paper and this implementation.
I find the model very confusing in the input it takes and the output it generates, and some of the layer dimensions didn't seem to match based on what I understood from the WaveNet paper, or am I misunderstanding something?

What is the input to the WaveNet, isn't this a mel-spectrum input and not just 1  floating point value for raw audio? E.g. the input kernel layer shows as shaped 1x1x128. Isn't the input to the input_convolution layer the mel-spectrum frames, which are 80 float values * 10,000 max_decoder_steps, so the in_channels for this conv1d layer should be 80 instead of 1?

inference/input_convolution/kernel:0 (float32_ref 1x1x128) [128, bytes: 512]

Is there reason for upsampling stride values to be [11, 25], like are the specific numbers 11 and 25 special or relevant in affecting other shapes/dimensions?

inference/ConvTranspose1D_layer_0/kernel:0 (float32_ref 1x11x80x80) [70400, bytes: 281600]
inference/ConvTranspose1D_layer_1/kernel:0 (float32_ref 1x25x80x80) [160000, bytes: 640000]

Why is the input-channels in  residual_block_causal_conv 128 and residual_block_cin_conv 80? What exactly is their inputs? (e.g. is it mel-spectrum or just a raw floating point value?) Is the wavenet-vocoder generating just 1 float value per 1 input mel-spectrum frame of 80 floats?

inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0/kernel:0 (float32_ref 3x128x256) [98304, bytes: 393216]
inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/kernel:0 (float32_ref 1x80x256) [20480, bytes: 81920]

I was able to print the whole WaveNet network using the print(tf.trainable_variables()), but the model still seems very confusing.


texts: How to compute denominator in Naive Bayes?

texts:
Suppose we have class C_k and input feature vector x in dataset

How to calculate probability p(x)?


texts: How can I test my trained model on a completely new dataset?

texts:
Preface

I have an annotated text dataset on hate speech. Simply put, the dataset consists of a column called text which includes a piece of text, and a column called label which can be either 0 (non-hateful) or 1 (hateful).  Let's call this dataset Dataset A.

I am using this dataset to train a very simple classifier using Logistic Regression. Code:

dataset_1_df = pd.read_csv(d_1_path, sep='\t')

text = dataset_1_df.text
X = text
y = dataset_1_df['label'].astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, test_size=0.1)

clf = Pipeline([
    ('vect', CountVectorizer(max_features=10000, ngram_range=(1, 2))),
    ('tfidf', TfidfTransformer(norm='l2')),
    ('clf', LogisticRegression()),
])

clf = clf.fit(X_train, y_train)
y_preds = clf.predict(X_test)

This classifier works fine. I can produce a classification report like this:

report = classification_report(y_test, y_preds)
print(report)

Which outputs:

              precision    recall  f1-score   support

           0       0.87      0.70      0.78       410
           1       0.94      0.98      0.96      2069

    accuracy                           0.93      2479
   macro avg       0.91      0.84      0.87      2479
weighted avg       0.93      0.93      0.93      2479

And I can now make predictions like this:

print(clf.predict(['I hate you']))
print(clf.predict_proba(['I hate you']))

Which outputs:

[1]
[[0.0749381 0.9250619]]

My Question

I have another dataset, let's call this Dataset B, independent to Dataset A. Dataset B is also an annotated text dataset on hate speech with the same columns.

What I want to do is to test the model I trained using Dataset A on Dataset B. I.e., I want to produce a classification report on my model's performance on Dataset B. Can I do that? If yes, how?

Clarification: I do not want to retrain my model using Dataset B. I want to see how good its performance is on the new dataset.

Similar Questions

There is this question that looks similar based on the title but does not ask what I'm asking.

There's is also this question which asks how to make predictions. However, I know how to make predictions, what I want to do is to test my trained model on a totally new dataset.


texts: What Does the Normalization Factor Mean in the AdaBoost Algorithm?

texts:
I am studying the AdaBoost algorithm. The update rule for a weak hypothesis is:

$Dt+1(i) = Dt(i)exp(−αtyiht(xi))/zt $

where $zt$ is a normalization factor chosen so that $Dt+1$ is a distribution.

What does the 'normalization factor' mean? Could I have an explanation with an example, please?


texts: Which is the best Machine learning technique for this Load forecasting problem?

texts:
I am trying to use Machine Learning to predict the load of a residence at any point in time for a whole year. I have past data pertaining to that house. So I have the training data and I need the algorithm to predict future loads of the house.

Based on my knowledge, I have found the "supervised" machine learning technique to be the one I must adapt. I figured this out since I have labelled test data, I have a prediction requirement and I can get feedback for my prediction (cross-checking with the actual value). Am I correct here?

Also, I read online that "Unsupervised" learning is to be used at places where we need to find "Hidden data structure". I assume it means pattern. If so, what is the difference between the unsupervised and supervised learning in my case. Both of them will give me a prediction about the future load pertaining to that house at any point in time.

My background

I am doing my Masters in EE (Power systems). I am new to Machine Learning as well.


texts: How do I use LIME in R to explain a Model created in KERAS?

texts:
I have created a model in KERAS R, using a data table, with the following script:

data <- 'csv filename'
data %<>% mutate_if(is.factor, as.numeric)
data <- data[, c(1:35,30)]
data <- data[ -c(31) ]
data <- as.matrix(data)

set.seed(1234)

ind <- sample(2, nrow(data), replace = T, prob = c(.7, .3))
training <- data[ind==1,6:29]
test <- data[ind==2,6:29]

trainingtarget <- data[ind==1,31]
testtarget <- data[ind==2,31]

m <- colMeans(training)
s <- apply(training, 2, sd)
training <- scale(training, center = m, scale = s)
test <- scale(test, center = m, scale = s)

model <- keras_model_sequential()

model %>% layer_dense(units = 10, activation = 'relu', input_shape = c(24)) %>% layer_dense(units = 5, activation = 'relu') %>% layer_dense (units = 1)
model %>% compile(loss = 'mse', optimizer = 'rmsprop', metrics = 'mae')

mymodel <- model %>% fit(training, trainingtarget, epochs = 100, batch_size = 32, validation_spit = 0.2)

I use this model to make predictions on another table, called 'newtest', using the following command:

y_data_pred=predict_classes(model, newdata)
glimpse(y_data_pred)

I have installed LIME into R Studio, however I am struggling to get the correct LIME script syntax to explain my model predictions, to plot these in a chart, etc.

Any help would be greatly appreciated.


texts: Righteous data analysis approach for this industrial process monitoring

texts:
I am a R&D operator which is in charge to conduct an evaluation study for implementing a monitoring system for our process of production. I generalized what the software must do but I don't know which is the approach/algorithm to perform what I designed.
Here's the process description:
we produce brake hoses. The hoses are produced in a continuous process. It starts from (1) extruding a plastic tube which get wrapped in reels of about 1000mt. Then (2) these 1000mt of extruded tube get braided with a textile or steel wire. The last phase (3) is coating the braided tube from phase (2) with an extrusor with a "T" head. At the end we get batches of brake hoses in reels of about 1000mt.
The variables in this process are detected by sensors and in order are:
(1)-extruder speed - temperature - vacuum pressure
(1)-diameter of the extruded tube
(1) human operator
(2) speed of the braiding machines
(2) temperature and humidity of the braiding room
(2) human operator of the braiding machine
(3) coating extruder speed - temperature - vacuum pressure
(3) diameter of the coated hose (which the brake hose = the process OUTPUT)
(3) human operator
(3) tests performed by the laboratory onto the brake hose
This process is in the time-domain as the three phases are in chronological order and the process is a linear process (it starts from an extrusion of the inner tube and ends with its coating).
The operators of the laboratory will perform tests on the finished product (the process output) which is identified by a univoque batch code. The software must correlate these data and the diameter detected in the phase (3) to the sensors data of the process of that batch of hose.
Everytime we input data into the system the software must learn to recognize patterns and find correlations among the variables.
For example: last year we had a leakage problem on a hose which wasn't determined and disappeared after one month without significant actions made onto the process. If the software had already been implemented we expected a result from its analysis like this (simplifying in a spoken language):
-"I detected that the batch n°3 had a leakage in the test of "pressure holding" same as batch n°2 and batch n°1. In all these batches I detected the coexistence of these three factors:
1) the presence of the operator John Doe which stopped the braiding machine 2 times
2) the humidity of the braiding room was 2% over the average level of the past 3 months
3) there plastic grains for the extrusion of the inner tube (phase 1) has dried only for 20 minutes instead of 40.

Of course, the software next time the process shows the same conditions listed above warns us that the same defect might happen.

Which kind of algorithm (supervised-reinforced-unsupervised) is best suited for this task and which kind of approach (decision three-neural network?
As I'm making a study to be submitted to the top management I would like to be well prepared and also don't depend to what a software house says it is the best solution because they tend to be "smart" when somebody is ignorant on an argument.
Thank you
Alessio Cazzaniga


texts: Machine Learning based Multivariate Time Series Prediction - How to create supervised data format

texts:
Q1:

I have a multivariate time series dataset. For each timestep, there are 11 features and 1 output. I am going to use supervised ML to predect the output. I understand that in univariate cases, if I am going to use the past 3 days to predict the t-th day, the dataset will be formatted as

x(t-3) | x(t-2) | x(t-1) | x(t)

, where x(t) is the output to predict.

How should I format the dataset when it is a multivariate problem?

I saw that in some kernels, the problem is formatted as

x12(t-3) | x12(t-2) | x12(t-1) | x1(t), x2(t), ..., x12(t)

, where x12(t) is the output to predict.

In this case, variables x1 to x11 for the past 3 days are ignored.

However, these variables may be important in my case. Can I format the problem into

x1(t-3),...,x12(t-3) | x1(t-2), ..., x12(t-2) | x1(t-1),..., x12(t-1) | x1(t), x2(t), ..., x12(t)

?

(some of the features are just day, month, day of week, etc. created from the datetime index)

Q2:

With only 11 features, is it necessary to conduct feature selection?


texts: If I wanted to determine the parameters of a deterministic algorithm that optimise some performance measure, there are any number of well-known techniques I could try. My problem concerns an algorithm whose behaviour is stochastic. Mathematically, the aim is to maximise $\mathbb{E}f(\mathbf{x})$, where for any $\mathbf{x}$ a random value of $f(\mathbf{x})$ is calculable but $\mathbb{E}f(\mathbf{x})$ is not (except by averaging empirical values of $f(\mathbf{x})$). My efforts with gradient descent and an EA have failed, and I think it's because $f$ is too noisy. (Sampling $f$ many times at the same $\mathbf{x}$ would give a less noisy average, but is too computationally expensive.) Are there any algorithms designed for this situation?


texts: Neural networks (keras): predicting a periodic output array

texts:
I have a non-linear multiple regression problem where my target arrays have a length of 256 (for a single sample). These arrays have a periodic structure, since it's actually composed of 16 semi-independent sigmoid-like arrays of equal lengths, whose exact form depend on the inputs (see figure). Although I seem to get good results with some basic RELU activations and a few hidden layers already, I was wondering if e.g. a sine activation function might be appropriate somewhere? How might I implement this in Keras?

EDIT: ok, adding a sine activation was trivial:

def sin(x):
  return K.sin(x)

However, it doesn't seem to increase performance. I guess because the outputs are not a periodic function of the inputs, but because the whole output array is sort of a static periodic array?


texts: What is the best way to encode features when clustering data?

texts:
I have a dataset with numerical and categorical features. I am trying to run a k-means algorithm to find clusters of data.

What is the best way to encode categorical features?

I have been doing one hot encoding but I guess that it can be easily improved.


texts: Node classification/regression with Node2Vec

texts:
My question is about the correct use of Node2Vec for learning features for node classification/regression.
Suppose we have n labeled networks (i.e., networks for which the classes/continuous values of the nodes are known) and m unlabeled networks.
Both types of networks are isolated (i.e., the nodes of a given n or m network do not interact with the nodes of another n or m network).
The goal is to predict the correct classes/continuous values for the nodes in the unlabeled networks.
The approach I was thinking of using is:

Use a single Node2Vec run to learn the embedding vectors for all of the nodes in all networks, both labeled and unlabeled.
Train a supervised machine learning algorithm using the output of step 1 and the known classes/continuous values.
Use the trained machine learning algorithm from step 2 to predict the classes/continuous values of all of the nodes in the unlabeled networks.

Is this the correct approach?
*A similar scenario is described in the documentation of stellargraph, although for a single labeled network.


texts: How to improve results in classification problems (SVM, Logistic Regression and MultiNaive Bayes)?

texts:
I am new on Machine Learning and building models but a lot of tutorials has given me the chance to learn more about this topic.
I am trying to build a predictive model for detecting fake news.
The percentage of data with labels 1 e 0 is the following:
       T
0    2015
1     798

It is not well balanced, unfortunately, as you can see.
I split the dataset as follows:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, stratify=y)

i.e. 70% train and 30% test.
I hope it makes sense, though I have unbalanced classes.
Then, after cleaning text by removing stopwords and punctuation (should I have done something else?), I ran different models, specifically MultiNaive Bayes, SVM and Logistic Regression, getting the following results:
MNB : 84%

  precision    recall  f1-score   support

           0       0.88      0.90      0.89       476
           1       0.45      0.40      0.42        95

    accuracy                           0.82       571
   macro avg       0.66      0.65      0.66       571
weighted avg       0.81      0.82      0.81       571

SVM:  Accuracy: 0.8336252189141856
Precision: 0.5
Recall: 0.2736842105263158 (Terrible results!)
Logistic regression: 0.8546409807355516
All the tutorial show that the steps for building a good model when you have some text, are removing stopwords and punctuation and extra words.
I have done all these things, but probably there will be something that I could do more to improve the results.
I read that, in general, who gets results above 99% met problems like overfitting: however, I would really have liked to get a 92% (at least).
What do you think? How could I improve further the models? Do you think that having unbalanced classes could have affected the results?
Any suggestions would be greatly appreciated it.


texts:
How do I go about applying cost sensitive classifier in Orange?

I need to get a  confusion matrix and predictions for a specified ratio of FP and FN. ROC curve widget lets me choose that ratio but where do I go from there?

I can't even get a reading on a threshold where performance line and ROC intersect. Also, I don't quite understand what prior target class probability is for. Model should be able to derive prior probabilities from the data set. Is it used for oversampling rare events?


texts: How can I increase my accuracy avoiding overfitting? CNN-Keras-VGG16

texts:
As I asked in this question: Why are my predictions bad, if my accuracy in train is roughly 100% (Keras CNN) , my problem was Overfitting, so, I reduce the number of layers, and now I have this model:

vgg16_model = VGG16(weights="imagenet", include_top=True)

# (2) remove the top layer
base_model = Model(input=vgg16_model.input,
                   output=vgg16_model.get_layer("block5_pool").output)

# (3) attach a new top layer
base_out = base_model.output
base_out = Reshape((25088,))(base_out)
# output layer: (None, 5)
top_preds = Dense(1, activation="sigmoid")(base_out)

# (4) freeze weights until the last but one convolution layer (block4_pool)
for layer in base_model.layers[0:14]:
    layer.trainable = False

# (5) create new hybrid model
model = Model(input=base_model.input, output=top_preds)

# (6) compile and train the model
sgd = SGD(lr=1e-4, momentum=0.9)
model.compile(optimizer=sgd, loss="binary_crossentropy", metrics=["accuracy"])

But, when I predict some images, the class 0 accuracy is roughly 96%, but the accuracy of class 1 is roughly 58%. So how can I increase the accuracy without overfitting.

I've trained my model with 700 images each class and to test I have 50 images each class.


texts: Genetic algorithm - Feature selection packages in Python

texts:
Can you share some packages in Python which are implemented that I can use for selecting features based on a genetic algorithm?

I did refer to this AUTO-ML post and found out that it is useful but doesn't really do a feature selection and return the best subset.

I am already aware of this sklearn-genetic, can you share if you know of any other genetic algorithm package?


texts: chart x-axis spacing terminology question

texts:
In the following hand made charts I show some value for years. In the first chart I've evenly spaced each year. On the second chart I've spaced them relativelly to their actual year value within time (i.e 2016 is closer to 2017 than 2010).

Is there a terminology for the spacing of the second chart? Imagine building a software which would have a toggle control to switch the view from A to B. How would you call it?


texts: Pre trained neural networks to recognize certain things

texts:
How do you approach the problem or is not classification? For example, I would like to recognize if a face has makeup. But, in order to do this, it would first need to be able to recognize if it is a face. Since a very small amount of pictures are faces, I can’t possibly feed the no_face category with everything else. Also, is it better to recognize if it a face first and then use a second neural network to recognize if it has makeup?

Are there pretrained models I can download for common things like faces, animals, etc.

PS: By facial recognition, I would like recognize whether it is a face or not, not who the face belongs to.


texts: Why is my accuracy 0 on the first epoch in continuous training on MNIST

texts:
I'm trying an experiment where I first train my model on MNIST labels [0-4], and then I freeze the first conv layers and continue training my model on labels [5-9].
When I dont re-initialize my Fully connected layer, the network seems to be stuck at 0 accuracy after 1 first epoch and then starts to learn efficiently.
What is the reason for this, why is the network stuck at 0 after 1 whole epoch ? on the other hand when I re-initialize the network it starts learning efficiently from the first epoch.


texts: Sementic segmentation data and model compile in Keras

texts:
I have multi-label data for semantic segmentation. For semantic segmentation for one class I get a high accuracy but I can't do it for multi-class segmentation.

I have 6 class labels so my Y train matrix is equal [78,480,480,6] ('channel last'), where 78 - number of images 480,480 -image size, 6-number of masks and X train matrix [78, 480, 480, 1]

The last lines of my CNN model:

l = Conv2D(filters=64, kernel_size=(1,1), activation='relu')(l)
output_layer = Conv2D(filters=6, kernel_size=(1,1), activation='sigmoid')(l)

model = Model(input_layer, output_layer)
model.compile(optimizer=Adam(2e-4), loss='categorical_crossentropy', metrics=['accuracy'])

I don't know what I hove done wrong that my multi-label semantic segmentation model doesn't have proper results.

Image and masks:


texts:
I am trying to generate descriptive statistics using agg function in Pandas.  I am having trouble with one line with a lambda function.  They work when I run them as separate lines of code, but when I put them as a single line I get errors.

Any guidance is much appreciated.

The following two lines of codes work when I run them individually.

First line of code:

bh_df.groupby('CAT.MEDV').agg(
  avg_Nox=('NOX', 'mean'))

Second line with lambda function.

bh_df.groupby('CAT.MEDV').agg(
   rng=("NOX", lambda x: (max(x) - min(x))))

However, when I combine them into a single line of code as:

bh_df.groupby('CAT.MEDV').agg(
   avg_Nox=('NOX', 'mean'),
   rng=("NOX", lambda x: (max(x) - min(x))))

I get a whole bunch of errors:

  File "", line 4, in

  rng=("NOX", lambda x: (max(x) - min(x))))

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\groupby\generic.py", line 1455, in aggregate
      return super().aggregate(arg, *args, **kwargs)

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\groupby\generic.py", line 264, in aggregate
      result = result[order]

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\frame.py", line 2986, in getitem
      indexer = self.loc._convert_to_indexer(key, axis=1, raise_missing=True)

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1285, in _convert_to_indexer
      return self._get_listlike_indexer(obj, axis, **kwargs)[1]

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1092, in _get_listlike_indexer
      keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1185, in _validate_read_indexer

Final error:

  raise KeyError("{} not in index".format(not_found))

  KeyError: "[('NOX', '')] not in index"


texts: Best Approach to Forecasting Numerical Value Based on time series and categorical data?

texts:
Consider a dataset of thousands of car repairs that have been performed. In simplest of terms, the columns to consider are the time of year when it was broken (seasonal changes in demand for car repairs), type of damage to car (some damages take longer to repair than others), and the type of car (some cars are more difficult to work on).
I am inquiring as to the best fit for trying to model data of this format where you are predicting the repair time based off of timeseries and categorical data as the inputs. Keep in mind that the data does not have a constant period.
Example Column Names:
datetime | Type of Damage to Car | Type of Car | Repair Time
Any suggestions?


texts: Optimizing vector values for maximum correlation

texts:
I'm new to ML, linear algebra, statistics, etc. so bear with me on the terminology...

I’m looking to find a vector that produces the maximum correlation for the relationship between 1) all relationships among dimensions of the vector (determined by subtraction) and 2) some output value produced by said relationships.  I'm specifically using this to create a sports ranking system that takes a number of matches and the resulting scores and attempts to assign a value to the teams that can be used to predict future scores.  In other words, the difference between any two team's ratings should be predictive of the score differential for the next match between the two.

So for example, if I have 3 teams, A, B, and C, each start with unknown ratings:

$$
\begin{array}{c|c}
  A&?\\
  B&?\\
  C&?\\
\end{array}
$$

If each team played each other team once, the left table would be used to calculate their rating differences (column team's rating minus row team's rating).  The right table would be the difference in scores in the respective matchups.

$$
\begin{equation}
\begin{array}{c|c}
  &A&B&C \\
  \hline
  A&*& B - A& C - A\\
  B&A - B& *& C - B\\
  C&A - C& B - C& *\\
\end{array}
\Rightarrow
\begin{array}{c|c}
  &A&B&C \\
  \hline
  A&0&3&6\\
  B&-3&0&3\\
  C&-6&-3&0\\
\end{array}
\end{equation}
$$

Here is a possible solution that, for this example, would result in a perfect correlation between team rating differentials and score differentials.

$$
\begin{array}{c|c}
  A&1\\
  B&2\\
  C&3\\
\end{array}
$$

This would be the regression line where x2 is the column team's rating and x1 is the row team's rating.

$$y = 3 * (x_2 - x_1)$$

It’s worth noting that what matters is the relationship between the various values (not their nominal values) since this would be another possible solution:

$$
\begin{array}{c|c}
  A&2\\
  B&4\\
  C&6\\
\end{array}
$$

Which would result in a linear equation that looks like this, which would also have a correlation of 1:

$$y = {3\over 2} * (x_2 - x_1)$$

What I want to do is find a method to determine values for A, B, and C that maximizes the correlation between the pairwise differences and resulting output values.  The one additional catch for the teams example is that not every team will play every other team so any resulting matrices will be asymmetrical (assuming that matters).

Are there any existing techniques to address this problem?


texts: How do I build a model to predict which service a customer will use on an app?

texts:
There's an app with over 50 services. I have the data on the type of service a specific customer (they have a unique customer number) does on the app, the date, location, time, duration on a service and the volume of that service.
I want to build a ML model to personalize the services the customer gets daily when they open the app. i.e. Top 6 services they're likely to use for that day
Which machine learning technique would be optimal to achieving this?


texts:
I'm a student of machine learning, and these days I was trying to learn how to use the TensorFlow library. I've gone through various tutorials and trial&errors with tensorflow, and I thought the best way to learn it for real would be to make use of it in a little project of my own.

I've decided that I should attempt to make a program that writes poems. I'm not aiming for top-end quality program; for my first model, I'd be happy with just a string of non-sense words groups together in poem format. The problem is that I'm having problem looking up books or videos about machine learning programs that deal with writing sentence structures.

Can you make any suggestions on what I could look for (even google keywords are fine) to get the sample programs and basic knowledge that I need?

Thank you.


texts:
I am solving a rare event cum classification problem. I have come across  a package called SMOTEBoost which oversamples the rare event and boost the results. But I'm not sure is that supported in R.

Could you please help me how can I use SMOTEBoost in R? Any examples?


texts: What's SHAP contribution dependency plots from xgboost package in R?

texts:
clf = xgboost(data.matrix(train2), label = train$MACROSOMIA,objective = "binary:logistic", nrounds = 20)
pred = predict(clf, data.matrix(test[, -12]))

xgb.plot.shap(data.matrix(test2), model = clf, plot_loess =TRUE,features= "GESTAC")

This is my code, and I get a SHAP feature contribution to prediction dependencies on feature value. I just have a hard time understanding it. I wonder what is it? How do I interpret it?

Here's a link! Could you help me understand what does this plot mean?

xgb.plot.shap()


texts: What is the difference between FC and MLP in as used in PointNet?

texts:
I am trying to understand the PointNet network for dealing with point clouds and struggling with understanding the difference between FC and MLP:

  "FC is fully connected layer operating on each point. MLP is
  multi-layer perceptron on each point."

I understand how fully connected layers are used to classify and I previously thought, was that MLP was the same thing but it seems varying academic papers have a differing definition from each other and from general online courses. In PointNet they seem to be used to mean different things?

My understanding was that up until the max pool in the network all the layer are run independently for each other for each of the N points and then the outputs of the nx1024 layer are max pooled.
Why is it described a shared MLP then?

How does an MLP with an input 3 coordinates lead to an output from the MLP of 64? Or from 64 to 1024 even?

If this MLP is actually taking inputs from all of the N points, how do 1024 points lead to 64 wide MLP layer?

What does it mean by global and local features?


texts: Match a uploaded image with another image using machine learning?

texts:
Hopefully someone may point me in the right direction for this problem of mine, what would be the best ML algorithm to use in order to match an image with another (or very similar) image on the web? When i say best i am thinking about performance and also speed on the user's end

Thanks!


texts: How to construct confidence bound for Time Series Prediction?

texts:
I have some time series data and am using some Deep Learning techniques to get its prediction. Now, I would like to construct confidence bounds for it.

I calculated the residuals, their mean and standard deviation. Then I plotted prediction vs residuals and calculated mean + 3$\sigma$ and mean - 3$\sigma$, and plotted them on my graph. So, I have something like this

Now, I would like to use a similar technique so that I can have confidence bounds on time series plots.


texts: What video card is necessary for resnet34 64 batches?(How much memory)

texts: I have an old nvidia gt240 and I was considering buying gt 730. However, despite being 7 years older, the gt240 is faster. Do I need at least 4g memory? Also, what card would you recommend?


texts:
I have a few word vectors-
recvfrom,sendto,epoll_pwait,recvfrom,sendto,epoll_pwait

getuid,recvfrom,writev,getuid,epoll_pwait,getuid

Now i want to tokenized them and then make them into sequences to feed into the model-
For a standard word vector I would do something like this-
### Create sequence
vocabulary_size = 20000
tokenizer = Tokenizer(num_words= vocabulary_size)
tokenizer.fit_on_texts(df['text'])
sequences = tokenizer.texts_to_sequences(df['text'])
data = pad_sequences(sequences, maxlen=50)

But in my data I have non dictionary words and also I have some repeating words. How do I convert this data into sequences?


texts: I understand the concepts of L1 and L2 loss (i.e. L1 loss will force some parameter coefficients to zero while L2 will only make them approach zero). What do these do when implemented in XGBoost? Does L1 loss prune the tree more significantly than L2 loss?


texts: Is it okay to train Triplet Loss where the anchor and the positive are the same?

texts:
Triplet loss roughly defined as total of$$  max (- distance(Anchor,Negative) + distance(Anchor,Positive) +margin,0)$$
I'm working with triplets data, and it turns out that some triplets(just small percentages) are having anchor=positive.
Is it okay to just train it? Or should I remove the cases of anchor=positive?
Why I think it's okay to let it train when anchor=positive :
$distance(Anchor,Positive)$ become zero, so the model will still try to push $distance(Anchor,Negative) = margin$
But somehow I'm still in doubt. Can someone clarify it for me? Thanks


texts:
I have a CNN architecture for CIFAR-10 dataset which is as follows:

  Convolutions:           64, 64, pool

  Fully Connected Layers: 256, 256, 10

  Batch size:             60

  Optimizer:              Adam(2e-4)

  Loss:                   Categorical Cross-Entropy

When I train this model, training and testing accuracy along with loss has a very jittery behavior and does not converge properly.

Is the defined architecture correct? Should I have a maxpooling layer after every convolution layer?

The Conv-2 architecture is an attempted implementation of the following paper.

The following table is taken from The Lottery Ticket Hypothesis By: Jonathan Frankle, Michael Carbin

Thanks!


texts:
I am trying to use a permutation test to test my hypothesis.
I want to make sure I am understanding concept of permutation correctly.
I have control and experimental group. Then I combine them and resample from combined dataset randomly calculating desired statistics. Do this N time and distribute them on a histogram to see where my observed statistics stand.
In the book "practical statistics for data scientists" it says that from combined data (without replacing) resample same size as groupA in my case control group. Then since I have only two groups rest are simply groupB(experimental group)?
So my question simply put, when using permutation test on two groups first you resample size of groupA then rest automatically becomes resample of groupB?


texts:
When googling "weighted KNN", the results appear to be focused on weighting the nearest neighbor values after those neighbors have been determined.  I'm looking for something that assigns a level of importance to various dimensions that could possibly change the neighbors that are considered nearest with the goal of maximizing the accuracy of predictions made using the resulting model.

For example, if I have a new observation defined as [1, 2], #1 and #2 below would be considered the nearest neighbors assuming K=2 under a normal KNN (as I understand it):

[1.5, 2.5] d=0.707107
[1.8, 1.5] d=0.943398
[4.0, 2.2] d=3.006659

However, if the dimensions were weighted by importance, such as [0.1, 1], the items above would have respective distances of 0.502494, 0.50636, 0.360555 based on sqrt((d1*w1)^2 + (d2*w2)^2), which would make #3 the overall nearest.

Given the goal of maximizing prediction accuracy, I'm wondering what methods are available to accomplish this?  I'd like to potentially use this for both classification and regression - are there methods other than KNN that I should consider?


texts:
I am trying to build a CNN+RNN model for a computer vision problem.
below is my code

def cnn_with_rnn(shape):
    model = Sequential()
    model.add(Conv2D(32, (3, 3), strides=(2, 2), activation="relu",kernel_initializer='truncated_normal',bias_initializer='truncated_normal', input_shape=shape))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(64, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(128, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(LSTM(50))
    model.add(Dense(1))
    model.compile(optimizer=Adam(lr=1e-5), loss="mse", metrics=[custom_metric])

My image is a RGB image with the following shape - [66,200,3], where 3 is the number of color channels.

When i am trying to run the above code , i am getting the following error

ValueError: Input 0 is incompatible with layer lstm_3: expected ndim=3, found ndim=4

How can i combine CNN+RNN for colored images and how to solve my above problem?


texts: Is There any RNN method used for Object detection

texts:
after reading the state of the art about object detection using CNN (R-CNN Faster R-CNN ,YOLO, SSD...) I was wondering if there is a method that use RNN's or that combine the use of CNN's and RNN's for object detection ??
Thank you


texts: Training custom NER on OCR text with SpaCy won't train

texts:
I want to perform information extraction from documents. I wanted to try Spacy's NER method, so I follow following steps :
1)OCR on text document, using Tesseract. As output I have a list of words with corresponding box on the document

Reconstruct text based on the box coordinates of each word detected (to get full text of document)

Tag each box with a custom entity

Generate Training data adapted for Spacy NER (located in variable TRAIN_DATA)

I have 3000+ samples inside Training and 9 custom entities. Documents are in french. My code for training is :
from __future__ import unicode_literals,print_function
import plac
import random
from pathlib import Path
import spacy
from tqdm import tqdm

model = None
import shutil
n_iter = 100

if model is not None :
     nlp = spacy.load(model)
     print(f"Loaded model {model}")
else :
    nlp = spacy.blank('fr')
    print('Import blank model')

if 'ner' not in nlp.pipe_names:
    print('creating ner pipeline')
    ner = nlp.create_pipe('ner')
    nlp.add_pipe(ner,last = True)
else :
    ner = nlp.get_pipe('ner')

for _,annotations in train_data:
    for ent in annotations.get('entities'):
    ner.add_label(ent[2]) #ent is (start_span,end_span,entity_name)

other_pipes = [pipe for pipe in nlp.pipe_names if pipe!= ner]
with nlp.disable_pipes(*other_pipes):
    optimizer = nlp.begin_training()
    for itn in range(n_iter):
    random.shuffle(TRAIN_DATA)
    losses = {}
    for texts,annotations in tqdm(TRAIN_DATA):
        nlp.update([texts], [annotations], sgd=optimizer, drop=0.35, losses=losses)
    print("Losses", losses)

The problem is that at each iteration I have Losses {}. It seems the train is not working.
If someone have tips of have experience with ner on ocr output thanks you. The reconstructed text is quit long on average (2347 characters). I can't provide example of training data.
In fact I think it is difficult to perform NER because my documents are not really structured with sentences (my documents are bills)


texts:
I'm looking for a metric that can be used to quantify how imbalanced the labels are in a dataset.
I'm not looking for a strategy to solve the imbalance problem, I just want to present how imbalanced my dataset is. I've computed the ratio of the most frequent and least frequent labels which is probably an ok way of doing it but I'm sure there's a more robust way?


texts: General question on EDA, correlations, classification, ML

texts: I am looking for a general best practices regarding classification and correlations. I created a new predictor feature call it B, based on a certain threshold in a feature A. Now I started to do EDA and I am not sure which feature to include in my EDA, A or B. When I do correlations plots, nothing correlates with feature B, but some features do correlate with feature A. Which one should I take into account then, A or B correlations? Also, how can I make use of those correlations and scatterplots and pairplots anyway and are they important? If I am using random forest or NN, do I even need to bother with all of the pairplots and correlations to extract features from? I have around 150 features and not sure how to approach the problem of which features to use. I haven't found a source saying how to make a proper use of all of this in a real world scenarios. Any help is appreciated.