texts: Extractive text summarization, as a classification problem using deep networks

texts:
I understand that this is broad, but I merely require a few pointers.

I wish to implement a text summarization system. I have already tried out an abstractive approach, now I am trying an extractive approach. The current extractive approaches out there(Textrank, Lexrank etc) are not giving optimal performance. It was suggested to me that I take it as an unsupervised learning task, using Autoencoders/RBM's or K means or any other unsupervised learning algorithm to classify sentences as important or not.

My questions are:

Is something like this feasible? (I know that nothing can be said for sure in data science unless tried out, but is it worth the shot?)
What kind of features should I feed into the classifier? I already know of word frequency distribution(TF-IDF), sentence position, co-occurance stats, but will these be enough? What additional features should I consider? Or should I consider directly feeding in word vectors?

I have done my bit of work on the same and have read several papers, but none of them offer clarity in terms of feature computation, but only explain network architecture.

Any form of help will be appreciated, thank you!


texts: Transposed convolution as umpsampling in DCGAN

texts:
I read several papers and articles where it is suggested that transposed convolution with 2 strides is better than upsampling then convolution.

However implementing such model with the transposed convolution resulted in heavy checkboard effect, where the whole generated image is just a pattern of squares and no learning takes place. How to properly implement it without totally messing up the generation? With the upsampling+convolution I got okay result but I want to improve my model. I am trying to generate images based on the CelebA dataset.

I use keras with tf and I used the following code:

model.add(Conv2DTranspose(256, 5, 2, padding='same'))

model.add(LeakyReLU(alpha=0.2))

model.add(BatchNormalization(momentum=0.9))

model.add(Conv2DTranspose(128, 5, 2, padding='same'))

model.add(LeakyReLU(alpha=0.2))

model.add(BatchNormalization(momentum=0.9))

model.add(Conv2DTranspose(64, 5, 2, padding='same'))

model.add(LeakyReLU(alpha=0.2))

model.add(BatchNormalization(momentum=0.9))

Here I try to turn a 4x4 image into a 32x32. Later it will be turned into a 64x64 image with 1 or 3 channels depending on the image. However I get the following pattern always. Some tweaking usually leads to some other pattern but it does not really change:

Checkboard effect

Thank you for your answers in advance


texts: Using Machine Learning to create marketing segments

texts:
I'm working on a project to try and predict which users would be most likely to subscribe to our services. I'm using R and am working with a data frame that has a user's email address, information about their past purchases (order count and revenue generated), how they have interacted with our website (different events triggered), and whether or not they have subscribed. I'm using this data to train and test different algorithms. I've used caret with knn, lda, cart, and nnet.

I've been able to get how accurate my models are but that doesn't help me know who to market to. When I plug new data into my model I would like to know which users it thinks are the most likely to subscribe and also give me their associated email addresses. I've looked all over but haven't been able to find anything that helps me pair a prediction to an email address. Any help would be much appreciated.

PS.

I'm not using the email addresses to classify the data. I simply have them to associate a row of data to a user. I had to get rid of them to run it through the training and testing algorithms and that is where I am having the issue of knowing what prediction data is associated with which email address and in turn which user.


texts: LSTM / GRU prediction with hidden state?

texts:
I am trying to predict a value based on time series by series of 24 periods (the 25th period)

While training I have a validation set with I babysit the training (RMSE) and each epoch, eval the validation:

Receive errors as:

Train RMSE: 3.02

Validation RMSE: 5.65
Validation r_squared: 0.75

However, when I evaluate the test (in the same size of the validation) I receive very bad results by:

    model.eval()
    x_test = x_test.cuda()
    y_test_pred, hidden = model(x_test)

Test RMSE: 8.18
Test r_squared: -5.62

I suspect the difference is that in the validation it creates while the model is training (performed model.eval()) so using the current hidden state and while test is with zero hidden state?
How can the hidden state calculated for the test? The train and the test are not the same size..


texts: How to use Scikit-learn's affinity propagation clustering with my own datasets?

texts:
I am trying to cluster my datasets using affinity propagation. I followed this and this links to grasp the basics of affinity propagation clustering. The sample code available at sklearn is as follows:
from sklearn.cluster import AffinityPropagation
from sklearn import metrics
from sklearn.datasets.samples_generator import make_blobs
# Generate sample data
centers = [[1, 1], [-1, -1], [1, -1]]
X, labels_true = make_blobs(n_samples=300, centers=centers, cluster_std=0.5,
                            random_state=0)
# Compute Affinity Propagation
af = AffinityPropagation(preference=-50).fit(X)
cluster_centers_indices = af.cluster_centers_indices_
labels = af.labels_

While running it, it works as stated on the website. However, I could not understand it completely. I want to modify this code to use with my datasets. My datasets consist of values from different sensors on a 2-D surface. I want to cluster the values with similar sensors readings at various points on the 2-D surface. How can I do it?
Thank you.


texts: What does Make Density Based Clusterer in Weka do?

texts:
In Weka, there is a clustering algorithm with the name as Make Density Based Clusterer. When going through its properties, it takes a clusterer as base clusterer(I took it as K-means with k=3).
It initially performs k-means and creates three clusters. I see prior probabilities for each cluster and attribute-wise normal distribution means and standard deviation in the result buffer.

What happens after k-means clusters are calculated?
What role mean, standard deviation and prior probabilities play here?
Why is it called density based?


texts: Explanation on how to interpret force_plot result by using SHAP

texts:
I have applied SHAP for explaining the outcome of my neural network. For the force_plot I have obtained the following output when trying to look at multiple instances:

My problem now is that I'm not sure how to interpret the results. I understand that the x-axis provides information about my data samples (making up 500 samples in total). I would appreciate it if someone nows how to approach this.


texts: Which open-source sgdb for kind of large data

texts:
I have a 7 giga confidential dataset which I want to use for a machine learning application.

I tried :

Every package recommanded for efficient dataset management in R like :

data.table,
ff
and sqldf with no success.

Data.table needs to load all the data in the memory from what I read, so it's obvious that it will not work since my computer has only 4g RAM. Ff leads to a memory error too.

So I decided to turn to sgdb and I tried :

Mysql which managed to load my dataset in 2 hours and 21'. Then I began my requests (I have a few requests to do to prepare my data before I export a smaller set in R for machine learning application), and then I had to wait for hours before I got the following message "The total number of locks exceeds the lock table size" (my request was just an update to extract the month from a date for each tuple).
I read that postgre was similar to mysql in performance so I didn't try
I read that redis was really performant but not at all adapted to massive importation like I want to do here so I didn't try
I tried mongoDb, the nosql upraising solution that I heard everywhere about. Not only I find rather disturbing that mongoimport is so limited in options (I had to change all semi-colon in commas using sed before I can import the data), but It seems to be less performant that mysql since I launched the loading yesterday and it is still running.

What I can't try : data are confidential so I don't really want to rent some space on Azure or Amazon clouding solution. I am not sure that it is that big that I have to turn to Hadoop solution but maybe I am wrong about that.

Is there an open-source performant solution that I didn't try that you would recommend to perform some sql-like requests on a biggish dataset ?

Edit :
Some more details about what I want to do with these data for you to visualize. These are events with a timestamp and a geolocalisation. I have 8 billions of lines. One example of what I want to do :

standardize series identified by geolocalisation (I need to compute mean grouping by geolocalisation for example),
compute average count of events by type of season, day... (usual group by sql request)...

Edit

As a beginning of answer for those who have limited hardware like me, rSQLite seems to be a possibility. I am still interested in other people experiences.


texts: Neural Network with Connections to all forward layers

texts:
In classical neural nets, we have that each layer is connected only with the following layer.  What if we relaxed this constraint and allowed it to be connected to any or all subsequent layers?  Has this architecture been explored?  It seems to be backprop would still work on this.

At the very least, this type of network could be emulated by artificially creating identity neurons at each layer that bring in the value of every earlier node, where the incoming weights to those layers are fixed at 1.


texts: Interpreting the results of the probabilistic neural networks (Monte Carlo dropout approach)

texts:
I have a Keras NN model where I apply the Monte Carlo dropout approach as a predictive method to evaluate the uncertainty of the model outputs. From my research in the probabilistic neural networks, I understand that the lower the entropy (close to 0), the more probable that the predicted value is really certain.
In my model, I ran the inference step 100 times, averaged model output probabilities, and calculated the Entropy. The following is a randomly chosen instance from my results (mean_proba is averaged over 100 iterations of predict):
instance_id       y_true   mean_proba     Entropy
    2301            1        0.9009       0.0940

My question is: from the instance# 2301 result,  does this imply that the model has a low uncertainty, i.e., high information gain, that instance# 2301 should really be predicted as 1? Is this interpretation sound?


texts:
In regards to ROC curves, how do you pronounce ROC? I have always spelled out the letters like R-O-C but I sat through a sales demo today where they guy pronounced it as a word like "rock" as in "the rock curve".

So who's right?


texts: Tableau: Displaying Averages and Aggregate Calculations In Filtered Visualizations

texts:
I'm new to Tableau and I just started by making a graph with multiple filters. What I want to do is display some averages of some columns and other aggregate functions dynamically as the filters are changed by the viewer.

Here is the viz.

Now, as someone changes the filters, I want to display an average of the  column, and let's say, a weighted average of the "Runs" column with "Strike Rate" as the weight, and so on. This should update live, as the filters are changed, and should be based on the currently selected values.

Is there a way to do this?

Thanks a ton!


texts: Use of decision trees for classifying images

texts:
I am new at Machine Learning and reading about it I wonder if it is possible (and convenient) to use decision trees to classify images.

For instance, to classify faces


texts: How does LDA model do inference on new documents?

texts: Through Latent Dirichlet Allocation (LDA) one can learn topics from texts. The trained topic model can then be used on unseen data to infer to what extent each of the learned topics are represented in the unseen text. How this (similarity) score for each topic is calculated?


texts: FP-Growth - find ALL patterns containing a specific item(s) only

texts:
The FP-Growth Mining Algorithm finds frequent itemsets i.e. it does not search for patterns by item. If I want to find ALL frequent itemsets for an item, I can use the nodelink property to traverse all paths in the fp-tree for that item, which means I have to traverse all paths for all children of that item.

In the worst case this effectively means I have to run the mining algorithm on the entire fp-tree (and then filter the results for the specific item I'm looking for). This seems like an expensive operation to perform repeatedly, as I'm only looking for a small subset of results. Is there any other mechanism that would allow me to mine patterns selectively?

1) Mining Frequent Patterns without Candidate Generation; Jiawei Han, Jian Pei, and Yiwen Yin


texts:
I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of those 'black boxes'. The explainers assign each feature an importance value for each prediction.

The thing is, those feature importance values differ from explainer to explainer, and it is rather tricky to evaluate which explainer actually performs best and produces the most sensible outcome. The authors of SHAP have tested the explainers' consistency with human intuition, assuming that:

  [...] good model explanations should be consistent with explanations from
  humans who understand that model.

Right, that makes sense. But how to evaluate the performances of different explainers in a more generic and scalable way, without the need for human intervention?

Some authors have recently attempted to come up with evaluation metrics. I found one approach to be interesting. The authors worked with the following intuition:

  [...] a feature-based local explanation has high quality only if it is
  difficult to change the prediction when the explanation features are
  kept intact. The quality of an explanation is quantified by the
  minimum amount of perturbation on features that are not in the
  explanation region in order to alter the prediction.

That makes sense, too. So I am planning to build up on that and focus on tabular data. The idea would be to, for each instance (row) in our test data:

Run the neural network and observe the predicted outcome.
Run the explainer and identify each feature's importance value.
Keep the features with the highest importance value intact and alter the remaining features.
Run the neural network again. If the predicted outcome hasn't changed, go back to step 3 and alter the features some more until a switch in the outcome.

Then, the explainer for which the necessary amount of alteration to obtain a change of outcome was the highest (aggregated across all instances), would be the best.

I was hoping to open a debate about quantitative techniques that can evaluate explainers. What do you think? Also, speaking of 'altering' the features, what would be the appropriate way to do so (when working with tabular data)? I'm open for comments and thoughts!


texts: Adjust predicted probability after smote

texts:
i  have an imbalance data set and I used smote to oversample the minority class and undersample the majority class. now, I want to check the test AUC using predict_proba of the model.

I have two questions: 1. Do I have to correct the probability if I am comparing AUCs? 2. How can I correct it (a combination of undersampling and oversampling!)


texts: For every Bayesian Network, is there a Neural Network that gives the same output?

texts:
After thinking about a question comparing Bayesian Network to Neural Network, I am now wondering if they may not be one and the same thing!

At this point, my maths (unused for over 20 years fails me!).   But....

If infinite random test data was used
The output of the Bayesian Network was found for each data item
And the Neural Network was then trained on it

Would the Neural Network not tent towards being the same as the Bayesian Network?


texts: Why is the learning rate for the bias usually twice as large as the the LR for the weights?

texts:
I've noticed in a few caffe models I've been working with that the learning rate for the bias is often set to be twice that of the one for the weights.
Another user mentions that this is the case in this thread but does not elaborate on what the reason for doing so is.
Is there some justification for having different learning rates for the bias as the weights? Did I miss some paper providing a theoretical or experimental justification of this?


texts: How does the number of trees effect the prediction time in gradient boost classification trees?

texts:
After tuning hyper-parameters for a gradient boosted model, I have found that the best tree count (iterations) is a few thousand.
I'm worried that such a high count might impact prediction performance.
Can someone explain the relation between trees count and prediction time?


texts: Dropping one category for regularized linear models

texts:
While reviewing the sklearn's OneHotEncoder documentation (attached below) I noticed that when applying regularization (e.g., lasso, ridge, etc.) it is not recommended to drop the first category. While I understand why dropped the first category prevents collinearity, I am unsure why it is needed for regularized regression. Wouldn't this this add an additional dimension that will need to be regularized?

drop{‘first’, ‘if_binary’}

Specifies a methodology to use to drop one of the categories per
feature. This is useful in situations where perfectly collinear
features cause problems, such as when feeding the resulting data into
a neural network or an unregularized regression.
However, dropping one category breaks the symmetry of the original
representation and can therefore induce a bias in downstream models,
for instance for penalized linear classification or regression models.


texts: F1 score vs accuracy, which metric is more important?

texts:
I have two multiclass classification models for making predictions (number of classes is three to be precise). One is Keras neural network, other is Gradient Boosted Classifier from Scikit Learn library.

I have noticed that after training on same data GBC has higher accuracy score, while Keras model has higher F1 score. Which model should I use for making predictions on future data? Which metric is more important?


texts: Transposing Every nth row to column in a large dataset

texts:
I am attempting to work with a very large data-set (~1.5mil lines) for the first time in SAS and I am having some difficulty. The data-set I have is formatted as a "long" .txt file as follows:

'cat1/: Topic1_Variable1'
'cat2/: Topic1_Variable2'
'cat3/: Topic1_Variable3'
'cat4/: Topic1_Variable4'

'cat1/: Topic2_Variable1'
'cat2/: Topic2_Variable2'
'cat3/: Topic2_Variable3'
'cat4/: Topic2_Variable4'

'cat1/: Topic3_Variable1'
'cat2/: Topic3_Variable2'
'cat3/: Topic3_Variable3'
'cat4/: Topic3_Variable4'
...

Fro analysis and sharing with others, I really would like to see it formatted as follows:

cat1              cat2              cat3              cat4
Topic1_Variable1  Topic1_Variable2  Topic1_Variable3  Topic1_Variable4
Topic2_Variable1  Topic2_Variable2  Topic2_Variable3  Topic2_Variable4
Topic3_Variable1  Topic3_Variable2  Topic3_Variable3  Topic3_Variable4

I think that this may be easier in R, but I honestly am drawing a complete blank in SAS. I've even played with MS Access to try to get it to look the way I want but the program crashes every time (due to the size?). At any rate, I have looked into some of the statements in PROC TRANSPOSE and PROC SQL but it seems that most functions within those procedures are utilized to combine duplicate 'Topics'. In the data I have been provided, each "group" represents an individual response to a question with several thousand individuals repeated, I want to retain the independence of each occurrence and not perform a UNION as defined in PROC SQL. At this point, I feel like I am over-thinking this but I just can't get around the mental block and actually do what I am working toward.
Any help or guidance is much appreciated. I'm open to trying all suggestions or ideas, I think I have access to most statistical computing programs.


texts: Tools to preprocess a big data for dashboards?

texts: I have a complex dataset with more than 16M rows coming from pharmaceutical industry. Regarding the data, it is saved in a sql server with multiple (more than 400) relational tables. Data got several levels of hierachies like province, city, postal code, person, and antigens measures, etc. I would like to create many dashboards in order to observe the changes & trends happening. I can use Pentaho, R (shiny) or Tableau for this purpose. But the problem is data is so huge, and it take so long to process it with dashboard softwares. I have a choice of making cube and connect it to dashboard. My question here is whether there are  any other solutions that I can use instead of making a cube? I don't want to go through the hassle of making & maintaining a cube. I would like to use a software where I specify relationships between tables, so the aggregation/amalgamation happens smoothly and output processed tables that can connect  to dashboards. I hear Alteryx is one software that can do it for you (I haven't tried it myself, and it is an expensive one!). I understand this task needs two or more softwares/tools. Please share your input & experience. Please mention what tools do you use, size of your data, and how fast/efficient is the entire system, and other necessary details.


texts: Algortihm for making predictions from minimal data

texts:
I am working with classification problem.
 I have a dataset with a lot of features. A lot of them can easily determine class. On production, I want to ask the user to provide me with only part of the information.

Which algorithm is able to make pretty accurate predictions from incomplete data?

I was thinking about decisions tree, but I don't want to determine the entry point to my algorithm.

Suppose that my features are binary. I want to give choice to my user which features are most common for him and based on that information make pretty accurate predictions.


texts: Creating a custom dataset for object detection

texts:
I'm currently trying to build a model to recognize approximately 10 labels (food items) in a fairly controlled environment (refrigerator). I was unable to find datasets that worked well enough for my task, so I'm attempting to curate one myself.

So far this has been my approach:

Capture images with a smartphone
Annotate in LabelImg
Train in Detectron2 / pyTorch

I have 2 questions:

Given that the environment is somewhat consistent throughout all samples, is there a ballpark number / rule of thumb for a decent number of samples to use per class? (100 train / 100 test)? This is a proof of concept project, so I'm just looking for something with reasonable accuracy (80%+)
After I've captured my images (let's say via smartphone) are there any preprocessing steps (other than annotating) that are necessary before using as training data? (i.e resize the images, reduce file size, format)

Absolutely any help is appreciated, and of course other tips/suggestions you feel may be useful.


texts: comparison of t-SNE and PCA and truncate SVD

texts:
How to compare the trucate SVD ,PCA, and T-SNE?

What we can say about features if  t-SNE and PCA and truncate SVD digaram is in this figure?


texts: Orange3 summarizing data, grouping data values

texts:
Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values?

For example, instead of plotting a scatter with lots of data points, I would like to plot just the average y value at every x value. In pandas, this is easily done with groupby().mean().

Is there a similar widget/feature I'm overlooking in orange?


texts: AlexNet CNN how can it be applied to my case?

texts:
I'm working on my last year project where I'm given digitized WSI (Whole Slide Images), though they're fairly small around 1390x1040 size (which is unusual). These images are of cases of Glioblastoma Multiforme (brain cancer) which is stained with Ki-64 index, which results in what I assume malignant parts being marked as brown. Here's a small example of what I'm looking at.

My objective in simple terms is to count the blue and brown cells (estimation of proliferation indices) which based on what I've researched is a segmentation problem. I've also came to conclusion that AlexNet is a successful architecture for this purpose. However, the trouble I'm having is that this data is unlabeled and as a Comp Sci student, I don't think I have enough expertise to annotate the ground truth values. My question the boils down to this, are there any alternative methods should I explore, or should I drop this dataset altogether?


texts: Expectation number of points in initial clustering for LSH

texts:
I have a very skewed, 10-dimensional data set. I need approximate nearest neighbours for my use case and I was looking into Locality senstive hashing. However after scaling and randomly generating hyperplanes through the origin and coding the data points like that I got very skewed buckets because of the nature of the data. After thinking about it for a bit I came up with an idea to take random points from the data and use these as cluster centers for the hash. Every point will be mapped to the ID of the closest randomly picked center. My question is if the expected size of the bucket of a specific point is the same for all the other data points. I think that is the case but others say it shouldn't. My rationale is that more dense areas have more randomly decided cluster points while outliers will not be picked very often. I can't find anything about this in literature.

Edit: I did run some tests and they seem to support my hypothesis somewhat but variance is relatively high because there is a high dependence between sizes of clusters (data points wise)


texts: What should be the input shape for convLSTM if ResNet-50 is applied before?

texts:
I have a video dataset, extracted all its frames, and applied ResNet-50 to extract features from all frames. ResNet-50 provides feature map of (2534, 7, 7, 2048), 2534 are the number of frames.
Now I have to apply convLSTM to train the model, but what should be its input shape.
Regards


texts: ML models: average of all versus average of averages?

texts:
I have data for 20 different people and am training a model (e.g. a neural network with the same hyperparameters) on the data from each person; so this gives me 20 models.

I chose to use RMSE to assess the performance. However, since the training data is shuffled, the computed RMSE is nondeterministic and so oscillates. So I thought running each model 10 times and averaging the results, i.e. the RMSE's, would give me a better estimate of performance. But this is for a single person/model. How do I combine the performance of everything, i.e. all 20 models, into a single measure?

Run each of the 20 models 10 times for a total of 200 RMSE values, and average all? Or first compute the average for each person, and then the average of these averages?

Maybe a different method is better? The end goal is to compare a couple of models (e.g. NN vs SVM).


texts: Is GEMM used in Tensorflow, Theano, Pytorch

texts:
I know that Caffe uses GEneral Matrix to Matrix Multiplication (GEMM) which is part of Basic Linear Algebra Subprograms (BLAS) library for performing convolution operations. Where a convolution is converted to matrix multiplication operation. I have referred below article.
https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/

I want to understand how other deep learning frameworks like Theano, Tensorflow, Pytorch perform convolution operations. Do they use similar libraries in the backend. There might be some articles present on this topic. If someone can point me to those or can explain with an answer.


texts:
In the context of multi-regression, I am wondering if there is a way to decompose $$VIF_i = 1/(1-R_i^2)$$ where $R_i^2$ is the r squared obtained from the regression of dependent variable = i and independent variables are all other factors.

I want to decompose $VIF_i$ or $R_i^2$ into individual factors to see how much each individual factor contributes to the $VIF_i$ or $R_i^2$

Someone recommended using the square of partial correlation coefficient and that value is linearly related to $R_i^2$. My undestanding is that partial correlation coefficient measures the correlation between two variables, holding the other variables constant.  Is this a viable option?


texts: Interpretation of som_model with kohonen package?

texts:
I've just started learning som_model(self-organizing map) on R Studio with kohonen package. I've tried to find resources on the topic to understand how the different maps works but most of the resources use the information. So my question is how do I interpret the smo_model if arguement for the type is quality.
My Code
 plot(som_model, type = "quality", main="Node Quality/Distance", palette.name=coolBlueHotRed)

Output

My Interpretation
The smaller the mean distance the lower the shade color will be which in this case it should be within the range of the blue shades. However, there are some outliers therefore it does not affect the interpretation of the map.


texts: Convert filters pre-trained with ImageNet to grayscale?

texts:
I am wondering how to convert caffe reference model trained with ImageNet (color pics) for grayscale image to save memory and to speed up.

The filters in the caffe convolution layers are different for RGB but this can be, for example, averaged to form one channel filter.

Since my data is grayscale and speed at test (prediction) is important for me, this may help.

I can of course convert my grayscale images in RGB representation to reuse pretrained convolution layers as they are. But I want to avoid it if possible.

My questions are:

Would this be possible?

How exactly one should combine filters? (Average?)

After filters converted, weights between conv layers need to be re-trained?


texts:
I'm doing experiments with GAN. I've successfully trained GAN on 28x28px MNIST dataset (samples scaled to (-1,1) interval)

My next experiment is to train GAN on bigger images. My dataset consists of grayscale 128x128px images. I've added one more convolutional layer to both generator and discriminator and updated dense layers accordingly.

Sample from MNIST dataset and sample from my custom dataset:

However, when I trained the model, all results from generator looks like this:

What am I doing wrong? Why does the generator generate only this noise pattern instead of samples from my dataset distribution?

Generator:

H = Sequential()

H.add(Dense(1024, input_shape=(noise_dim,)))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Dense(128*16*16))
H.add(BatchNormalization())
H.add(Activation('relu'))
H.add(Reshape([16, 16, 128]))

H.add(Conv2DTranspose(128, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(64, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(32, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(1, (3, 3), padding='same'))
H.add(Activation('tanh', name='out_image'))

gen_in = Input(shape=(noise_dim,), name='in_noise')
gen_out = H(gen_in)

Discriminator:

H = Sequential()

H.add(Conv2D(32, (3, 3), strides=(2, 2), padding='same', activation='relu', input_shape=(128, 128, 1)))
H.add(LeakyReLU(0.2))

H.add(Conv2D(64, (3, 3), strides=(2, 2), padding='same', activation='relu'))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))

H.add(Conv2D(128, (3, 3), strides=(2, 2), padding='same', activation='relu'))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))
H.add(Flatten())

H.add(Dense(1024))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))

H.add(Dense(1, activation='sigmoid', name='out_binary'))

disc_in = Input(shape=(128, 128, 1), name='in_image')
disc_out = H(disc_in)


texts: Unusually High BLEU score on a NMT model

texts:
This is the project on Neural Machine Translation on the English/Irish language pair. I have been spending the past month or so trying to train a good baseline to do 'experimentation' on. I have a corpus of ~850k sentences (unfortunately Irish is very limiting). When I trained it and evaluated it with BLEU, I got a score of 65.02, which is obviously absurdly incorrect. These were my Fairseq-train settings:

!CUDA_VISIBLE_DEVICES=0 fairseq-train data-bin-full_corp/MayNMT \
  --lr 5e-4 --lr-scheduler inverse_sqrt --optimizer adam\
  --clip-norm 0.1 --dropout 0.2 --max-tokens 4096 \
  --arch transformer --save-dir checkpoints/full-tran

I know not everyone uses Fairseq in NLP, but I hope the arguements are self-explanatory.

I deduplicated the dataset (converted to a Python set() which only takes unique entries) so I don't think the issue is the dev/valid and test sets contain duplicate entries, but I'm not sure what else causes this. Some suggest overfitting may be a cause, but I feel that would only affect the BLEU if the dev set shared training entries.
I've tried to find the problem myself, but there aren't many places that cover NMT, let alone BLEU.


texts: Unable to Print or Save Predicted Results in Orange?

texts:
I have applied some models (Logistic, Tree, SVM, Neural Network, etc) in order to predict my results. However, when I printed or saved my predicted results, I was unable to print the Complete predicted Instances or rows. While in Orange, predicted result to display all and Complete instances but was unable to save or print the same results.

Here is a screen shot:

See the printed results (+52 more) but is unable to see these +52 instances.


texts: Approach to classify spatial trajectories of vessels

texts:
I'm trying to create a classifier to distinguish different boats by their trajectories. I have training data of the longitude and latitude of a boat and time in seconds.

Vessels like a ferry will have a straight predictable trajectory between two points, whereas fishing vessels can have zig-zag like trajectories for example.

My initial approach is to create features for example the mean speed, standard deviation of the speed, standard deviation on the course, such that each trajectory table is distilled into 1 row of features. Then I can train something like a random forest classifier on these rows.

Is this a good approach, any other suggestions that could account for the characteristic trajectory shapes.

Thanks


texts: Why does Feature Importance change with each iteration of a Decision Tree Classifier?

texts:
After applying PCA to reduce the number of features, I am using a DecisionTreeClassifier for a ML problem

Additionally I want to compute the feature_importances_. However, with each iteration of the DecisionTreeClassifier, the feature_importances_ change.

Iteration #1

Iteration #2

Why would it change? I thought the initial split was made on a feature to "produce the purest subsets (weighted by their size)". Acting on the same training set, why would that change?

Thanks in advance for any help.


texts: How can I increase the speed and performance of my implementation of an AI for Reversi?

texts:
I made an AI for Reversi, aka Othello (8×8), like Alpha Zero, using this book. This book is written in Japanese.  The source code of the AI I implemented can be found in this Github repository. There are comments written in Japanese, but there are also comments written in English. Check the master branch's README file and source code on GitHub for more details.

I finished writing the code and I trained the model on Google Colab with a GPU. I stopped learning around the 3rd training cycle because it is very slow. I tried to rewrite to bitboard, but, it also very slow. So, how can I make my implementation faster or, in general, how can I improve my implementation, given that I have a limited amount or resources (including computing resources and money)?

Here are some ideas.

Reduce the number of hidden layers (it can be weak)
Rewrite the source code with Cython (it also run on Google Colab).
Rewrite the source code in C++, and run on my PC with GPU.
Increase the learning rate (it can do over-fitting)


texts: Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams?

texts: I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc.  I want to know if the cluster features or microclusters are suitable or not.


texts: I don't really understand how to approach this problem. I know that $w=\sum_{n=1}^{N}{a_ny_nx_n}$ and $y_n(w^T\cdot x_n+b)=1. $ So I can solve for $b$ from that equation but I can't figure out how to apply equation(1) that I'm supposed to use.


texts:
I am reading the Chapter-1 of the Deep Learning book, where the following appears:

  A wheel has a geometric shape, but its image may be complicated by
  shadows falling on the wheel, the sun glaring off the metal parts of
  the wheel, the fender of the car or an object in the foreground
  obscuring part of the wheel, and so on.

  One solution to this problem is to use ML to discover not only the
  mapping from representation to output but also the representation to
  itself. This approach is called representation learning.

Here, I did not understand the exact definition of representation learning. I have referred to the wikipedia page and also Quora, but no one was explaining it clearly. The lack of explanation with a proper example is lacking too.

The authors cite the autoencoder as an example for a representation learning algorithm.

  An autoencoder is the combination of an encoder function, which
  converts the input data into a different representation, and a decoder
  function, which converts the new representation back into the original
  format

Can someone explain in simpler terms what exactly representation learning is?

[Would be much better if explained with an example. The car wheel example would do too.]


texts: Why do I get different coefficients from Logistic regression in Python and SPSS

texts:
I am a bit confused in regards to the model coefficients calculated by SPSS and sklearn's LogisticRegression.
I am getting different coefficients and intercepts for both methods.
in Python, I am running the following code:
import numpy as np
from sklearn.linear_model import LogisticRegression

vals = [0.01, 0.04, 0.07, 0.08, 0.08, 0.09, 0.10, 0.15, 0.20, 1.85, 1.93, 1.97 ,2.02, 2.09, 2.12, 2.13, 2.21, 2.25]
labels = [0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1]

X = np.array(vals).reshape(-1, 1)
y = np.array(labels)

solvers = ['liblinear', 'newton-cg', 'lbfgs', 'sag', 'saga']

for i in solvers:
    model = LogisticRegression(solver=i, random_state=0).fit(X, y)
    print(f'intercept: {round(model.intercept_[0],3)} coefficient: {round(model.coef_[0][0],3)} ... solver: {i}')

which outputs:
>> intercept: -1.012 coefficient: 1.239 ... solver: liblinear
>> intercept: -1.609 coefficient: 1.498 ... solver: newton-cg
>> intercept: -1.609 coefficient: 1.498 ... solver: lbfgs
>> intercept: -1.609 coefficient: 1.496 ... solver: sag
>> intercept: -1.609 coefficient: 1.498 ... solver: saga

(In reality, my dataset is much larger of course)
In SPSS if I use Analysie > Regression > Binary Logistic at default settings. I am getting a different coefficient and intercept.

Is there a transformation on the coefs that is done by skleanr but not by SPSS or is the Logistic regression method used by sklearn so much different from the one in SPSS. When I calculate the log_odds and the the predicted probabilities for the equations derived from either python and SPSS, I am getting quite similar results, however still a little different.
Does anyone know why this difference occurs or whether there is a common method for the two that would yield similar results?
thank you in advance.


texts: What model to fit pseudo-logarithmic trend?

texts:
I am dealing with a scatterplot where I am trying to figure out the relationship between two variables, but I have so far failed to identify what the best fit could be. The presence of zero values prevents me from using exponential and power law regression lines.

Zero values are not shown as the Y axis starts from 1, but they are present. My bad.

This is the scatterplot. To me it looks like this is some sort of logarithmic trend but I am not sure about it.

What model will best interpret this trend?


texts: Sklearn Decision Tree as weak learner in Adaboost not working properly

texts:
I'm trying to implement Adaboost algorithm with sklearn decision tree as the Weak Learner - at each step I want to choose one feature with one threshold to classify all samples.
I have 1400 long feature vectors and want to label them 1 or -1.
The features are words from movie ratings and the label represents "bad" or "good".
At some of the iterations, the decision tree decides on a feature, threshold 0.5, and classifies all samples as -1 (no matter their value)
and on the next iteration chooses the same feature, this time classifying the samples as it is supposed to.
Can anyone find a reason for this?
tree prints:

feat: 311
==================
|--- worst <= 0.50
   ---> class: 1.0

| --- worst >  0.50
   ---> class: -1.0

==================
alphas = 0.16872595425475514
feat: 27
==================
|--- bad <= 0.50
   ---> class: 1.0

|--- bad >  0.50
   --->class: -1.0

==================
alphas = 0.21421414954211687
feat: 371
==================
|--- boring <= 0.50
   --->class: -1.0

|--- boring >  0.50
   ---> class: -1.0

==================
alphas = 0.1881155411693614
feat: 371
==================
|--- boring <= 0.50
   ---> class: 1.0

|--- boring >  0.50
   ---> class: -1.0

==================
alphas = 0.12644785644997397
feat: 822
==================
|--- ridiculous <= 0.50
   ---> class: -1.0

|--- ridiculous >  0.50
   ---> class: -1.0

Code:
def run_adaboost(X_train, y_train, T):
    hypotheses = []
    alpha_vals = []
    num_of_samples = len(X_train)
    D = [1/num_of_samples for _ in range(num_of_samples)]
    for t in range(T):
        h = weak_learner(D, X_train, y_train)
        idx, threshold = h.tree_.feature[0], h.tree_.threshold[0]
        tup = (get_prediction(h, X_train[0]), idx, threshold)
        print_tree(h, [vocabulary[idx] for idx in range(len(X_train[0]))])
        hypotheses.append(tup)
        epsilon = 1-h.score(X_train, y_train, sample_weight=D)
        alpha = 0.5*np.log((1-epsilon)/epsilon)
        alpha_vals.append(alpha)
        D = new_distribution(D, X_train, y_train, alpha, h)

    return hypotheses, alpha_vals

##############################################

def weak_learner(D, X_train, y_train):
    clf = tree.DecisionTreeClassifier(max_depth=1, criterion="entropy")
    clf.fit(X_train, y_train, sample_weight=D)
    return clf

def new_distribution(D, X_train, y_train, alpha, h):
    Z = 0
    Dt = [0]*len(D)
    print(f"alphas = {alpha}")
    pred = h.predict(X_train)
    for i in range(len(X_train)):
        exponent = (-1) * alpha * y_train[i] * (pred[i])
        Z += D[i]*np.exp(exponent)
    for i in range(len(X_train)):
        exponent = (-1) * alpha * y_train[i] * (pred[i])
        Dt[i] = (D[i]*np.exp(exponent))/Z
    return Dt

def get_prediction(clf, vector):
    feat = clf.tree_.feature[0]
    print(f"feat: {feat}")
    vec = vector.copy()
    vec[feat] = 0
    vec = vec.reshape(1, -1)
    return int(clf.predict(vec)[0])

def print_tree(clf, feat_name):
    r = tree.export_text(clf, feat_name)
    print(r)
    print("==================")

##############################################

def main():
    data = parse_data()
    if not data:
        return
    (X_train, y_train, X_test, y_test, vocab) = data
    global vocabulary, X_test_g, y_test_g
    X_test_g, y_test_g = X_test, y_test
    vocabulary = vocab
    T = 80
    run_adaboost(X_train, y_train, T)

if __name__ == '__main__':
    main()


texts:
I am trying to figure out how to use NLTK's cascading chunker as per Chapter 7 of the NLTK book. Unfortunately, I'm running into a few issues when performing non-trivial chunking measures.

Let's start with this phrase:

"adventure movies between 2000 and 2015 featuring performances by daniel craig"

I am able to find all the relevant NPs when I use the following grammar:

grammar = "NP: {<DT>?<JJ>*<NN.*>+}"

However, I am not sure how to build nested structures with NLTK. The book gives the following format, but there are clearly a few things missing (e.g. How does one actually specify multiple rules?):

grammar = r"""
  NP: {<DT|JJ|NN.*>+}          # Chunk sequences of DT, JJ, NN
  PP: {<IN><NP>}               # Chunk prepositions followed by NP
  VP: {<VB.*><NP|PP|CLAUSE>+$} # Chunk verbs and their arguments
  CLAUSE: {<NP><VP>}           # Chunk NP, VP
  """

In my case, I'd like to do something like the following:

grammar = "MEDIA: {<DT>?<JJ>*<NN.*>+}
           RELATION: {<V.*>}{<DT>?<JJ>*<NN.*>+}
           ENTITY: {<NN.*>}"

It occurs to me that a CFG might be a better fit for this, but I only became aware of NLTK's support for this function about 5 minutes ago (from this question), and it does not appear that much documentation for the feature exists.

So, assuming that I'd like to use a cascaded chunker for my task, what syntax would I need to use? Additionally, is it possible for me to specify specific words (e.g. "directed" or "acted") when using a chunker?


texts: Using GPUs of other machines in the network with Keras

texts:
My situation is as follows:

I have a rather cheap laptop with Ubuntu 18.04 running on it that unfortunately is not powerful enough (old, cheap GPU) to train deep learning models with. I am located in city A.
My father (located in city B) has two computers (with powerful GPUs) with Windows 10 Professional running on both of them that I could theoretically use at nighttime. I can connect to the network via VPN and have SSH root access to both machines.

I am now looking for a Keras setup that allows me to train neural networks from my local machine making use of the remote GPUs. Ideally, I would have a Jupyter notebook with all the logic running on my conputer and could somehow connect to a thin client (probably exposing some interface to CNTK) running on the remote machines. I'd prefer to keep the Windows machines as dumb as possible and would ideally only make use of their GPUs (with some local caching of course). Also, I would prefer to stick with Windows on the remote machines not having to set up some dual-boot with a Linux distro. Enabling the Linux subsystem would be fine though.

Any thoughts on how to tackle my problem?


texts: How to claim that a CNN model is lightweight?

texts: CNN model has some parameters that can show the a model is lightweight compared to others. The parameters can be Size(after training), Trainable parameters or multiply add. Which parameters are exactly necessary to show that a CNN model is lightweight? How to claim this in research paper? Which kind of proof is necessary?


texts: Should I transform a multiple regression with outliers into ordinal regression?

texts:
I have small dataset of about 60 samples that performs poorly in regression. So I wonder how can I transform this task into predicting intervals instead of values. Is it possible to make it perform better for intervals instead of values?

The dataset is imbalanced. There are just few high values. The algorithm constantly underestimates these values. So I think if I make prediction in intervals, it'll perform better for values, say, more than 50.

But, overall, the problem is predicting for values near the limits of intervals. Is it better to mark this values as belonging to both intervals?


texts: I am a data science neophyte struggling to solve a churn prediction problem, i would be grateful to have someone help with the regression model


texts: What is the efficiency difference between different cost functions in case of neural networks?

texts:
I'm studying the theory behind neural nets and I wonder if there is any actual difference between using different lost/cost functions?

Let's say I could use either MAE or MSE for backpropagating loss; does this decision have any actual effect on the model efficiency?

At the end of the day both functions just calculate the difference between $y$ and $\hat y$ (albeit on a different scale). But for the optimizer, only the error tendency is what really matters, not the absolute difference.

Of course, this question is relevant for any other model evaluation.


texts:
I want to know the do not want to how to use library

I will denote a $n\times p$ data matrix by $X$, where $n<p$. That is, each row of $X$ is one sample data with $p$ feature variables.

By using singular vector decomposition method, I can decompose $X$ into $A$, $B$, and $C$ such that $X=ABC$, where $A$ is a $n\times n$ matrix satisfying $A^TA=I_{[n\times n]}$, $B$ is a $n\times p$ matrix whose diagonal elements are all non-negative and real but non-diagonal elements are all zeros, and $C$ is a $p\times p$ matrix such that $P^TP = I_{[p\times p]}$.
Without the loss of generality, I assume that $B$ is obtained satisfying that the diagonal elements of $B$ is descending order, i.e., $b_{i,i} > b_{j,j}$ for $i<j$.

The score matrix, $S$, is obtained by $S=XC$, whose dimension is $n\times p$.

In order to reduce the dimension of variables, we do as follows:
Denoting the $i$-th column of $C$ by $c_i$, which is a loading vector with respect to $i$-th principal component.
Hence, deleting $c_{k+1}$ to $c_{p}$, I can obtain $C_k = [c_1~c_2~\cdots~c_k]$.
The dimension-reduced score matrix, $S$, is obtained by $S_k=XC_k$.

In this situation, how can I run KNN?
Actually, I already made my own KNN code, whose input is trainX, trainY, and testX, and whose output is evaluatedY.
I want to compare (i) basic KNN and (ii) dimension-reduction-applied KNN.

I obtained error rate of (i) basic KNN using 5-fold cross validation method.

yhat = myKNN(trainX, trainY, testX);
errRate = myErr(y, yhat);

To obtain error rate of (ii) dimension-reduction-applied KNN, is the following correct?

trainS= myDimensionReduction(trainX, k)
testS= myDimensionReduction(testX, k)
yhat = myKNN(trainS, trainY, testS);
errRate = myErr(y, yhat);

where myDimensionReduction is a function that converts a data matrix $(n\times p)$ into a reduced data matrix $(n\times k)$.


texts: How would you use a series of events as a factor for a NN?

texts: I am working with an LSTM nn built with Keras. I have the need to pass in a history of events as a single factor and I'm thinking of doing it with as a list. But I've never passed in a list in this manner - is that something that is possible with Keras? Or do I need to go through some sort of categorization process for this?


texts: PRML Bishop - Doubt in Entropy Formula (Section 1.6, equation 1.94)

texts:
Please refer to the following excerpt from "Pattern Recognition and Machine Learning" Bishop.

  We can understand this alternative view of entropy by considering a
  set of N identical objects that are to be divided amongst a set of
  bins, such that there are $n_i$ objects in the $i^{th}$ bin. Consider
  the number of different ways of allocating the objects to the bins.
  There are N ways to choose the first object, $(N - 1)$ ways to choose
  the second object, and so on, leading to a total of N! ways to
  allocate all N objects to the bins, where N! (pronounced ‘factorial N
  ’) denotes the product N \times (N - 1) $\times \ldots \times 2 \times
> 1$.  However, we don’t wish to distinguish between rearrangements of
  objects within each bin. In the $i^{th}$ bin there are $n_{i}!$ ways
  of reordering the objects, and so the total number of ways of
  allocating the N objects to the bins is given by:
  $${\bf W} = \frac{N!}{\prod_{i}n_{i}!}$$ which is called multiplicity.

I am confused about the bold-faced part.

If bin 1 were to have n=1 object and bin 2 were to have 2 objects and so on, then the number of options for placing the objects in $i^{th}$ bin would be like - (with successive bin have only remaining objects to choose from):

$$^{N}C_{1} * ^{N-1}C_{2} * ^{N-3}C_{3}* ^{N-6}C_{4} \cdot \cdot $$

Were am I missing the point?

P.S.: If each bin has only one object, then I can understand N!.


texts: I build xgboost model for regression problem. By the default xgboost optimize $(y - y_{pred})^2$, so the RMSE will be the best eval metric to measure performance. But my task is to build the best model for evaluation metric which check if predicted value is in range $-/+10%$ of true value i.e. $y_{pred} \in [0.9*y,1.1*y]$. Do you have any idea for this or maybe MSE is still the best option?


texts: Why does my linear regression model converge to a non-zero gradient value?

texts:
I have a basic 2D Linear Regression model coded out (using gradient descent), yet it doesn't seem to work as well as it should.

What I expect is that m and c should approach 4 and 3 respectively, and m's slope or c's slope should tend to 0; yet what is actually happening is that c's slope approaches a non-zero value, and c itself approaches a value depending on the epoch (around 0.5 with an epoch of 100.)

If I look at the graph of c, it very slowly tends up over time, though.

Code here:

import random, math
import matplotlib.pyplot as plt

def linreg(x, y):
    """ Performs linear regression: input x, output y. """
    n = float(len(x))
    m = random.random()
    c = random.random()
    dm, dc = [], []
    rate = 0.00001
    epoch = 100
    for run in range(epoch):
        d_m = 0
        d_c = 0
        for i in range(len(x)):
            d_m += (y[i] - m*x[i] - c)*x[i]
            d_c += (y[i] - m*x[i] - c)
        d_m *= -2/n
        d_c *= -2/n
        m -= d_m * rate
        c -= d_c * rate
        dm.append(d_m)
        dc.append(d_c)
    return m, c, dm, dc

x = [i for i in range(400)]
y = [4*i + 3 for i in x]

m, c, dm, dc = linreg(x, y)

print(m, c)

plt.grid()
plt.scatter(x, y)
plt.plot(x, [m*i + c for i in x], color='red')
plt.show()

plt.grid()
plt.plot([i for i in range(len(dm))], dm)
plt.plot([i for i in range(len(dc))], dc, color='red')
plt.show()


texts: How to convert the reconstruction error from a variational autoencoder into a probability score?

texts:
I am faced with a dilemma while working with a variational autoencoder model. I am thinking of converting the reconstruction error into a probability score (aim is to use to determine anomalies because of the complete lack of prior knowledge about actual anomalies). A google search turned up this paper: An, J., & Cho, S. (2015). Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2(1). (this can be accessed here here).
On pp 8, section 3.1, the paper gives a short description of the algorithm followed by pseudo-code. In that description they mentioned:

"For each sample from the encoder, the probabilistic decoder outputs
the mean and variance parameter.
Using these parameters, the probability of the original data generating from the distribution is
calculated. The average probability is used as an anomaly score and is called the reconstruction
probability."

I am having some difficulty in understanding the fact that "the decoder is outputting mean and variance of the data generation process." I thought it is the encoder which learns the mu and sigma and decoder reconstructs the input from the sampled input.
Can anybody clarify my doubt?
I greatly appreciate your input.


texts: SKLEARN GridSearchCV hinting higher accuracy than Pipeline but with same parameters as Pipeline estimators

texts:
I have pipeline estimators like this:
text_clf = Pipeline([
      ('tfidf', TfidfVectorizer(max_df=0.95, min_df=2, max_features=n_features, stop_words='english')),
  ('clf', SGDClassifier(loss='hinge', penalty='l2', alpha=1e-3, random_state=random_state, max_iter=5, tol=None)),
])
text_clf.fit(dataset.data, dataset.target)

Then when evaluating the model accuracy like this
mean = np.mean(predicted == twenty_test.target)
print("mean %0.3f" % mean)

I get score 0.802.
Then when I add the GridSearch to get the best params like this:
parameters = {
     'tfidf__use_idf': (True, False),
     'clf__alpha': (1e-2, 1e-3),
}

gs_clf = GridSearchCV(text_clf, parameters, cv=5, n_jobs=-1)
gs_clf = gs_clf.fit(dataset.data, dataset.target)

(Note that i am fitting GridSearch on TRAIN data same as the pipeline - although when i first tried fit it on TEST data by mistake the result was same though.)
It reports this:
grid search best score 0.868
Best params: clf__alpha: 0.001
Best params: tfidf__use_idf: True

Note, that these params are already set on the model but the score is lower in the pipeline.
Is it because the other parameters I set in Pipeline are not kept when using GridSearch or?
Btw how the Grid Search knows the best parameters if I didn't provide any testing data.
Another problem is, that when I added more parameters to adjust in Grid search and then applied it to Pipeline, the accuracy didn't change at all. (or changed from 0.802 to 0.805, but GS hinted 0.867)


texts: The use of feature scaling in scikit learn

texts:
I'm studing machine learning from here and the course uses 'Scikit Learn'  for regression - https://www.udemy.com/machinelearning/

I can see that for some training regression algorithms, the author uses feature scaling and for some he doesn't because some 'Scikit Learn' regression algorithms take care of feature scaling by themselves.

If I apply feature scaling to an algorithm which already has feature scaling internally, will it affect in any way ?
Can I always apply feature scaling, no matter whether the algorithm I'm using has a system of feature scaling internally or not ?


texts: How to create a person-specific (personalized) classification model using machine learning

texts:
I have labeled datasets for 10 subjects that contain photoplethysmogram (PPG) signal value and the affective state as a label.

How can I create a person-specific(Personalized) classification model to classify the subject affective state?

Do I need to merge all 10 datasets in one dataset and train the model? or shall I create a separate model for each subject by using his data (this may not help in classifying the affective state of unseen persons?
?


texts: How can I train a machine learning model with below characterstics?


texts: Stacking and Ensembling methods in Data Science

texts:
I understand that using stacking and ensembling has become popular, and these methods can give better results than using a single algorithm.
My question is: What are the reasons, statistical or otherwise, behind the improvement in results?
I also understand that at a high level that combining these methods will combine the predictions from different algorithms. Each algorithm has its own strengths and weaknesses, but not sure how combining them will actually improve the results.
Here is a simple balanced example (50% of the labels are actually T and 50% are actually F), where I think stacking or ensembling algorithms will still give the same result as the original model. In this case, for the ties (1 F and 1 T), I decided to select T. The same problem would occur if I selected F instead. This is a little more complicated if I used the predict_proba, but think the same problem would occur.


texts:
Regularization is used to handle over-fitting problem.

Similarly can we have some methodology to overcome under fitting problem or merely adding new features or training data will help us in reducing under-fitting issue?


texts:
For me, the original data looks to have like a decreasing or constant trend but stl() is giving a different trend altogether. Can someone here please explain why?
The decomposition plot is as follows:


texts: Filling missing values with pyspark using a probability distribution

texts:
I want to fill missing values in my dataframe.

 In [1]: df = spark.createDataFrame([[1],[1],[2],[3],[3],[None],[3],[None],[3],[2],[None],[1],[4]], ['data'])
 In [2]: df.show()
 +----+
 |data|
 +----+
 |   1|
 |   1|
 |   2|
 |   3|
 |   3|
 |null|
 |   3|
 |null|
 |   3|
 |   2|
 |null|
 |   1|
 |   4|
 +----+

I know I can use pyspark.ml Imputer to fill with the mean / median, or use this method to fill with the last valid value. These are fine options, but I would like to impute with a random sample from the data distribution. For example, in the data provided, nulls will be filled according to these probabilities:

 P(1) = .3
 P(2) = .2
 P(3) = .4
 P(4) = .1

What would be the best way to fill these values from a random sample?


texts: Low memory error while performing degree 2 polynomial regression on (3000*1835) sized array

texts:
I am working on a problem to predict the revenue, a film will generate. Some of the features available in the data set are json collection for the crew, cast which worked in the film. I applied onehotencoding to these columns.
As a result, I have a (3000*1835) sized array. This too I got after extracting only director's data from 'Crew' columns and applying PCA with 60% variance retention.
But, when I apply polynomial regression, I get the below mentioned error:

  $\lib\site-packages\sklearn\model_selection_validation.py:532:
  FitFailedWarning: Estimator fit failed. The score on this train-test
  partition for these parameters will be set to nan. Details:
  MemoryError: Unable to allocate 30.2 GiB for an array with shape
  (2400, 1686366) and data type float64

I am using the code as shown below for polynomial regression:

polyFeature = PolynomialFeatures(degree=2)
linearRegression = LinearRegression()
pipeline = Pipeline([('polyFeature',polyFeature),('linearRegression',linearRegression)])
score = cross_val_score(pipeline,XTrain,YTrain,n_jobs=4,cv=5)

I am using a system with 6 cores, 32 GB RAM.


texts: Tensorflow: Save History of Images from Generator

texts: I'm building a Generative Adversarial Network. I need to be able to save the output of the generator in some data structure, and then at each training step, sample from the current generator, as well as from the historical images. How can I do this in TensorFlow? I don't want to use a feed_dict for performance reasons.


texts:
I am a beginner in the field of machine learning, I have small doubts for which I did not find any suitable answer. How to select the best model for prediction of unknown data. I have learned two ways I am not able to figure out which one is correct one.

Stating with, train your model by splitting the data into training and test data, then fit the model to predict the test output and error, shuffle the data and average the error over some 100 or more cycles. This will give us an average error (rmse for test) over 100 cycles (python way random state). Now to predict the unknown data (validate the model), which model should I consider the model for prediction.

1-Which have rmse closer to average rmse: pick a model which reports error approximately equal to the average error achieved over 100 cycles to predict the unknown data and called that as the model for prediction, or

2-Best performance: Chose a model out of 100 which is best (lowest rmse for test) for prediction of unknown data?

Another thing I am also struggling is that if considered the lowest error model as the model for unknown data prediction, by keeping X_train and y_train same of the best model.And If I chose my X_test and y_test from the same database (10% only) over 400 different times and predict the error, will it be an overfitted prediction?

Thanks in advance


texts: Feeding data to Xgboost for recomender system

texts:
I am using xgboost for a recommender system. There are 3-4 recommended content on each page. My data consists of columns like page_id and advertisement_id. Currently for every page_id, there are 3-4 rows of advertisement_id in data. Xgboost then gives me the probability of likelihood of an ad being clicked.

It is known that one content(advertisement_id) from each page(page_id) is definitely clicked. I want to leverage this fact while training xgboost.

How can I tell xgboost that one ad_id from each page_id is definitely clicked?

Can this be done somehow by using a vector of advertisement_id for each row of page_id?


texts: Does a neural network continue to change after SGD stops improving?

texts:
I would like to understand the behaviour of stochastic gradient descent (SGD) over long time periods for a fixed learning rate.

Let us assume that after a number of epochs we expect no further improvement to our cost function.  If we continue to train over more epochs then one of two things can happen:

The solution keeps changing but remains close to a local minimum; in the space of weights it remains in a local region.  Or,
The solution keeps changing but explores a large region of the space of weights.

Let's make this precise.  Take a fixed shape of neural network and use the Euclidean distance metric on the space of weights.  Now train this network using SGD on a dataset, e.g. MNIST, and for a fixed choice of learning rate.  After a large number of epochs, N start recording the weights of the network,
$$ {\bf w_N}, {\bf w_{N+1}} \ldots $$
How does $d({\bf w_N}, {\bf w_M})$ behave?  To gauge whether these distances are large or small, they can be compared to the distances you get by comparing ${\bf w_N}$ with equivalent networks given by permuting the neurons within a layer.

Ofcourse the answer may vary depending on the neural network, the data, the learning rate, etc.  I suppose I would like to know the answer for typical "real world" networks.  Unfortunately I lack the processing power (and expertise) to adequately investigate this myself, and am confident this will be known to experts.


texts: How to go about creating embeddings (especially, token to Id mapping) for categorical columns in tensorflow 2.0+?

texts:
I have a csv with both categorical and float dtypes. I want to do the following:

For each categorical column i will use pandas to compute the unique values (pd.unique()) that are present in the column. say u_l for a column
I will use the len(u_l) to decide upon the dimension of embeddings that i use for a particular categorical column that i want i embed (this step is the reason i cannot use tensorflow_transform)
I want to create some stateful node that can map category (token) value to embeddings index thus subsequently i can lookup the embedding from embeddings matrix that i created in step 2

I dont know how to go about doing it currently. A very inelegant solution i can see is using tensorflow_datasets:

encoder = tfds.features.text.TokenTextEncoder(u_l,decode_token_separator=' ')
concatenate the entire column using space delimiter (c_l) (c_l is one string now) and then using encoder.encode(c_l)

This is a very basic thing that i think tensorflow would be able to do relatively easily. Please guide me to the right solution


texts:
My dataset is a simple table of 20 columns and 100,000 rows.It is not a image data as commonly used in CNN. What input shape should I provide in this case?

Right now I did-

input_shape = (21,109713,1)

model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),
                 activation='relu',
                 input_shape=input_shape))

which gives the error-

ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (109713, 19)


texts:
I have this array (500,):
https://pastebin.pl/view/adeccfa1
Want to make it look like this one (500,20):
https://pastebin.pl/view/6650ad6b
How do I do it?


texts: Machine Learning Approach for Aerodynamic Design?

texts:
I am new to ML so please excuse my ignorance. I am trying to use machine learning to enhance the design of turbomachinery components. Traditionally, a component such as a compressor is parameterized via maybe 30 or so parameters and then these parameters are tuned by hand using an iterative process involving computational fluid dynamics (CFD) to evaluate potential designs. The engineer uses his/her judgment to guide the design until it is satisfactory with regard to efficiency, stress etc. Essentially the engineer is trying to find the best combination of the input variables.

This process is quite tedious when done by hand, and I would like to develop a machine learning tool to speed it up. An optimization routine could be used, and I have done this, but then it would require successive, expensive design evaluations to optimize the design. I would like to train an ML model produce an ‘optimal design’ without expensive iterations. I am OK with an expensive training phase.

Essentially I would like the computer to act as an engineer would: take the state of the surroundings (the complete fluid dynamic information from a CFD simulation) and adjust the design variables until an ‘optimal’ solution is found.

Essentially, I just want the best combination of all variables. Any thoughts on what ML strategy I should consider?

I initially thought reinforcement learning would be a good fit, but all the examples I can see seem to have a time element, that is the changes in the inputs need to be done in a certain order. For my problem, there is no order, the results have no memory.

Maybe a GP surrogate model would be good? But I would like the tool to develop a generalized strategy for when it sees this type of flow structure to make this type of change to the geometry. I am not sure a GP surrogate model could do this.

Thanks,


texts: What network is called high-capacity network? Why?

texts:
In this paper deep_image_prior :

  To demonstrate the power of this parametrization, we consider inverse
  tasks such as denoising, super-resolution and inpainting. These can be
  expressed as energy minimization problems of the type

  where E(x; x0) is a task-dependent data term, x0 the
  noisy/low-resolution/occluded image, and R(x) a regularizer. The
  choice of data term E(x; x0) is dictated by the application and will
  be discussed later. The choice of regularizer,  which usually captures
  a generic prior on natural images, is more difficult and is the
  subject of much research. As a simple example, R(x) may be the Total
  Variation (TV) of the image, which encourages solutions to contain
  uniform regions. In this work, we replace the regularizer R(x) with
  the implicit prior captured by the neural network, as follows:

Then I see :

  A parametrization with high noise impedance. One may wonder why a
  high-capacity network fθ can be used as a prior at all.

Here, I wonder why they call fθ high-capacity network ? What is the definition? Google just lead me to many non-relevant pages.


texts:
I have built one segmentation model using KMeans ,I am getting average Silhoutte score of 0.4 in scaled data .
But when I do the clustering on Unscaled data Silhoutte scoring improved to 0.80.
Why I am getting Low score on Scaled data
I have used MinMax Scaling .


texts:
I have images with 12 bits per color channel which I use for several detection networks (YOLO, RetinaNet, etc.).
Can I expect any precision difference between 12 bpp and 8 bpp as network input? Or is it better to just scale the images down to 8 bpp?


texts:
I have some problems with layers construction on Keras.
I explain the whole problem:

I have a feature matrix, with dimensions: 2023 (rows) x 65 (features);
I tried to build a CNN, with Conv1D as first layer;

My code is:
def cnn_model():
    model = Sequential()
    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
    model.add(Dropout(0.25))
    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
    model.add(Dropout(0.25))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(64, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='mse', optimizer='adam', metrics=['mse', 'mae'])
    model.fit(X, Y, epochs=100, batch_size=64, verbose=0)
    model.evaluate(X, Y)
    return model

scoring = make_scorer(score_func=pearson)
# evaluate model with standardized dataset
estimation = []
estimation.append(('standardize', StandardScaler()))
estimation.append(('mlp', KerasRegressor(build_fn=cnn_model, epochs=50, batch_size=32, verbose=0)))
pipeline = Pipeline(estimation)
kfold = KFold(n_splits=10, shuffle=True, random_state=1)
results = cross_val_score(pipeline, X, Y, cv=kfold, scoring=scoring)

The problem is that, when it runs, I get the following error:
ValueError: Input 0 of layer sequential_9 is incompatible with the layer: : expected min_ndim=3, found ndim=2. Full shape received: (None, 64)

I really don't know why this error occurs. Probably it's just a problem with parameters passing, I'm quite new on this field. Can you tell me something more? I tried a bunch of things in order to solve this error but every time I get some new errors instead of solving. Thank you all.


texts:
I am trying to optimize the hyperparameters of SVM and CART with tune() function of e1071 R package, but I have a doubt.
Should I tune the parameters on the training data, fit the model on the training data and then test it on the test data, or may I avoid the second step?


texts: Is regularization included in loss history Keras returns?

texts:
I'm getting to know Keras. Right now, I'm testing with regularization and how to use them. Comparing the results of loss history for a training session with and without regularization, it seems to me that the loss history reported by Keras has the regularization term added to it, is that right?

When my model has no regularization term, the loss value starts from something less than 1 but when I retrain the model with regularization (L1L2), the same problem's loss value starts from 500. The only logical explanation I've got for it is that Keras is reporting loss value after it added the regularization term to it. And I believe the loss value without regularization is as valuable as with it, if not more valuable. Don't you think?


texts: Different number of features after using OneHotEncoder

texts:
I have train and test data in two separate files.

OneHotEncoder gives different number of features for Train and Test Data based on the different values they have. But the classifier requires that the number of features for test and train data should be equal, how can I solve this problem?


texts: How to select topology for neural network?

texts:
I was given a target function to design neural network and train: (y = (x1 ∧ x2) ∨ (x3 ∧ x4))

The number of input and number of output seems obvious (4 and 1). And the training data can use truth table.

However, in order to train as a multilayer artificial neural network, I need to choose number of hidden units. May I know where can I find some general guideline for this?

Thank you!


texts: Training an LSTM on two merged writing styles (say Shakespeare and Frost)

texts:
I was trying to develop intuitions on how two writing styles can be merged (if at all they can be merged) into a single LSTM and then get meaningful results from the same.

Can anybody provide me with some insights on how to go about this? I wish to see if I can go about merging Shakespeare and Frost into one model and then get the generated text be like picking up nature themes from Frost and mixing in them a flavour of Shakespeare?

How do you all feel about this?


texts:
My mentor wants me to write and submit an academic paper reporting a predictive model, but without any validation score.

Everything I have read in textbooks or the Internet says that this is wrong, but is there any case where only reporting a train score makes sense?

Background

The model was fit "by hand" by someone in our team, using a visual inspection of features extracted from our entire dataset. It is a linear model based on hand-crafted features extracted from some very nonlinear and high-dimensional data. The linear model is based on less than fifty features, but those features were extracted from thousands. We do not have any data left to use as validation.


texts: Warning when plotting confusion matrix with all sample of one class

texts:
I have two arrays: the first one with all the correct labels (they are all set to zero since each sample belong to the same class) and another one with all the labels predicted by my neural network. What I want do to is plot a simple confusion matrix to show the results. This is my code:
test_labels = [0, 0, 0, ...., 0] #all set to zero
predict_labels = [0, 0, 1, ...., 1]
matrix = confusion_matrix(test_labels, predict_labels)
fig = plt.figure()
colors = ['orange', 'green']
labels = ["Normal", "Anomaly"]
sns.heatmap(matrix, xticklabels=labels, yticklabels=labels, cmap=colors, annot=True, fmt="d")
plt.title("Confusion matrix")
plt.ylabel("Actual")
plt.xlabel("Predicted")
plt.close()

The problem is that when I run this code I get the following warnings:

UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use zero_division parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))
RuntimeWarning: invalid value encountered in true_divide   recall =
tps / tps[-1]
UndefinedMetricWarning: No positive samples in y_true, true positive
value should be meaningless   UndefinedMetricWarning)

How can I solve it?


texts: Comparing different classification results with different trainig and test data

texts:
I have different  samples with different sizes. The instances of each sample have different features in comparison to the instances from the other samples.  For each sample i train my model and tested it with 30% of the data as an unseen data. After obtaining the performance result, i would like  to choose the best one that can predict properly ( in other words, the used features in that sample are more predictive than the other). Now, the problem of comparing their performance is that the size of test sets are not the same. I don't know if the calculation of the 95% confidance intervals can lead to a conclusion by chossing the maximum the min ranges of the confidance intervals.

Thanks in advance.


texts: How to remove irrelevant text data from a large dataset

texts:
I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed that even though the text (around 1-5 %) mentioned some covid-related keywords, the context of those texts are not actually about the pandemic, they are telling a life story (from 5-year-old to 27-year-old) instead of how covid affects their lives.
The data I want to use and am looking for is some texts that tell people how covid makes depression worse and what not.
Is there a general way to clean those irrelevant data whose contexts are not covid-related (or outliers)?
Or is it ok to keep them in the dataset since they only count for 1-5% ?


texts: Prove that the expected value of the training error is equal to generalization error for a class of binary clasiffiers

texts:
Take $ H $ to be a class of binary classifiers ever a domain $X$. $D$ is a unknown distrbution over $X$, and $f$ is the target hypothesis in $H$. If we fix some $h$ $\in H$ we have the expected value of the training error equal to generalization error, that is:

$\mathbb{E}[L_S(h)]=L_{(D,f)}(h)$

Could you please come up with the proof of this statement?
Would you recommend another page inside stackexchange for these theoretical questions about machine learning?

Thank you.


texts: How to build model(s) for simulating conversations between two parties?

texts:
I'm working on a problem for chat based buying and selling of products. Due to lack of training data I most likely have to use synthetic data. I was trying to come up with ways to do that, but so far not able to find a concrete approach.
The scenario is something like this. Two parties are conversing, one of them is a buyer, and the other one is a seller. Buyer is going to ask the price, description and other relevant queries related to the product she's interested in. The seller is going to answer all/some of those and maybe follow up with questions regarding discounts, availability etc.
Now, given this scenario, the first thing that came to my mind is to have two models who are interacting between themselves and getting better and better (somewhat like GANs). But after digging through some blog posts, it seems like GAN for text generation is still far from being great. The other approach might be to have a common corpus. One model (representing the buyer) would be asking question from that corpus, the other model would answer that, also from the same corpus. We can probably leverage BERT for the QA module. But this 2nd model has to have a generative component that'd get back with a follow-up question. This part is where I'm stuck at.
Is there a better way of approaching this problem? Any pointer is welcome.


texts: Is it a good practice to exclude the noisy data for training a model?

texts:
I executed a linear search on an array containing all unique elements in range [1, 10000], sorted in increasing order with all search values i.e., from 1 to 10000 and plotted the runtime vs search value graph as follows:

Upon closely analysing the zoomed in version of the plot as follows:

I mainly have 2 questions:

1. Will piecewise regression will be a better idea to fit the data instead of linear regression, as the plot represents a step function (runtime is step function of search value) ?

2. In this case we are sure that when the step function breaks (or starts taking news values), we have NOISE, that may lower the accuracy of the model. So, should we consider this noisy data in our training dataset or exclude it and why ?

Clarification for what is NOISE : In the second screenshot, the runtime for some of the higher search values in lower than the runtime for lower search values and vice versa

Any suggestion is greatly appreciated!


texts:
I am trying build a classifier for malware analysis for which basing in the instructions of an assembly code, such as push, mov,... I want to predict the compiler, and in a second time the optimization op, and I am having some troubles. My code is the following:

#pakages
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

json_data = None;
with open('training_dataset.jsonl') as data_file:
    lines = data_file.readlines()
    joined_lines = "[" + ",".join(lines)+"]"

    json_data = j.loads(joined_lines)

data = pd.DataFrame(json_data)
data.head()

which gives:

now, when I look at:

len(data['instructions'])

I have as output :  30000

but if I do the following:

for value in data['instructions'].iteritems():
    myList = list(value[1]);

myList

opcodes = [instruction.split()[0] for instruction in myList]

len(opcodes)

I get as output : 151

Why don't I have an output 30000? I don't understand why I have less elements. I want to use the opcodes to build a feature vector, but don't understand why the  number of elements become so low.

Can somebody help me? Thank's in advance.

[EDIT] if it can be useful, if I do:

data['instructions']

I get as output:


texts:
This is my first time to use Data Analytics tool to figure out a solution to a problem.
I have a table with following columns Person ID, Person Name, Note ID, Note (notes is a free form text where a call representative can enter their comments) Insert User. I have been given list of key phrases to be identified from the Note column. These key phrases are in a sentence format.
Example:
I have to find exact or a similar matching phrase from those notes. The format of my final report would look something like this

I looked into couple of articles related to text matching which suggested options such as

fuzzywuzzy, Doc2vec, Difflib, python-levenshtein

It is all so confusing. Even if I get a starter to pick the best suitable option, I can may be take it from there. Any suggestions?
Thank you so much!


texts: XgBoost error: contrasts can be applied only to factors with 2 or more levels

texts:
Error in XGBoost: Error in contrasts<-(tmp, value = contr.funs[1 + isOF[nn]]):
  contrasts can be applied only to factors with 2 or more levels

param <- list(  objective           = "binary:logistic",
                booster             = "gbtree",
                eval_metric         = "auc",
                eta                 = 0.01,
                max_depth           = 5,
                subsample           = 0.6,
                colsample_bytree    = 0.6
)

t1.y <- t1$target

t1 <- sparse.model.matrix(target ~ ., data = t1)
#Error occurs after the above command. If i change the above statement to (replacing . by -1 to drop collinear columns)
t1 <- sparse.model.matrix(target ~ -1, data = t1)

No error occurs after changing to -1 but the dgcMatrix that is generated looks suspicious as it contains just one column and lot of rows. I was expecting the columns of the original matrix to be one-hot encoded rather than getting one columns as an output. If i proceed further ignoring the inspection of generated sparse matrix then I get an error after i run the xgboost below:

Error in xgb.iter.update(bst$handle, dtrain, iteration - 1, obj) :
  [05:14:55] amalgamation/../src/objective/regression_obj.cc:108: label must be in [0,1] for logistic regression

dtrain <- xgb.DMatrix(data=t1, label=t1.y)
watchlist <- list(train=dtrain)
bst1 <- xgboost(   params              = param,
                   data                = dtrain,
                   nrounds             = 1750,
                   #watchlist           = watchlist,
                   verbose             = 1,
                   maximize            = FALSE
)

I tried to convert the target column to both factor and numeric (converted it to 0 & 1, since as.numeric converts it to 1 & 2). But it did not help :'(

When I convert the target to numeric then after running the Xgboost, I get the following error : Error in xgb.iter.update(bst$handle, dtrain, iteration - 1, obj) :
  [05:20:12] amalgamation/../src/tree/updater_colmaker.cc:162: Check failed: (n) > (0) colsample_bytree=0.6 is too small that no feature can be included

*t1 is a data.frame initially containing factor,numeric and integers which i am converting to sparse matrix

Thanks for the help!


texts: Resampling : My dataset is categorical or numerical?

texts:
I have a dataset with 203 variables. Like age>40 (0 -yes, 1-no), gender(0 or 1), used or not 200 types of drugs (one hot encoded into 200 variables), and one target variable (0 or 1). This is an imbalanced dataset where Counter({0: 5607, 1: 1717}).
May I know what kind of resampling strategy I should adopt for this kind of dataset?
Is this dataset considered as numerical or categorical datset?
I tried random under sampling and over sampling, but not satisfied with the ROC curve obtained after modeling.
Can I apply SMOTE considering this as numerical dataset?
I read in this , that In case the dataset only contains categorical variables, the Hamming distance is applied for resampling purpose and If the dataset only contains numerical variables, it is possible to apply traditional distances such as Euclidean, Manhattan or Minkowski.
In case of my dataset, is it okay to apply Euclidean distance for resampling? Could you please direct me to some sources showing how this is done for a datset with only binary values?


texts: In DQN, why not use target network to predict current state Q values?

texts:
In DQN, why not use target network to predict current state Q values, and not only next state q values?
In doing a basic dq learning algorithm with nn from scratch, with replay memory, and minibatch gd, and I'm implementing target neural network to predict at every minibatch samples current and mext state q values, and in the end of minibatch, sync target network, bu I notice the weights diverge very easily, maybe because I used nn to predict current state q values? If yes, why not?
the agent code:
def start_episode_and_evaluate(self, discount_factor, learning_rate,
                                epsilon, epsilon_decay = 0.99, min_epsilon = 0.01):
        state = self.env.reset()
        done = False
        while not done:
            if np.random.uniform(0, 1) < epsilon: action = self.env.action_space.sample()
            else: action = np.argmax(self._nn.predict(state))
            next_state, reward, done, _ = self.env.step(action)
            self._replay_memory.put(state, action, reward, done, next_state)
            if len(self._replay_memory) >= self._batch_size:
                for state_exp, action_exp, reward_exp, done_exp, next_state_exp in self._replay_memory.get(batch_size=self._batch_size):
                    z, a = self._target_nn.forward_propagate(state_exp) #HERE I HAVE TO USE TARGET NETOWRK (_target_nn) OR THE MAIN NETWORK (_nn)?
                    q_values_target = np.copy(a[-1])
                    if done: q_values_target[action_exp] = reward_exp
                    else: q_values_target[action_exp] = reward_exp + discount_factor * np.max(self._target_nn.predict(next_state_exp))
                    self._nn.backpropagate(z, a, q_values_target, learning_rate)
                self._sync_target_nn_weights()
                epsilon *= epsilon_decay
                if epsilon < min_epsilon:
                    epsilon = min_epsilon
            state = next_state

source: https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project/blob/main/src/core/dqn_agent.py


texts: sklearn: sklearn.linear_model.HuberRegressor vs sklearn.linear_model.ElasticNet

texts:
I am experimenting different loss functions for my regression model. I noticed that in the sklearn, there are:

sklearn.linear_model.HuberRegressor and sklearn.linear_model.ElasticNet

To me, both use the combination of L1 and L2 loss. What exactly is the difference of this? Thanks!


texts:
Detectron2 is really good and it supports large number of models / but not all. Eg. Yolo.
Do we have alternative to detectron2 which provide easy to use API for both model inference and retraining? just like huggingface transformers for NLP.
Base library like keras/Tensorflow/torch are out of question as they are not API's rather whole framework for custom model development


texts:
I have come here from this great answer. I have come across many approaches for using cross validation and the answer to the attached question is by far explaining it the best to me. My dilemma is that now that I m not able to figure out what to use Kfold cross validation for:-

Testing overfitting?
Hyperparam tuning?
Any other use case?

and that too how? I am unable to figure out what to do with the average score that comes after kfold cross val, what to do with the folds and what to do with a model trained on k-1 folds of train data?


texts: How is the number of grid cells in YOLO determined?

texts:
I'm now learning YOLO but I don't understand how the number of grid cells is determined. Some article like this one uses 13 x 13 while others use 19 x 19 (e.g. Andrew Ng's course in Coursera). It seems that the height and width should always be the same, but how is it determined?

Is there any general guideline regarding how many grid cells should be picked up over the others?


texts: Data preprocessing, relative scale problems in features of same type

texts:
I am using Keras NN with theanos backend in Python. In my data i have multiple features of the same type but in different columns (on purpose). Here is an example.

   F1   F2
0. 8673 7490
1. 5602 5602
2. 4352 2365

When i go to process this, say MinMaxScaler 0-1, even though '1.' in both features should be the same, because both columns have different min-max values won't the normalized value be different and therefor cause problems in pattern finding? If so, what method can i use to make sure that they are of the same relative scale? Thanks for any help.


texts: Choice of weights for the Laplacian Eigenmaps algorithm

texts:
In his thesis (section 2.3.3) Belkin uses the heat equation to derive an approximation for $\mathcal{L}f$:

$$\mathcal{L}f(x_i)\approx \frac{1}{t}\Big(f(x_i)-\alpha \sum_{x_j, ||x_i-x_j||<\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}f(x_j)\Big)$$
where $$\alpha=\Big(\sum_{x_j, ||x_i-x_j||<\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}\Big)^{-1}$$.

However, I'm not sure how these considerations lead to this choice of weights for the weight matrix (which will be used to construct the Laplacian):

$$W_{ij} =
\begin{cases}
e^{-\frac{||x_i-x_j||^2}{4t}} & if\ ||x_i-x_j||<\epsilon \\
0 & otherwise
\end{cases}$$

A very vague idea of mine was that the factors $\alpha$ and $\frac{1}{t}$ don't change for a given $x_i$ so if one choses the weights like above the resulting discrete Laplacian would (let aside those two constants) converge to the continuous version.

Any ideas or tips what I'd have to read up to in order to get a better understanding?


texts: How does the multi-input deep learning work?

texts:
I have a multi-input convolutional neural network model that inputs 2 images from 2 datasets to give one output which is the class of the two inputs. The two datasets have the same classes.
I used 2 vgg16 models and concatenate them to classify the two images.

vgg16_model = keras.applications.vgg16.VGG16()
input_layer1= vgg16_model .input
last_layer1 = vgg16_model.get_layer('fc2').output

vgg16_model2 = keras.applications.vgg16.VGG16()
input_layer2= vgg16_model .input
last_layer2 = vgg16_model.get_layer('fc2').output

con = concatenate([last_layer1, last_layer2])  # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer])

My questions are:

1- Which case from the following represents how the images enter to the model?

One to One

database1-img1 + database2-img1

database1-img2 + database2-img2

database1-img3 + database2-img3

database1-img4 + database2-img4

.........
Many to many

database1-img1 + database2-img1

database1-img1 + database2-img2

database1-img1 + database2-img3

database1-img1 + database2-img4

database1-img2 + database2-img1

database1-img2 + database2-img2

database1-img2 + database2-img3

database1-img2 + database2-img4

.........

2- In general in deep learning, Does the images enter from the two datasets to the model at the same time have the same class (labels) or not?


texts:
We would like identify similar text (clauses) on a contract based on a trained corpus.

For instance:

Contract - small sample

NOW, THEREFORE, the parties hereto mutually agree as follows:

1. Scope of Services. The CONTRACTOR shall, in a proper and satisfactory manner as determined by OHA, provide all the goods and services set forth in Attachment – S1,
which is hereby made a part of this Contract.

2. Time of Performance. The performance required of the CONTRACTOR under this Contract shall be completed in accordance with the Time Schedule set forth in Attachment – S2, which is hereby made a part of this Contract.

3. Compensation. The CONTRACTOR shall be compensated according to the Compensation provision set forth in Attachment – S3, which is hereby made a part of this Contract.

4. Standards of Conduct Declaration. The Standards of Conduct Declaration of the CONTRACTOR is attached and is made a part of this Contract.

Trained clauses

We already have classified a few clauses from previous contracts. For instance:

#time_of_performance =
[
    "Under this contract the performance required to be completed in accordance with the a predefined schedule."
,
    "The completion of each phase of the project will be used to define the performance of this contract"
,
    etc.
]

Where #time_of_performance is the classification for these clauses.

Expected result

Given the contract and the trained set, we would like to get parts/ranges of the texts and its classifications:

#time_of_performance = ["2. Time of Performance. The performance required of the CONTRACTOR under this Contract shall be completed in accordance with the Time Schedule set forth in Attachment – S2, which is hereby made a part of this Contract."]

Is there a known approach for this problem or a recommended processing pipeline?


texts: Learning cost function for linear regression

texts:
(Me: Never learned calculus or advanced math and I started Stanford openclasses for machine learning. I know basic matrix calculations.)

One chapter of my course is about cost function. I have been trying to find any example calculation of it with numbers. Googling only finds the same formula everytime, and also on Octave. But I want to do the same thing first with pen+paper and without it, I cannot understand. Please give me a very simple example of using the formula with numbers. Thanks a lot.

I require a cost function calculation example for following sample dataset:

#Rooms = Rent
1 = 4000
2 = 10000
3 = 22000
4 = 30000


texts: A robust metric in the presence of class imbalance

texts:
When evaluating the performance of a multiclass classification problem, on a highly imbalanced dataset, what is the most robust metric for this purpose?

I read a paper that states:

  "Average precision is a robust metric in the presence of class
  imbalance since it excludes the ‘true negatives’ constituent in
  specificity, focusing instead on precision, or positive predictive
  value."

The confusing part, for me, is the applied methodology in this paper. They used balancing techniques (SMOTE, class_weigh, random sampling) to oversample minority classes, but they are still concerned about the evaluation metrics and 'true negatives'.


texts: Why in spectral clustering number of eigen vectors is same as number of clusters that we want?

texts: In spectral clustering we take eigenvector corresponding to K smallest eigenvalues. Then we do K means clustering on these eigenvector to get final clusters. What will happen if we take different number of eigenvectors than number of clusters we want ?