texts: Difference between using BERT as a 'feature extractor' and fine tuning BERT with its layers fixed

texts:
I understand that there are two ways of leveraging BERT for some NLP classification task:

BERT might perform ‘feature extraction’ and its output is input further to another (classification) model
The other way is fine-tuning BERT on some text classification task by adding an output layer or layers to pretrained BERT and retraining the whole (with varying number of BERT layers fixed

However, if in the second case, we fix all the layers and add ALL the layers from the classification model will be added, 1st and 2nd approaches are effectively the same, am I right?


texts:
I have a large training set of ~300GB (which is a subset of an even larger dataset ~15TB).

I am trying train a Convnet with Keras (Tensorflow backend) to do something similar to semantic segmentation.

I couldn't find any valuable resources to handle such large data. Any suggestions for best practices for such humungous data is appreciated.

Thanks!


texts: How to backpropogate error from convolutional layer with respect to the input when using multiple channels

texts:
I have been attempting to implement a Convolutional Neural Network in python and have run into a bit of a roadblock. When backpropogating the error in a convolutional layer let us say that we receive the error from the next layer as a matrix dL/dH and that we have the input for this convolutional layer, X, and the filters used for this layer's convolution, W. Taking for example the layer in the illustration in this question where we have dL/dH of dimensions (5,96,96), X of dimensions (3,100,100), and W with 5 filters of dimensions (3,5,5): I understand how dL/dH is "tiled" so that the depth of dL/dH matches that of X - allowing one to convolve dL/dH over X and calculate the error for each of the filters in W (dL/dW). What I do not understand is how to then go about calculating dL/dX.

If we pad and dilate dL/dH as per this source then the convolution of the flipped filters over the padded, dilated error produce an output with the same width and height as X, however, in this case, the filters each have a depth of 3 and dL/dH has a depth of 5 - which, as far as I understand, means that we can not convolve W over dL/dH.

Is there a way to pad or otherwise manipulate the filters or dL/dH so that the depth of the resulting convolution is the same as the depth of X?

Thanks for reading, please let me know if I need to make the question clearer or if any other details are needed.


texts: SVC classification not working at all on MNIST dataset

texts:
I'm sure I probably did something stupid but I'm trying to fit a simple SVC classifier on MNIST dataset as an example, and it completely failed by only predicting result 1 (sometimes 7 depends on how I slice the data) for all testing set.

to get data

from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1)

X, y = mnist["data"], mnist["target"]
y = y.astype(np.uint8)
X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]

to train a SVC data

from sklearn.svm import SVC

svm_clf = SVC(gamma="auto", random_state=45, probability = True)
svm_clf.fit(X_train[:5000], y_train[:5000]) # y_train, not y_train_5
svm_clf.predict_proba(X_test[0:100])

to save time I only trained on a small subset of training data, and tested on a small subset of testing data. however all prediction results are "1".

svm_clf.predict(X_test[0:100])
Out[14]:
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint8)

I tried to train a random forest on same training set

from sklearn.ensemble import RandomForestClassifier
forest_clf = RandomForestClassifier()
forest_clf.fit(X_train[:5000], y_train[:5000])
forest_clf.predict(X_test[0:100])

and at least I'm getting reasonable results and a reasonable accuracy rate without any fine tuning.

forest_clf.predict(X_test[0:100])
Out[39]:
array([7, 2, 1, 0, 4, 1, 4, 9, 2, 9, 0, 1, 9, 0, 1, 5, 9, 7, 5, 4, 9, 6,
       2, 5, 7, 0, 7, 4, 0, 1, 3, 1, 3, 6, 7, 2, 7, 1, 3, 1, 1, 7, 4, 2,
       3, 5, 1, 2, 4, 4, 6, 3, 9, 5, 2, 0, 4, 1, 9, 4, 7, 2, 6, 3, 9, 5,
       5, 4, 3, 0, 7, 0, 0, 4, 1, 7, 3, 7, 9, 7, 7, 6, 2, 7, 2, 4, 7, 3,
       4, 1, 3, 6, 4, 3, 1, 6, 1, 7, 6, 9], dtype=uint8)

y_test[0:100]
Out[41]:
array([7, 2, 1, 0, 4, 1, 4, 9, 5, 9, 0, 6, 9, 0, 1, 5, 9, 7, 3, 4, 9, 6,
       6, 5, 4, 0, 7, 4, 0, 1, 3, 1, 3, 4, 7, 2, 7, 1, 2, 1, 1, 7, 4, 2,
       3, 5, 1, 2, 4, 4, 6, 3, 5, 5, 6, 0, 4, 1, 9, 5, 7, 8, 9, 3, 7, 4,
       6, 4, 3, 0, 7, 0, 2, 9, 1, 7, 3, 2, 9, 7, 7, 6, 2, 7, 8, 4, 7, 3,
       6, 1, 3, 6, 9, 3, 1, 4, 1, 7, 6, 9], dtype=uint8)

so my question is, what am I missing here that the SVC completely failed predicting anything useful? I understand that fine-tuning is required to achieve good results but it's strange to think SVC completely failed at predicting anything and output all results regardless how I sliced trianing/testing set.

thanks


texts: Fully connected layer output explodes, but weights, gradients, and inputs all have sane values

texts:
I'm trying to train a GAN, and the architecture includes a fully connected layer before the output activation function. In my case, by the second training iteration this layer's output always explodes.

I have not been able to identify any causes through debugging. The layer implementation is extremely simple:

def fully_connected_d(inp, output_dim, name="Linear", stddev=0.02):
    shape = inp.get_shape().as_list()
    with tf.variable_scope(name):
        w = tf.get_variable("w", [shape[1], output_dim], initializer=tf.truncated_normal_initializer(stddev=stddev))
        #biases = tf.get_variable("b", [output_dim], initializer=tf.constant_initializer(0.0))
        return tf.matmul(inp, w) #tf.nn.bias_add(tf.matmul(inp, w), biases)

Here I've removed bias to rule out bias explosion as a cause, but this occurs with or without bias.

In any given run, no matter what the input is to this layer, its activation jumps from around zero to roughly around 10 within the first few training iterations:

I can't figure out why the output is in this range, as the weights and the inputs are fairly tiny:

Weights:

Inputs:

Finally, I checked the gradients of loss with respect to the weights, and they also appear to be sane before activations jump. Once activations are increased they go towards zero because I'm using $\tanh$ and its gradient is small for extremely large inputs.

Essentially, I'm unsure how (seemingly sane input) x (seemingly sane weights) yields (insane activations). Because the output jumps so quickly I think it would be a design flaw, but the layer itself is so simple that I can't see what I'm missing. Any help is appreciated.


texts:
I read for example in this answer:
Does the performance of GBM methods profit from feature scaling?
that scaling doesn´t affect the performance of any tree-based method, not for lightgbm,xgboost,catboost or even decision tree.
When i do feature scaling and compare the rmse of a xgboost model without and with minmax scaling, i got a better rmse value with feature scaling. Here is the code:
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error as MSE
import math
from math import sqrt
import pandas as pd
import numpy as np
from xgboost import XGBRegressor
import xgboost as xgb

data = pd.read_excel(r'C:...path.xlsx')
X = data.drop(['colA'], axis=1)
y = data['colA']

scaler = MinMaxScaler()
scaler.fit(X)
minmax_scaled_X = scaler.transform(X)
minmax_scaled_X
y = np.array(y).reshape(-1, 1)
scaler.fit(y)
minmax_scaled_y = scaler.transform(y)

from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(minmax_scaled_X, minmax_scaled_y, test_size =0.3, random_state=0, shuffle=True)

xg_reg = xgb.XGBRegressor(objective ='reg:squarederror', colsample_bytree = 0.7, learning_rate = 0.05,
                max_depth = 8, min_child_weight = 4, n_estimators = 600, subsample = 0.7)

xg_reg.fit(xtrain,ytrain)
preds = xg_reg.predict(xtest)
rmse = sqrt(MSE(ytest, preds))
print(rmse)

the result with min max scaling is 0.003, while the rmse without is about 3.8. I did the same with simple decision tree and got always a better result with minmax scaling.
Where is my mistake?
In other posts like the link above, answers are about that it is not good to scale when using trees.
Can I say, that min max scaling does have a positive effect on the rmse on my data?


texts:
i create a decision tree model using c4.5 algorithm. After create the model, i evaluate model using 10 fold cross validation and classify model using test data to get accuracy. And then I run prune tree with REP.

My question is what data should i use to compare accuracy (after prune and before prune) to decide to remove the leaf? and from where i get the accuracy? using cross validation or classify test data?


texts: what kind of distribution is followed by word and sentence vectors generated by TFIDF ,word2vec,glove,bert,flair?

texts:
what kind of distribution is followed by word or sentence embedding vectors generated by TFID or pretrained models like word2vec,glove,bert,flair ?
is it continuous or discrete or any other distribution ?
I want to cluster sentence using Gaussain mixture model so im not sure if the distribution of vectors generated by TFIDF or pretrained models like word2vec,glove,bert are continous or discrete ?
since Gaussian distribution models continuous data well im not sure if sentence vectors are of continuous distribution , can GMM be used  for text clustering .


texts: Train a GAN on "before and after" images of dental surgeries

texts:
I want a GAN to train on "before and after" images of dental surgeries; so that it can generate "after" pictures for fresh patients.

Input images are like these: https://img.webmd.com/dtmcms/live/webmd/consumer_assets/site_images/article_thumbnails/slideshows/cosmetic_dentistry_slideshow/650x350_cosmetic_dentistry_slideshow.jpg

Now being a web developer, I am getting into AI.

I have read a hell lot on Basic ML, AI, GANs, VAE, StyleGANs, etc.

Made small projects on Tensorflow, etc.

I have my basics and theoretical knowledge in place. But I dont know how to get started practically on the above idea I have.

What should be my approach?

Following are my questions:

How many "before and after" images will I need?

What architecture should I use? i.e. StyleGAN, VAE?

As I will need my custom model; training a fresh model would need a good GPU. What platform should I go for? Colab, Floyhub, or something else

One approach that comes to my mind is that -
Process the image, do the color segmentation or something.

Add a static tooth image to area where tooth is missing. Then feed this image to a GAN, so that it can polish that image to look real.

Please guide me in the right direction. Sorry if any of my question is nonsense.


texts: Multiclass Classification Task - Performance on Each Class Compared to Chance?

texts:
As a part of a classification task, a classifier has decided whether different books belong to class A, B or C (which are imbalanced) by looking at certain feature of the book.

I have calculated different classification performance metrics (e.g. accuracy, (macro and micro) recall, precision, f1, etc.

I have also assessed the overall performance of the classifier compared to chance using kappa and MCC.

What I want now is the performance on each class (A,B,C) in relation to chance.

Up to now I know that the performance of the classifier overall is not better than chance. I would like to know whether:

the performance of the classifier on class A is better than chance.
the performance of the classifier on class B is better than chance.
the performance of the classifier on class C is better than chance.
Does it make sense to run individual kappa/MCC for each of the classes?


texts: Difference between Global Outlier and Contextual Outlier?

texts:
I am studying "Data Mining: Concepts and Techniques" by Han, Kamber & Pei. In Chapter 12 "Outlier Detection", they have stated that there are 3 types of outliers:

Global Outlier - deviates significantly from the rest of the data set
Contextual Outlier - deviates significantly with respect to a specific context of the object
Collective Outlier - the individual data objects may not be outliers, but the objects as a whole deviate significantly from the entire data set.

According to the authors:

  “The temperature today is 28 C. Is it exceptional (i.e., an outlier)?” It depends, for example, on the time and location! If it is in winter in Toronto, yes, it is an outlier. If it is a summer day in Toronto, then it is normal. Unlike global outlier detection, in this case, whether or not today’s temperature value is an outlier depends on the context—the date, the location, and possibly some other factors. In a given data set, a data object is a contextual outlier if it deviates significantly with respect to a specific context of the object. Contextual outliers are also known as conditional outliers because they are conditional on the selected context. Therefore, in contextual outlier detection, the context has to be specified as part of the problem definition.

My question : Is not Contextual Outliers the same as Global Outliers where the original feature vector describing each data object may have to be extended to incorporate some other attributes (information regarding the "context") ?
Obviously, if the data objects do not contain enough attributes, they can never be classified as outlier ! This holds true for global outliers as well. So it makes no sense to me to form a new category of outliers as Contextual Outliers.


texts: Tableau can be used as a big datastore as well?

texts:
Company wants to use Tableau solution for visual reporting. The inputs are various files. The plan is to use latest SQL Server as database and server.
Q. Does Tableau has their own database ? Can perform own reporting without SQL Server?


texts: "Recursive ConvNets for Dummies" Library

texts:
I've look at the questions on here regarding the different python libraries around for deep learning and neural nets. They include:

Keras
Caffe
Lasagne
PyLearn2
Deepy
Theano
Torch

My understanding is that Keras and Lasagne require the user to have varying degrees of interaction with Theano.

I'm looking for the one that would be the easiest to start testing an algorithm I have already developed an apply deep learning on it in a packaged way. I am not looking to extend, modify or tweak anything. I certainly won't be doing any research in the field with it.

The only advanced feature I would need would be GPU support.

In short, what would be the best "dummy" library to test recursive convolutional neural nets?


texts: Classification Algorithm for Segmented Regression Time Series

texts:
I have Segmented Linear Regression dataset in JSON format like this:
var series = [
    {x:0,w:10,m:0.5,b:50},
    {x:10,w:5,m:0.6,b:40},
    {x:15,w:20,m:1,b:100},
    // So on....
];

the Dataset formatted for typical line equation: y=mx+b
Here is a JSFiddle link that describes the situation with canvas element: https://jsfiddle.net/digerkam/7krb50z8/
My problem is that classifying/grouping the similar segments like the yellow one:

I tried many things, I researched eveywhere I found, I thought so many times but no luck to find the alghorithm, or any equation for detect continuity/similarity...
Espacially detecting changes between A and B is so confusing my mind, because it has very similar characteristic like noizes in C part.
So how can classify this dataset as expected? Which alghorithm can solve this mystery? Thank you!


texts: How to think about prediction error that is not convex in hyperparameter, or over the course of training

texts:
Take the following case of a hyperparameter and prediction error:

Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- too high and you're underfit and too low and you're overfit.

I keep getting nonconvex plots like the one above when doing cross-validation.

I guess this just points to a lot of noise during training -- I've got a lot of variables for a modest sample size, and I need to regularize heavily to get a good model.  But still I'm a little bit unsure whether this sort of thing might point to a bug in some aspect of my implementation.

Has anyone come across this sort of thing before?  And did you just shrug off the nonconvexity and go with the model that minimized the prediction error?

If so, that begs the question: why not just compute a prediction error at each update during training, saving any set of weights that minimizes prediction error -- even if the model is nowhere near converged.  Basically letting the noise work in your favor.  This seems appealing, because sometimes I get really low prediction errors early on, only to have them evaporate as the loss function declines.

This seems horribly unprincipled, but I ask myself "why should I care if it is"?  And "is it unprincipled anyway?"


texts: Optimization problem: Given Beta Bounds Maximize sharpe

texts:
I would like to maximize a portfolio's Sharpe Ratio while keeping Beta in bounds.
Could anyone supply a calculation please?

***************Starting Metrics**********************
symbols          : ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H']
Beta             : [ 0.09  1.2   2.1  -1.3   1.1   0.3   0.9   1.2 ]
Sharpe           : [-1.7  5.4 -0.3  0.8  0.4  2.7  0.9  1.9]
optimized weights: [x0, x1, x2, x3, x4, x5, x6, x7]

***************Unadjusted Portfolio**********************
AvgBeta unadjusted  : 0.69875
AvgSharpe unadjusted: 1.2625000000000002

***************Ajusted Portfolio**********************
Beta bounds: -.3 < x <.3
Maximize sharpe: X


texts: Database on time to churn in telecomunication

texts:
I would like to research about time to churn1 in the telecommunication market. Does anyone have a link to such a database? The only ones I found did not include the time of churn, but only if a customer is labeled as churn or non-churn, what I would need is time to event data.

The papers I researched all seemed to use private databases.

1churn is defined here as the moment in time, where a customer quits the service that he/she book from the service provider.


texts: Train an LSTM neural network with time series containing seasonal and trend

texts:
I am working on a project for predicting the number of DNS queries from the site:
DNS queries statistics. The data I use is minutely data, which means the number of DNS queries of every minute.

If you look at the number of DNS queries from South Korea or any other countries, it has the seasonality and trend characteristics: increase then decrease day by day.

The requirement for me is: given the number of DNS queries of every minute, then for the given data of 20 previous minutes, predict the number of DNS queries in the next 20 minutes.

My problem is: my trained LSTM could not detect these factors, it always predicts my data to decrease everytime.

I have employed some hand-defined features like the minute of the day, period of the day like morning/afternoon... But it still keeps the same problem.

So I want to ask if there is any possible improvement to make my LSTM to understand when it should decrease and when it should increase with the seasonal data?


texts: I made a pipeline with standard scaler and k means .When I fit the pipeline to the training data, Does the standard scaler just fits or fits and transforms the training data?


texts: Intersection Over Union / Bounding Box for multiple bounding boxes in an image

texts:
My image has 4 ground truth bounding boxes and the prediction just has 3 bounding boxes

I need to calculate the quality of bounding boxes predicted.

I know how to calculate IoU / IoBB for 2 bounding boxes ( 1 pred and 1 ground )

But whats the algorithm to decide which pair to calculate for?

Would it be correct to calculate it for all pairs of prediction/ground truth
exhaustively?

I am a bit confused here


texts: RL Invertible Value Function approach - why it prevents rewards from exploding?

texts:
Authors of "Recurrent Experience Replay in Destributed RL", page 3, use the function $h$ to prevent rewards from exploding:

$$h(x) = \operatorname{sign}(x)(\sqrt{|x|+1} -1) + \epsilon x$$
where $\epsilon$ is a very small number, for example $0.0001$ and $x$ is the score (aka reward). h re-scales the rewards the network receives.

This is different to a more common "reward-clipping" technique.

Question 1 (most important):

The Q-vaue formula looks like this:

$$y = h \bigg( nstepReward + \gamma^n \cdot h^{-1}(Q(s_{t+n}, a^*)) \bigg)$$

Why do authors pass the Q-scores through the inverse (exponential) function and then through the squashing function $h$? The effect of this is that Q-target-scores remain unchanged, because composition of a function and its inverse is identity. So to me, the $Q$ is still a big source of threat of "exploding rewards", because it's left untouched. Why do we keep it untouched?

It looks like the whole formula merely squashes $nStepReward$, but why not just squash everything, including Q to prevent exploding rewards? In other words, why not to have:

$$y = h \bigg( nstepReward + \gamma^n Q(s_{t+n}, a^*) \bigg)$$

Question 2:

Why to use $\epsilon x$ in the formula for $h$


texts: Grouping profiles strings having the same words, but occurring out of order

texts:
I have a dataframe containing a column of profile types, which looks like this:
0                                    Android Java
1                  Software Development Developer
2                            Full-stack Developer
3                      JavaScript Frontend Design
4                          Android iOS JavaScript
5                             Ruby JavaScript PHP

I've used NLP to fuzzy match similar profiles, which returned the following similarity dataframe:
left_side                       right_side                  similarity
7   JavaScript Frontend Design  Design JavaScript Frontend  0.849943
8   JavaScript Frontend Design  Frontend Design JavaScript  0.814599
9   JavaScript Frontend Design  JavaScript Frontend         0.808010
10  JavaScript Frontend Design  Frontend JavaScript Design  0.802881
12  Android iOS JavaScript      Android iOS Java            0.925126
15  Machine Learning Engineer   Machine Learning Developer  0.839165
21  Android Developer Developer Android Developer           0.872646
25  Design Marketing Testing    Design Marketing            0.817195
28  Quality Assurance           Quality Assurance Developer 0.948010

While this has helped, taking me from 478 unique profile to 461, what I'd want to focus on are profiles like this:
Frontend Design JavaScript  Design Frontend JavaScript

The only tool I've seen which looks to address this problem is difflib?
My question is, what other techniques would be available so as to go through and standardize these profiles that consist of the same words, but out of order, to one standard string.
So desired output would be, taking a string containing "Design", "Frontend" and "JavaScript" and replacing it with "Design Frontend JavaScript".
Right now, I'm merging my original dataframe with the similarity dataframe to replace all occurrences of profile string on the right_side with the left_side, but that means I'm replacing the right_side below ("Java Python Data Science") with the left_side below ("JavaScript Python Data Science").
53  JavaScript Python Data Science  Java Python Data Science

Any help would be greatly appreciated!!!


texts: Finding Patterns in Error Reports - How to Approach?

texts:
I am new to data science and have been doing research to familiarize myself and try to find a solution to my problem but I have not come across anything that seems to fit. I am looking to learn and use Python for this process.

My situation is as follows:
I have multiple error logs for the cars I am testing on. Every time there is an error (there are multiple types) I have a log that has many different characteristics of the car at that time of the error. These include things like time of day, speed, location, temperature, and many more.
EDIT: If needed I could also have data for normal operation as well.

My goal is to find patterns in the characteristics that have a correlation with and could be contributing to the error. I am not looking to develop a model that will predict future errors. I am only focused on seeing the patterns themselves and understanding them, and having it become more accurate the more data I give it when we encounter the error again.

For example, I want to feed it all my data for one type of error, and have it tell me something along the lines of "When speed is above x you will likely get this error", "When car is in reverse and running at night you will likely get this error", "There is no correlation between outside temperature and this error", etc. And then proceed to do this for every type of error.

Some of the variables could be dependant on each other. Some of the variables might not be relevant/meaningful.

How can I do this in Python? Any guidance is appreciated. I have tried to be as specific as I can and will update my post if further details are needed.


texts: WEKA Random Forest and numerical attributes

texts:
I am working with Random Forests in Weka. I thought the ID3 algorithm is used to find the best split attribute at each level. But after reading a bit I noticed that ID3 can not handle numerical attributes (one needs for instance C4.5 for that). Random Forests in Weka do have no problems with numerical features as input...
How are they handeled?

(Small additional question: Is Weka still something you can work with today or do I need to switch the tool for current research? Weka seems a bit "old-fashioned" sometimes...)


texts: Is consecutive multiple transformation of Y variable a valid approach statistically?

texts:
I am going to run a regression analysis. But my response variable is highly right-skewed

Firstly I used a log(x+1) transformation and it increased the situation dramatically. But there is still right-skewness in my transformed data.

My question is; can I apply another transformation to my already transformed variable?

For example I have applied a sqrt transformation to my log(x+1) transformed variable and it is now very close to normality.

Is it valid in terms of data science and statistics? Can I apply such multiple data transformations to my data?

Thanks


texts: Looking for a ml algorithm to predict a path based on millions of data

texts:
I have a dataset with following data format:

3 -> a -> b -> c -> d -> ikd
a -> c -> 3 -> dk -> 2 -> l2i

each row represents a path from start to end, let's take the first row as an example, the start point is 3 and the endpoint is ikd. I have millions of row like that. And each row may have a different length. What I want to do is let users input a path like above format and I predicts a new path based on the existed dataset. The idea is to find the closest path for users. Each node on the path may have a weight to indicate how important this node is. So the closest path should respect this weight. Is there any ML algorithm for that kind of problem? I am new to ML so I need some guidance on solving this problem.

--EDIT--

There is no specific meaning for the type of each node like number, letter, etc. All these node are of type string and users' input are from an existed string list.


texts: Evaluate imbalanced classification model on balanced testing sample

texts: Why it would be too optimistic to compute presicion, recall and f1-score to evaluate a model trained for imbalanced classification on a balanced testing sample ?


texts: What are the model configuration parameters in tensorflow object detection API?

texts:
In the Config file there are some parameters which I don't understand properly . I will mention them here.

first_stage_features_stride - Is this the ration of input/output ?

height_stride (In the first_stage_anchor_generator) - What is this ?


texts:
Can anyone tell me good approaches to detect blinking light in a video? The background may change constantly. As of now,  the color would remain same but the intensity and angle of vision can change. There can be multiple glowing lights too but I want to identify only the blinking one.

My approach till now:
Used a high density mask to retain only the blinking flash light and then observe the change in intensity over a time frame,  but unfortunately the noise in non blinking videos are so much that they too are being classified as blinking.
As of now,   I am going to train on more data.  Can you think of any other approach? Rule based also is okay to me.


texts: Predicting parameters of simple configured trajectories using RNN

texts:
What I'd like to do: predict orbital elements given an input observation sequence in 2D, that is

$$input = X = [position_{t_{0}}, position_{t_{1}},\ ..., position_{T}]$$
$$output = y = parameters\ of\ observed\ orbit$$

Positions are (x, y) coordinates and the parameters are a handful of values that describe an ellipse in a 2D plane.

What I've done so far: I used a very, very basic LSTM-stack model with Keras that I trained with $([X_{o_{i}}^{1},X_{o_{i}}^{2},\ ...], y_{o_{i}})$ pairs where $o_{i}$ are the 2D orbits. $[X_{o_{i}}^{1},X_{o_{i}}^{2},\ ...]$ is just a couple dozens of simulated observations of $o_{i}$. I assumed it is a regression problem so I replicated some time series prediction models I found on the internet: those usually take a two or three dimensional vector as input sequence and give a 1D value prediction of the next step. I also Googled configured trajectory LSTM or shape parameter prediction RNN.
The Issue: Results are bad, my ellipses do not have the expected shape. I have absolutely no idea if this is the RNN I should be using and if it is, what layers I should or shouldn't add. I couldn't find relevant literature on the internet and pages that tackled configured trajectories were talking about predicting the next position of a car, while in my case I'm trying to compute an estimation of the whole trajectory.
What I'm looking for: Really anything related to determining geometrical parameters based on sequential observation or relevant literature about how to design models that use LSTM for this kind of problems. Any advice from LSTM-RNN-savvy people or orbital-element-estimation-specialists would be greatly appreciated.


texts: Predictive Model for Customer Payment Pattern

texts:
I am working for a logistics firm and there are approx. 750+ customers who avail our services. I am in the process of building and generating some insight for the business based on the payments made by these customers for the last 1 year. Some make payment on time whereas some are late & some are extremely late. Could you please advice which modeling technique or statistical approach would be best in this case.
I can think of creating clusters based on the payment history and highlights & placing all defaulters in one cluster where our company can focus.
PLease suggest.


texts: Obtain a model where each feature vector is past few samples and labels are future few samples?

texts:
So, I have this data set where each instance is made of past 20 samples of 2 variables. Labels are next five samples. So every instance looks like this:

Instance1_feature = [s1_var1 s2_var1 ... s20_var1 s1_var2 s2_var2 ... s20_var2]
Instance1_label = [s21_var1 s22_var2 ... s25_var5 s21_var2 s22_var2 ... s25_var2]

Instance2_feature = [s2_var1 s3_var1 ... s21_var1 s2_var2 s3_var2... s21_var2]
Instance2_label = [s22_var1 s23_var2 ... s26_var5 s22_var2 s23_var2 ... s26_var2]

All the data are categorical. var1 takes value in {1,2,3,4,5}; var2 takes value in  {6,7,8,9,10}
I am familiar with machine learning. Hence I am looking for a way to develop a model to predict labels for a new instance. I have reasons to believe that by looking at the past samples, it is possible to predict future samples. Its like stock prices.

Any help, links to code samples will be highly appreciated.

EDIT This is a dataset I collected while driving. I want to see if I can predict next 5 seconds of my driving, given I have past 20 seconds of data. That's why I created the feature vectors and labels like this. The values  {1,2,3,4,5} and {6,7,8,9,10} are categorical, as in the variables are split into segments.


texts: predict with Multinomial Logistic Regression

texts:
If I execute the following code I have no problem:

require(foreign)
require(nnet)
require(ggplot2)
require(reshape2)

ml <- read.dta("https://stats.idre.ucla.edu/stat/data/hsbdemo.dta")
ml$prog2 <- relevel(ml$prog, ref = "academic")
test <- multinom(prog2 ~ ses + write, data = ml)
predict(test, newdata = dses, "probs")

but if I try:

require(caret)
ml <- read.dta("http://www.ats.ucla.edu/stat/data/hsbdemo.dta")
ml$prog2 <- relevel(ml$prog, ref = "academic")
test <- train(prog2 ~ ses + write,method="multinom" ,data = ml)
predict(test$finalModel, newdata = dses, "probs")

it returns Error in eval(expr, envir, enclos) : object 'sesmiddle' not found, why?


texts: Pandas - Create a Integer Primary Key based in string columns - python

texts:
I need to create a primary key based in string columns in my dataframe
Month       Name                   ID
01/01/2020  FileName1 - Example    1
01/02/2020  FileName2 - Example    2
01/03/2020  FileName3 - Example    3

I'm using the hash, but its generating the largest values, I would like that ID was the integer numbers.
This is my code
all_data['unique_id'] = all_data._ID.map(hash)

where  _ID is :
all_data['_ID'] = all_data['Month'].astype(str) + all_data['Name']

This group by return 0 for all rows
all_data['sequence']=all_data.groupby('_ID').cumcount()


texts: xgboost feature selection and feature importance

texts:
when you create the new feature for data analysis
for linear regression, it is clear that the feature has to be linear with other features is better

but for xgboost what is the guideline to make a new feature.
1. what is the guideline to new features for xgboost
2. when xgboost split out the feature importances. is it better just drop the non-important features?
3. if I want to know about this related field what should I study please let me know the link and books or other resorucees


texts: I was wondering mathematically how Precision Recall curve is plotted. How is this curve useful?


texts:
I have recently started kearning CNN and I coukdnt understand that why are we using a 2D kernel like of shape (3x3) for a RGB data in place of a 3D kernel like of shape (3x3x3)?
Are we sharing the same kernel among all the channels because the data would look the same in all the channels?


texts: Adding extra (meaningful) features does not improve model performance

texts:
I am struggling with confusion matrices and their outputs. I thought to follow all the steps right, but unfortunately it seems that something is not going well.
I had a dataset built and labelled on my own. It shows a class imbalance so I decided to apply undersampling and oversampling, looking at F1-score and Recall as in many papers and on the web.
The steps were:

split data in train and test (80/20)
apply resampling only on train set
apply pre-processing algorithm (BoW, TF-IDF, ...)
use different classifiers to get results
look at performance using confusion matrices (or alternatively ROC)

I tried with different features: in one dataset with less features engineering, i.e., using only features from Text, I got a maximum value of F1-score equal to 68%. With more features, that I thought to be significant for improving the model, I am getting max 64%, that is weird considering the problem (email classification for spam detection).
In theory, if I extract features only from text, I get a better score rather than extracting also features from email addresses.
I would like to ask you for some tips and suggestions, if you have any, as I think that this cannot be possible, as the expected results should be higher in the second case, when I consider also information from email address (number of dots, suffix, registration date,...).
I am thinking at a problem of overfitting or some other issues with model building.
I would appreciated if you could tell me your thoughts on this.
Thank you for all your help.


texts: Locally Weighted Regression (Loess) - Robustifying iterations

texts:
As a self study exercise I am trying to understand the implementation of locally weighted regression (Loess) in python. Alexandre Gramfort (Sklearn developper) provides the following code on his github page.

def lowess(x, y, f=2. / 3., iter=3):
    '''....the number of robustifying iterations is given by iter. The
    function will run faster with a smaller number of iterations'''

    n = len(x)
    r = int(ceil(f * n))
    h = [np.sort(np.abs(x - x[i]))[r] for i in range(n)]
    w = np.clip(np.abs((x[:, None] - x[None, :]) / h), 0.0, 1.0)
    w = (1 - w ** 3) ** 3
    yest = np.zeros(n)
    delta = np.ones(n)
    for iteration in range(iter):
        for i in range(n):
            weights = delta * w[:, i]
            b = np.array([np.sum(weights * y), np.sum(weights * y * x)])
            A = np.array([[np.sum(weights), np.sum(weights * x)],
                          [np.sum(weights * x), np.sum(weights * x * x)]])
            beta = linalg.solve(A, b)
            yest[i] = beta[0] + beta[1] * x[i]

        residuals = y - yest
        s = np.median(np.abs(residuals))
        delta = np.clip(residuals / (6.0 * s), -1, 1)
        delta = (1 - delta ** 2) ** 2

    return yest

Question

What I don't understand is the part related to the 'robustifying iterations'. How does this work and how could I learn more about this ?

Vectorized implementation

For those who might be interested here is my attempt at describing the vectorized implementation mathematically.

Consider the 1D case where $\Theta = [\theta_0, \theta_1]$ and $x$ and $y$ are vectors of size $m$. The cost function $J(\theta)$ is a weighted version of the OLS regression, where the weights $w$ are defined by some kernel function

\begin{aligned}
 J(\theta) &= \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right)^2
 \\
 \frac{\partial J}{\partial \theta_0} &= -2 \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right)
 \\
 \frac{\partial J}{\partial \theta_1} &= -2 \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right) x^{(i)}
\end{aligned}

Cancelling the $-2$ terms, equating to zero, expanding and re-arranging the terms:
\begin{aligned}
 & \frac{\partial J}{\partial \theta_0} = \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right)  = 0
 \\
 & \sum_{i=1}^m w^{(i)}  \theta_0 + \sum_{i=1}^m w^{(i)}  \theta_1 x^{(i)}  = \sum_{i=1}^m w^{(i)} y^{(i)}  &\text{Eq. (1)}
 \\
 \\
 & \frac{\partial J}{\partial \theta_1} = \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right) x^{(i)}  = 0
 \\
 & \sum_{i=1}^m w^{(i)}  \theta_0 + \sum_{i=1}^m w^{(i)}  \theta_1 x^{(i)} x^{(i)}  = \sum_{i=1}^m w^{(i)} y^{(i)}  x^{(i)} &\text{Eq. (2)}
\end{aligned}

Writing Eq. (1) and Eq. (2) in matrix form $\mathbf{A \Theta = b}$ allows us to solve for $\Theta$
\begin{aligned}
 & \sum_{i=1}^m w^{(i)}  \theta_0 + \sum_{i=1}^m w^{(i)}  \theta_1 x^{(i)}  = \sum_{i=1}^m w^{(i)} y^{(i)}
 \\
 & \sum_{i=1}^m w^{(i)}  \theta_0 + \sum_{i=1}^m w^{(i)}  \theta_1 x^{(i)} x^{(i)}  = \sum_{i=1}^m w^{(i)} y^{(i)}  x^{(i)}
 \\
 & \begin{bmatrix} \sum w^{(i)} & \sum w^{(i)} x^{(i)} \\ \sum w^{(i)} x^{(i)} & \sum w^{(i)} x^{(i)} x^{(i)} \end{bmatrix}  \begin{bmatrix} \theta_0 \\ \theta_1 \end{bmatrix}   = \begin{bmatrix}  \sum w^{(i)} y^{(i)} \\  \sum w^{(i)} y^{(i)} x^{(i)} \end{bmatrix}
 \\
 & \mathbf{A} \Theta = \mathbf{b}
 \\
 &  \Theta = \mathbf{A}^{-1} \mathbf{b}
\end{aligned}


texts: Predict the average temperature for next 30 years

texts:
Objective: I want to predict the average temperature for next 30 years.

Q1: What type of dataset is suitable for this (what columns should it contain)

Q2: What are the independent variables for average temperature?

Q3: Let's say I trained the model with 30% data. Does that mean predicted 30% is for next 30 years? Or is it present data? If not how do I predict the average temperature for next 30 years with the trained model?


texts:
I would like to hear some views on a problem I have with my dataset (which I presume to be a common one).

Let's say I have the following dataset

SKUID   PRODUCT QTY        MFGDT      ......     EXPDT     SUPPLIERID SUPPLIERPH CUSTOMFIELD
FD001     MILK    3   12/01/18 14:12:02 ...    18/01/18    SV01    04053XXXX
FD002    CREAM    3   12/01/18 14:12:02 ...    18/01/18    SV01    04053XXXX
FD003   CHEESE    5   12/01/18 14:12:02 ...             18/01/18    SV01    04053XXXX
FD004   BUTTER    2   12/01/18 14:12:02 ...    18/01/18    SV01    04053XXXX
FD005 ICECREAM    1   12/01/18 14:12:02 ...      SV01   04053XXXX

The dataframe is of shape (123078, 199) and there are few records where the field values are jumbled.

When reading this csv using pandas, I used the error_bad_lines=False attribute to skip the lines where there is a mis-match in the field.

However, I was wondering if there is a way to FIX the data by some means (say, pattern matching with the previous items in the column; based on the dtype etc.)

How do we generally handle a scenario where every record is crucial (or say inter-dependent) and there is mismatch in the field?


texts:
With companies and governments hungry for all the data about people, I was wondering if it was possible to gain some privacy by drowning relevant information in a sea of random data. For example a browser extension which keeps searching for random words and expressions in the background. Maybe sending generated emails to generated addresses. Or producing made up location data on my phone.

Would these measures help at all or the data miner AIs would just see right through them?


texts: Closed Domain Question Answering which doesn't answer Questions

texts:
I've been exploring Closed Domain Question Answering Implementations which have been trained on SQuAD 2.0 dataset. Ideally, it should not answer questions which the context text corpus doesn't contain answers to. But while implementing such models using the Haystack repo or the FARM repo, I'm finding that it always answers these questions even when it shouldn't. Is there any implementation available that takes into account the fact that it shouldn't answer questions when it doesn't find a suitable answer.
References:

https://colab.research.google.com/github/deepset-ai/haystack/blob/update-tutorials/tutorials/Tutorial3_Basic_QA_Pipeline_without_Elasticsearch.ipynb#scrollTo=KS4nTwxIbRb6

https://github.com/deepset-ai/FARM

https://github.com/deepset-ai/haystack

https://huggingface.co/deepset/bert-large-uncased-whole-word-masking-squad2

https://colab.research.google.com/drive/1UrKlHlf68hD3wwQDTctx2cQs6FMUxLLH?usp=sharing#scrollTo=J4jxYsxaG77O


texts: SGDClassifier : Get the weights and bias for each training step

texts:
I would like to get the coef_ (weight vector) and the intercept_ (bias) for each iteration or training step for the sklearn.linear_model.SGDClassifier.

However, I cannot find anywhere such a functionality? Is this even possible with sklearn?

In tensorflow I could add a callback. Is there something similar available for SGDClassifier?

Thanks :)


texts: Vertical lines of overlapping points in Seaborn Swarmplot

texts:
When I plot my data to see its distribution via Swarmplot, these two vertical lines that seems to even overlap occur in the figure.

It seems to occur even around values that are not that dense, such as around -2.5.
I feel like I might be missing how Swarmplot works but anyone knows the explanation?


texts: I'm new in DeepLearning. I'm not good at understanding and commenting on graphics.Can you help me with these graphs


texts: unique predictions for "multi-label multi-output" classification task

texts:
Let’s assume that four participants (A, B, C and D) take on five sport-challenges (e.g. swimming, running, ...). Our goal is to predict the placement of each participant for each challenge. Moreover, let’s assume we have appropriate predictors. We know that each placement (1 to 4) is unique for each challenge (only one winner ...).

My questions:

I think this prediction task is a multi-label multi-output classification, right?

Are there any algorithms, which provide unique forecasts (for each class). In other words, the algorithm should classify each person in one unique class (1 to 4). Obviously, we know a priori that same placements are very unlikely.

Thank you!

Greets


texts: Does the choice of normalization change dramatically the result of a KMeans

texts:
I'm using a KMeans to get the profile of several users according to several columns (I'm working with RStudio).

To analyze my clusters, I decided to realize a radar chart, so I decided to use feature scaling : x-min(x)/diff(range(x)), to have my values in [0,1] (to get a quite good idea of my data per cluster).
However, since there are multiple choice for normalization, I was wondering if doing my analysis with another choice for normalization - for instance : x-mean(x)/sd(x) - would give me the same results (in a general way at least)

Or am I completly wrong for considering my scaled data and should I use my unscaled data in my radar chart ?


texts: How to implement random cropping during training?

texts:
I'm developing a U-net like model which segments the damaged tissue of the brain between two time-points in Multiple Sclerosis patients. The model is given the baseline and follow-up images as x and the segmentation mask as y. The images are 3D (192, 218, 192) and the model input size is (128, 128, 128) and is being trained by the dice loss. The model is based on this paper: https://www.sciencedirect.com/science/article/abs/pii/S0895611120300732
What I'm currently doing is a centre crop for each image before the training, and the training dice score seems to be learning properly, but in the validation data it is not.
I have read that random cropping 10 times or so helps to reduce overfitting, but I don't exactly know how to implement it. I have written this function to do the random cropping:
def random_crop(img_bl, img_fu, mask, width=128, height=128, depth=128):
    x = random.randint(0, img_bl.shape[1] - width)
    y = random.randint(0, img_bl.shape[0] - height)
    z = random.randint(0, img_bl.shape[2] - depth)
    img_bl = img_bl[y:y + height, x:x + width, z:z + depth]
    img_fu = img_fu[y:y + height, x:x + width, z:z + depth]
    mask = mask[y:y + height, x:x + width, z:z + depth]

    return img_bl, img_fu, mask

Do I just apply this function 10 times before the training for each image? Or is there a way to include the random cropping inside the model and overlap the predictions of these 10 subvolumes?
Any tips, links to articles or insights are greatly appreciated. Thanks!


texts: How to predict next year's gross revenue given this year's data?

texts:
I have the following dataset (.csv format) which contains:

100 columns: $\textbf{Period}$ (in years, e.g. 2017, 2018, ..., 2028), $\textbf{Gross Revenue}$, $\textbf{Region}$ (e.g. APAC, NEMEA, etc), 97 other columns as $\textbf{factors}$ (e.g. # of customers, # of goods produced, etc).
1,000,000 rows

where for $\textbf{Period}$, only 2017 is actual data, and 2018, ..., 2028 are $\textbf{planned}$, in the sense that data from these years are what the company expects to commit to and receive in these years in the future.

Now, I want to predict the gross sales for each $\textbf{Region}$ in 2018.

I tried using regression with stepwise, but the linear model that resulted from the stepwise had NA values for a majority of the predictors, and also the probability arising from the t-test.

I've consulted a few of my pals whom are working in the field of data science & analytics, and they all told me that it is impossible to predict for 2018..

Is it possible to do so given the state of the data? Some insight on this will be deeply appreciated!


texts: The differences between BNf and JSGF in NLP?

texts: I wonder what the differences are between the BNF(Backus-Naur Form) and JSGF(Java Speech Grammar Format)? The former is a kind of context-free grammar taught in CS224, but I learned that the latter is also being used. Could anyone tell me which one is better and what are their differences?


texts: How much text is enough to train a good embedding model?

texts:
I need to train a word2vec embedding model on Wikipedia articles using Gensim.

Eventually, I will use the entire Wikipedia for that
but for the moment, I'm doing some experimentation/optimization to improve the model quality and I was wondering how many articles would be enough to train a meaningful/good model? How many examples needed for each unique word in vocabulary?


texts:
Consider the network :

The learning rule (weight update )used for the hidden-to-output weights is:

The learning rule (weight update )used for the input-to-hidden weights is:

So what about the bias weights? The above diagram doesn't contain any bias weights. So which rule to use to update the bias weights?


texts: How to calculate the memory usage of a deep LSTM network?

texts:
I was trying to estimate the memory usage for my LSTM network by referring to an examples of CNN memory usage calculation at http://cs231n.github.io/convolutional-networks/#computational-considerations . The LSTM network architecture is as follows:

My input data is of dimension (None,4,34569) float 32 values and With batchsize=8, the following are my estimations of memory usage

Parameters: 482646529*(4/2^20)*3=5.4GB

Given as No_of_parameters *memory_per_float32*factor.

The factor of 3 is to consider the  memory to hold parameters, gradients in BP and optimization cache(if adam, RMSprop etc are used).

Activations: No_of_activations*(4/2^20)x2 =129857*(4/2^20)*2 =1MB

The factor of 2 since the memory must hold the activations for backward pass

Here the No_of_activations are calculated as follows:

lstm_1: 6*4*3072=73728

(used '6' because of forget,input,output gate, cell state, cell candidate,output activation and '4' for the length of time sequence)

dropout_1: 4*3072=12288

lstm_2: 6*4*1024=24576

dropout_2: 4*1024=4096

lstm_3: 6*4*512=12288

dropout_3: 4*512=2048

time_distributed_Dense_1: 256

dropout_4 :256

time_distributed_Dense_2 : 128

dropout_5: 128

time_distributed_Dense_3: 64

time_distributed_Dense_4: 1

Total number of activations: 129857

Miscellaneous : 4*34569*(4/2^20)=0.52MB

Total Memory :parameters_memory+activations * batchsize+Miscellaneous * batchsize = 5.4GB + 1MB *8 +0.52MB *8 ~=5.4GB

I am not sure if the no of activations required has been calculated correctly. Also Please let me know if the above memory calculations are appropriate. I am trying to train this network on 12GB GPU but I am running out of resource in the first epoch. If the above estimates are not appropriate, then could you please hint me towards a way to find roughly the memory usage for an LSTM network?


texts:
I'm trying to plot a bar chart to represent the frequency of two variables (Dead and Alive) in my test and training data. I want it to look like the second example described in the documentation (minus the error bars).

Currently i have
yfreq = pd.DataFrame(data = [Counter(Ytrain),Counter(Ytest)],
                     index = ['Train','Test'])
yfreq.columns = ['Dead', 'Alive']
print(yfreq)
>>>        Dead  Alive
    Train   384    239
    Test    165    103

And then for the graph
sns.barplot(data = yfreq)

Which correctly outputs a graph but does not split the data into test and training data.

I know it has something to do with the hue kwarg but i can't work out how too do it.
Thanks in advance


texts: R - using heatmaply for a 2d histogram / density

texts:
I have a rather long series of x, y coordinates representing a character in 2d space. Let's say that space is 200 x 400. I want to represent the amount of time the character is in each x, y coordinate by means of a pretty chloropleth.
I want to use heatmaply for this because I think the output is pretty and I want my audience to be able to zoom in on the data. It isn't really meant to do density estimation (I think?) so I'm trying to work around it.
I suppose the way to do this is to fill a 200x400 dataframe with counts of the number of occurrences of each x, y coordinate in the data.  Essentially, to build a 2d map out of the dataframe.
So, I suppose my questions are:
1). How do I get the count of each unique x, y coordinate in my set
2). How might I pass those counts easily to the matching x, y cell in my 200x400 dataframe full of zeroes?
This seems like it should be easy but I can't seem to figure it out! I'm a novice to r and can't see the shape of what I need to do.


texts:
I am little confused about how to structure my specific data for multilevel analysis.

I have 10 categories and each category has some items in them.

The dataset is available for 117 weeks. There is quantity share and price index available on all of them.

My usual regression takes place with predicting the quantity mix by using the price indices for items within a category. This gives me regression coefficients for items but by treating every category separately.

I aim to use the multi level modeling approach to see the regression coefficients effects between items and categories.

This makes the model quite complicated and I’m not sure how to structure the data.


texts: Is my understanding of On-Policy and Off-Policy TD algorithms correct?

texts:
After reading several questions here and browsing some pages on the topic, here is my understanding of the key difference between Q-learning (as an example of off-policy) and SARSA (as an example of on-policy) methods. Please correct me if I am misled.

1) With an on-policy algorithm we use the current policy (a regression model with weights W, and ε-greedy selection) to generate the next state's Q.

2) With an off-policy algorithm we use a greedy version of the current policy to generate the next state's Q.

3) If an exploration constant ε is set to 0, then the off-policy method becomes on-policy, since Q is derived using the same greedy policy.

4) However, on-policy method uses one sample to update the policy, and this sample comes from on-line world exploration since we need to know exactly which actions the policy generates in current and next states. While off-policy method may use experience replay of past trajectories (generated by different policies) to use a distribution of inputs and outputs to the policy model.

Source: https://courses.engr.illinois.edu/cs440/fa2009/lectures/Lect24.pdf

One more reading: http://mi.eng.cam.ac.uk/~mg436/LectureSlides/MLSALT7/L3.pdf


texts: K-Means clustering - What to do if a cluster has 0 elements?

texts:
I'm writing code for k-means clustering. I have around 100000 vectors of size 128x1 (SIFT descriptors). I'm trying different initialization methods such as Forgy and Random Partition.
What if suppose, no vectors are classified to a cluster (in one of the iterations)? How to calculate centroid for that cluster?


texts: How to get significance level for ranked features?

texts:
I am aware of below approaches of feature selection

a) Feature Importance methods which are available in tree based models like Random Forest and Xgboost,GradientBoost etc.

b) statsmodel.logistic regression which in it's summary output provide us the results which contains whether variables are significant or not (P-value)

c)  SelectKbest which uses ANOVA, Chi-square etc to compute the influence of input variable on target attribute

But unfortunately with methods b and c, it doesn't consider the feature interaction. Am I right? It works by considering each column to the target variable

Whereas with methods a it returns the ranking but we aren't sure about whether they are significant or not.

Is there anyway to know from Feature Importance whether the Features are significant or not? I understand features occurring in top 4-5 places could be significant but is there anyway to test/validate this?

Or is it like I pick each feature (out of say top 20 assuming they have a role) from feature importance result and do a SelectKbest test or statsmodel summary?

How can I know that the features that I select from Feature importance model are significant?


texts: Is spark als item feature comparable between several runs

texts:
I am using spark als.train() to build my user-items recommendation system.

The problem is I want to cover more item feature. So, I need to input 7 days user action data. But the als train become slow than just input 1 day data.

So, is it possible that I just input 1 day data, and compare the similarities between other runs(every time just input 1 day)?


texts: how can i detect medicine name and info(use and contents) by using medicine wrappers

texts:
I got one project idea creating a Cross-platform react-native app
the project title is creating an app that can detect medicine name and other info from the medicine wrapper
I'm thinking of using Google Cloud Vision to extract all the text from the medicine wrapper.
The text contains medicine mfg info, address, pin code, expiry date, ingredients, and most importantly medicine name.
I want to extract only the medicine names and other info out of it.
What are the possible ways to get medicine name and other info?
and for rural people in India they can't spell the word correctly for that I can give one button that can spell that name for folks can you suggest me any ideas how can I do this and one important part
how/where can I get medicine wrapper ( Packaging ) of different medicines data????


texts:
I have gotten very used to coding in R and especially in RStudio. I like that interface. Nonetheless, I have some work that I ought to do in Python. I know that I can run Python code in RStudio if I use the "reticulate" package, and I've written some code that has switched between the two. As far as I can tell, anything I could run in Python from the command line or in Jupyter could be done in RStudio with reticulate.

Should I expect to have longer run times using Python commands in R, than I would if I ran Python regularly (whatever "regularly" means)?


texts: Optimising Kernel parameters using training data in GaussianProcessRegressor of Scikit-learn

texts:
I want to optimize the Kernel parameters or hyper-parameters using my training data in GaussianProcessRegressor of Scikit-learn.Following is my query:

My training datasets are:

X: 2-D Cartesian coordinate as input data

y: radio signal strength (RSS) at the 2-D coordinates points as observed output

What I've done so far:

I've installed python and Scikit-learn software. I've successfully tested the sample codes. I'm able to predict RSS at test points using training data. I use squared exponential Kernel.

What I want to do:

I want to train the Kernel parameters (hyper-parameter) with different optimizing algorithms like gradient descent, swarm intelligence, and trust-region-reflective algorithms.

What I learned and What help I am asking for:

I've learned that, in the GaussianProcessRegressor class of scikit, the optimizer is an argument where I can use my own optimizing algorithm. Since it is callable, I need to write my own function/method for it. Can I use any inbuilt library (library of optimization algorithm) in GaussianProcessRegressor class? Are there such libraries available for python? Could anybody provide any sample code for using kernel parameter optimization algorithm in GaussianProcessRegressor? I've learned that we use the training datasets for optimizing the hyper-parameters. Could anybody provide any insight about relating the training datasets with the optimization algorithm, please?

Thank you!


texts: which open-source frameworks exist for fact-extraction

texts:
probably question is silly, but still: what tools one might use to transform an English text to a set of "facts"

for example a sentence "Julius Caesar crossed the Rubicon in 49 BC" would be parsed as something like:

{
"subject": "Julius Caesar (person)",
"object": "Rubicon (river)",
"verb": "to cross",
"verb aspects": ["time": "49BC"]
}

or anything remotely close to this


texts: Does anyone know a way to add cell numbers (not line numbers within cells)?  I have been using nbextensions for a while but it does not seem to have the ability to label cell numbers.


texts: Is it possible to make a label automatically in supervised learning(Machine Learning)?

texts:
My background knowledge: Basically, supervised learning is based on labeled data. Using the labeled data, the machine can study and determine results for unlabeled data. To do that, for example, if we handle picture issue, manpower is essentially needed to cut raw photo, label on the photos, and scan on the server for fundamental labeled data.

I know it sounds weird, but i'm just curious if there are any algorithms/system to make a label automatically for supervised learning.


texts: I have ASCII data which converted from raster data in ArcGIS. I would like to use Support Vector Machine  (SVM) algorithm to create a flood prediction map. My question is that can I use ASCII file with SVM algorithm in python or how can I create an appropriate file from raster data by using ArcGIS or another method for use it in Python?


texts: Dealing with events that have not yet happened when building a model

texts:
I was building a model that predicts user churn for a website, where I have data on all users, both past and present.

I can build a model that only uses those users that have left, but then I'm leaving 2/3 of the total user population unused.

Is there a good way to incorporate data from these users into a model from a conceptual standpoint?


texts: A weird result from a recommender system

texts:
Say there're the top 10 most popular items among 100 sales products and about 100k users regularly purchase items on daily basis.

A = has been purchased by 100k users.
B = has been purchased by 30k users.
C = has been purchased by 20k users.
D = has been purchased by 18k users.
E = has been purchased by 10k users.
F = has been purchased by 8k users.
G = has been purchased by 7k users.
H = has been purchased by 4k users.
I = has been purchased by 3k users.
J = has been purchased by 1k users.

X = never bought by anyone.
Y = never bought by anyone.
Z = never bought by anyone.

So basing on this fact, the training data is going to have more than 50m rows of data like this.

User Id  |  User Name  |  Item Id  |  Item Name  |  label  |  Purchase Date  |
1           Thomas        1           A             true      12, Mar 2019
1           Thomas        1           A             true      13, Mar 2019
1           Thomas        1           A             true      14, Mar 2019
1           Thomas        1           A             true      15, Mar 2019
1           Thomas        2           B             true      11, Mar 2019
1           Thomas        3           C             true      09, Mar 2019
1           Thomas        4           D             true      07, Mar 2019
2           Angelica      1           E             true      12, Mar 2019
.
.
.

The preferences of users will be like this, they might be countless but let's take one example.

Thomas bought A, B, C, D
Angelica bought A, B, C, D
Gloria bought A, B, C, D
Jennifer bought A, B, C, D and I

Using the user based collaborative filtering, it is quite obvious that Thomas, Angelica, Gloria are likely to get the item I as a recommended item because Jennifer likes I item and also has the exact same purchase pattern as the others do.

With this in mind, I was starting to think that if I have another two users who bought the unpopular items X,Y,Z, the predictions on them will result in recommending the unsold items.

So I added dummy data manually before training the model like this.

User Id  |  User Name  |  Item Id  |  Item Name  |  label  |  Purchase Date  |
1           Thomas        1           A             true      12, Mar 2019
1           Thomas        1           A             true      13, Mar 2019
1           Thomas        1           A             true      14, Mar 2019
1           Thomas        1           A             true      15, Mar 2019
1           Thomas        2           B             true      11, Mar 2019
1           Thomas        3           C             true      09, Mar 2019
1           Thomas        4           D             true      07, Mar 2019
2           Angelica      1           E             true      12, Mar 2019
.
.
.
100001      Andrew        24          X             true      19, Mar 2019
100001      Andrew        25          Y             true      19, Mar 2019
100002      Andy          24          X             true      19, Mar 2019
100002      Andy          25          Y             true      19, Mar 2019
100002      Andy          26          Z             true      19, Mar 2019

As I mentioned above, I thought Andrew will get Z as a recommended item because Andrew has a common in the item preference with Andy and he bought Z as well, even if the purchase data for X,Y and Z has a extremely small portion of training data ( only 5 records exist among the 10M records of data ).

But the result was totally unexpected.

Every users have X, Y, Z in their recommended list, although the prediction score is very low compared to the others and what's more puzzling is that Andrew and Andy have no outstanding scores on the unpopular items even if they actually bought them!

I don't know why this happens, do I misunderstand the user based collaborative filtering concept?


texts: how to update a face recongition ( e.g facenet)

texts:
I'm searching for a face recognition that detects new faces not just faces getting from the datasets,
for example, facenet is a project that can detect and recognize a face from labels.
I would like to know how can I create a face recognition that can update faces, that means to detect new faces.
I have an idea that it this can be using SQL database that includes a faces table (face_id,face_name,face_image) but I don't know how can I fasten the database with the face recognition system.


texts: Is it possible to have differential weights as per input importance for a Neural Network

texts: I have a Computed Tomography data set where central slices are more important as the scan angle approaches 90 degrees. The information in the initial and last slices(0 degrees and 180 degrees) may assigned lesser weight as they contain lateral information. Could anyone please suggest me a way to assign weights as per importance of the input? Shall I focus on any specific hyper-parameter while training?


texts:
As what type of "thing" are various objects and functions one deals frequently with in R stored internally?

To clarify: For languages like Python it is very easy to understand conceptually how your data is stored internally: Everything is stored as an object and this uniform way of handling data makes it very to do advanced stuff like assigning new meaning to methods or writing functions that take other functions as input. What I am looking for is a similar conceptual understanding for R (e.g. I presume not everything in R an object as well).

(One concrete point that I would hope to understand as a consequence of the answer to this question would be: why setting,e.g., the second column of a dataframe to NULL like sodf[2] <- NULL deletes it. This means: You don't have to answer this question particularly, but please answer in such a way that I understand how various recipes spread out on the internet for dealing dataframes actually make sense.)

Also, please don't just point me to the documentation. I'm sure somewhere deep inside it is the answer, but reading the documentation is just too time consuming.


texts: Error when drawing random numbers from a custom continuous distribution using scipy.rv_continuous

texts:
I am trying to generate a sample of random numbers from a custom distribution

$$ p(x) = x^{n}e^{-xtn}. $$

After reading the tutorial on scipy's website, I wrote a subclass which I called kbayes:

class kbayes(rv_continuous):
    def _pdf(self, x, t, n):
        p = x**n * np.exp(-t*n*x)
        s = np.sum(p)
        return p/s

The line s=np.sum(p) is there to normalize the distribution.

The pdf seems to be ok when I check it on some numbers: running the following code

ks = np.logspace(-5, -2.3, 1000)
p = kb._pdf(ks, 500, 12)
hist, _ = np.histogram(p, ks)
plt.plot(ks, p)

results in

which is what I expect.

But when I try to generate a sample, via e.g.

p_test = kb.rvs(size=1000, t=500, n=12)

I get the following error:

/home/[redacted]/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:5: RuntimeWarning: invalid value encountered in double_scalars
  """
/home/[redacted]/anaconda3/lib/python3.6/site-packages/scipy/integrate/quadpack.py:385: IntegrationWarning: The occurrence of roundoff error is detected, which prevents
  the requested tolerance from being achieved.  The error may be
  underestimated.
  warnings.warn(msg, IntegrationWarning)

I understand that this distribution has a vert small practical range (where the probability is higher than $10^{-10}$), but I expected the rv_continuous class to be able to cope with that.

I would appreciate any input on how to make this work, or any other way to draw a sample from this distribution using python.


texts: Text Classification with mixed features in Random Forests

texts:
I am working on a text classification problem on tweets. At the moment I was only considering the content of the tweets as a source of information, and I was using a simple bag of words approach using term frequencies as features, using Random Forests (this is something I cannot change).

Now my idea is to try to incorporate information present in the URLs used in tweets. Now, not all the tweets have URLs, and if I decide to use the same term frequency representation also for URLs I will have a huge number of features only from URLs. For this reason, I suppose that having a single set of features containing both the tweet term frequencies and the URL term frequencies could be bad. Besides I'll have to fill some impossible values (like -1) for the URL features for tweets that do not have URLs, and I will probably worsen the classification for this tweets, as I will have a huge number of uninformative features.

Do you have any suggestions regarding this issue?


texts: How to detect Class label in Apriori algorithm in MATLAB?

texts:
Sorry if my my question is simple. I just would like to know how to detect class labels in data mining algorithms such as Apriori.

As I know, in the normal use of the Apriori algorithm our patterns have no label. Should I use another algorithm or technique?


texts: Covariance-Matrix Adaptation Evolution Strategy (CMA-ES) implementation with Tensorflow Sequential model

texts:
After coming across this article about evolution strategies http://blog.otoro.net/2017/10/29/visual-evolution-strategies/, it seems clear that the Covariance-Matrix Adaptation Evolution Strategy (CMA-ES) has the potential to be incredibly effective.

I'd like to try to create a similar algorithm to be used with a Tensorflow Keras Sequential model.  I've only seen simple examples, but here's how I think it could be done with a Sequential model:

Build the model as if backpropagation were being used and split the data into training and test sets
For some number of items representing the search size of the CMA-ES model, initialize weights randomly for each item
Use Tensorflow's model.predict method to evaluate the loss for each instance of the model on the training set
Select the best instances (let's say the top 20%) based on the calculated loss
Calculate the mean and standard deviation in accordance with the CMA-ES methodology for each weight among the weights of the selected models
Generate a new list of random weights based on the new weight means and standard deviations
Repeat steps 3-6 for some number of iterations or until convergence

The model I'm working on has almost 27k weights.  So if I use 100 search instances and iterate through 50 times, that's 135M individual weights.
 With that said, before I go down this rabbit hole, I'd be curious if anyone has general feedback or experience with this method, but I'm specifically wondering:

What is the best way to initiate weights for each instance of the model and set the initial standard deviations for each weight?  How is Tensorflow doing this under the hood?
Will the weights converge with high probability within a reasonable time like the image below (taken from the article) or could it get stuck between multiple low points?

I'd be interested to hear from anyone who has explored this topic.


texts:
I am working on the problem of speaker clustering. I am using k-means clustering. The ground truth cluster values and k-mean cluster values do not correspond due to different methods of labelling (manually labelled for ground truth and via scikit's k-means function). So I need to find all permutations of matching values and choose the matching values with the highest percentage of match/correctness. I have a ground truth npy array and a labels_by_kmeans npy array as shown below:

Ground truth

Labels by Kmeans

As you can see, arrays 0 and 2 in both tables have 0s and 2s,3s that correspond to each other. However, in array 1 5s seems to corresponds to 0s. I am trying to find the best fit for 0s in ground truth in Labels by k-means.

How can I do this via code?


texts: Error with pandas dataframe (needs to be 1-dimensional)

texts:
I am trying to determine the conformal predictions for my model with my data. But it gives me following error that occurs at  icp.calibrate(X_cal, y_cal) :
Exception: Data must be 1-dimensional

Below you can find the most recent traceback error about this. Unfortunately I am not sure on what this actually infers based on the code from above. I am using a pandas dataframe for this.
#Code Snippet
from sklearn.tree import DecisionTreeRegressor
from nonconformist.cp import IcpRegressor
from nonconformist.base import RegressorAdapter
from nonconformist.nc import RegressorNc, AbsErrorErrFunc, RegressorNormalizer, NcFactory
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd

# -----------------------------------------------------------------------------
# Setup training, calibration and test data
# -----------------------------------------------------------------------------
df = pd.read_csv ("prepared_data.csv")

# Initial split into train/test data
train = df.loc[df['split']== 'train']
valid = df.loc[df['split']== 'valid']

# Proper Validation Set (Split the Validation set into features and target)
X_valid = valid.drop(['expression'], axis = 1)
y_valid = valid.drop(columns = ['new_host', 'split', 'sequence'])

# Create Training Set (Split the Training set into features and target)
X_train = valid.drop(['expression'], axis = 1)
y_train = valid.drop(columns = ['new_host', 'split', 'sequence'])

# Split Training set into further training set and calibration set
X_train, X_cal, y_train, y_cal = train_test_split(X_train, y_train, test_size =0.2)

# -----------------------------------------------------------------------------
# Train and calibrate underlying model
# -----------------------------------------------------------------------------
underlying_model = RegressorAdapter(DecisionTreeRegressor(min_samples_leaf=5))
print("Underlying model loaded")
model = RegressorAdapter(underlying_model)
nc = RegressorNc(model, AbsErrorErrFunc())

print("Nonconformity Function Applied")
icp = IcpRegressor(nc)  # Create an inductive conformal Regressor
print("ICP Regressor Created")

#Dataset Review
print('{} instances, {} features, {} classes'.format(y_train.size,
                                                   X_train.shape[1],
                                                   np.unique(y_train).size))

icp.fit(X_train, y_train)
icp.calibrate(X_cal, y_cal)

#Example Dataframe
new_host  split     sequence    expression
FALSE     train     AQVPYGVS    0.039267878
FALSE     train     ASVPYGVSI   0.039267878
FALSE     train     STNLYGSGR   0.261456561
FALSE     valid     NLYGSGLVR   0.265188519
FALSE     valid     SLGPSNLYG   0.419680588
FALSE     valid     ATSLGTTNG   0.145710993

I've tried splitting the dataset in various ways but I am continuing to have trouble with this. In this case I want to split the data into train and test sets according to an observation's Data Split value. After which, I will split the train set into train and calibration in a second step. Where myfeatures, X_train and my target, y_train


texts: Best way to search for a similar document given the ngram

texts: I have a database of about 200 documents who's ngrams I have extracted. I want to find the document in my database that is most similar to a query document. In otherwords, I want to find the document in the database that shares the most number of ngrams with the query document. Right now, I can go through each one and compare it one by one, but this will take O(N) time and is expensive if N is very large. I was wondering if there are any efficient data structures or methods in doing efficient similarity search. Thanks


texts:
I would like to work on a project for Fake News Detection especially for Indians news which are in different languages and different formats.

Fake news as image with no or very less text
Fake news on a blog site
Fake news as Tweets
Fake news in Hindi
Fake news in the watsapp group and shared across.

Need your help on the approach. One approach I can think of is using OCR we can read the content of the post, then search those content in the google. If the news is not present in any of the famous print media then we can tag it as fake. However there can be many challenges in this. What if the print media itself gives any fake news shared by someone.

How to handle the scenario where there is no text in the image but the information shown as image is fake.

How to handle posts written in Hindi. ?

And even if we detect fake news, is there any way to make the person accountable for sharing it. ? I know it is little difficult problem to solve. But is there any work currently done by any company on this. ?
Any starting point for me to get into this domain ?


texts: Different accuracy values using the same saved model in tensorflow

texts:
I have trained a model in Tensorflow (for some signal classification problem, using mostly convolutional layers, no RNNs), saved It using the callback checkpoints. When I'm testing the said model on a testing dataset I get different accuracy values (very similar but different) at each run. What can be the cause?

All dropout layers are skipped for testing and no shuffle is done on the dataset.


texts: CIFAR-100: What is the difference between vehicles 1 and vehicles 2?

texts:
The superclasses in the CIFAR-100 dataset are mutually exclusive and all but the vehicle ones are quite well defined by its label.
Example: It is very clear why bees belong to the superclass insects and none of the other superclasses.
This appears not to be the case for the two superclasses vehicles 1 and vehicles 2: The two appear not to be clearly separable if we ignore their subclass labels.
Example: It is not clear why pickup trucks belong to the category vehicles 1 and not vehicles 2.
Clearly, vehicles 1 vs. vehicles 2 are not distinguished by size, number of wheels, number of passengers, ….
Hence, my question is: What distinguishes classes vehicles 1 and vehicles 2?
As a reminder, these are the two vehicle-classes:

vehicles 1:   bicycle, bus, motorcycle, pickup truck, train
vehicles 2:   lawn-mower, rocket, streetcar, tank, tractor


texts:
I am looking for an online console for the language R. Like I write the code and the server should execute and provide me with the output.

Similar to the website Datacamp.


texts:
Can someone please tell me the difference between those stacked LSTM layers?

First image is given in this question and second image is given in this article. So far what I learned about stacking LSTM layers was based on the second image. When you build layers of LSTM where output of one layer (which is $h^{1}_{l}, l=..., t-1, t, t+1...$) becomes input of others, it is called stacking. In stacked LSTMs, each LSTM layer outputs a sequence of vectors which will be used as an input to a subsequent LSTM layer. However, in the first image, the input variables are fed again into second layer. Can someone tell me whether there is something wrong about stacking LSTM layers like given in the first image?


texts: Why averaging the gradient works in Gradient Descent?

texts:
In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally use it to correct the network, at once.

But why does averaging the gathered gradient work? Each training sample ends up in a distant, completely separate location on the error-surface

Each sample would thus have its direction of a steepest descent pointing in different directions compared to other training examples. Averaging those directions should not make sense? Yet it works so well. In fact, the more examples we average, the more precise the correction will be: full-batch vs mini-batch approach


texts: I am a programmer, how do I get into field of Data Science?

texts:
First of all this term sounds so obscure.

Anyways..I am a software programmer. One of the languages I can code is Python. Speaking of Data I can use SQL and can do Data Scraping. What I figured out so far after reading soo many articles that Data Science is all about good at:

1- Stats

2- Algebra

3- Data Analysis

4- Visualisation.

5- Machine Learning.

What I know so far:

1- Python Programming
2- Data scrapping in Python

Can you experts guide me or suggest a roadmap to brush up both theory and practical? I have given around 8 months of time frame to myself.


texts: How to correctly pass Word2Vec vectors as input to an LSTM

texts:
I am trying to build a text classifier using lstm which, in its first layer, has weights get by a Word2Vecmodel.

In order to build a matrix containing the indexes of each word for each sentence, I have tried: (as mentioned here)

X_tr_word2vec = np.array(X_tr_word2vec)
y_tr_word2vec = np.array(y_tr_word2vec)

train_x = np.zeros([X_tr_word2vec.shape[0], max_sentence_length], dtype=np.int32)
train_y = np.zeros([y_tr_word2vec.shape[0]], dtype=np.int32)

for i, sentence in enumerate(X_tr_word2vec):
    for j, word in enumerate(sentence[:-1]):
        train_x[i,j] = model_word2vec.wv.vocab[word].index

but, when I run the code, I get this error:KeyError: 'enquiringly', what does it mean?
I suppose that it is about a wrong train_xdimension.

Update:

I have trained Word2Vec model before, with the entire training set:

model_word2vec = models.Word2Vec(X_tr_word2vec, size=150, window=9)


texts:
Is there a regression approach for predicting the rankings (ordering) as an endogenous variable?

I would like to fit a regression to a dataset where the response is the order of a variable: 1st, 2nd, 3rd. Mainly, Mainly, I would like to know the influence of the 5 $X$s variables on this ranking.

My approach so far has been calculating the uniform-associated variable:

$U_i = \frac{R_i}{n+1}$

where $R_i$ is the ranking of the i-th individual.

Any further ideas or comments would be appreciated.


texts: Using KNN to categorise inventory (physical stock items) - is it the best way?

texts:
I'm working on a machine learning problem involving inventory (i.e. physical retail stock), however through the cleaning (outlier removal) process some of the items (via their corresponding transactions) will be removed. Therefore, I thought of using KNN to group similar items into respective categories.
There are 1245 items
The info for each item is

Average Weighted Price
Total Quantity Sold
Total Revenue Achieved
Min Sold per Transaction
Max Sold per Transaction
Min Sell Price
Max Sell Price
Number of Unique Transactions

Am I right in thinking that KNN is a good option - and if so, how do I decide on the number of clusters?


texts: How to generate 12 independent random weights which all add up to one

texts: I'm using Palisade's @Risk software with a triangular distribution to generate 12 random weights which must add up to one, but I get a lot of negative numbers.  Is there a straightforward way to set this up?


texts: How to train model on specific classes from dataset for object detection?

texts:
I want to use darkflow to perform object detection. I have 5 classes of objects I want to detect, say ,

  chair
  bottle
  door
  phone
  computer

I want to specifically train my model again, not use pre-trained weights. I plan to train it on images which are annotated with bounding boxes. But all available datasets like imagenet, COCO have images that have objects of other classes also annotated in them. So how can I use these datasets to  train specifically on my 5 classes? I don't want my model training on all the other objects in the images.


texts: mysql could not recieve table from Citespace

texts:
I created was database through commands (see lower)

Started Citespace, clicked DATA - IMPORT/EXPORT
Citespace-Built-in-database
Database-Connect to mySQL
Input directory - choosed directory with WOS files
Project - import data (WOS)

Now check:

click Count records
Recieved:
projects  0
articles    0
authors  18020
refs   144499
keywords  64260
phrases    0
institutions  11695
verbs

so tables

- articles
- bursts
- clusters
- phrases
- verbs

are empty
and system see no projects

CAN YOU help me to understand what i do wrong?

=== database schema

CREATE DATABASE wos;
USE wos;
CREATE TABLE articles
(
    id int,
    uid VARCHAR(100),
    project VARCHAR(100),
    author VARCHAR(100),
    title VARCHAR(100),
    abstract VARCHAR(1500),
    source VARCHAR(200),
    j9 VARCHAR(100),
    volume VARCHAR(100),
    issue VARCHAR(100),
    bp VARCHAR(100),
    ep VARCHAR(100),
    page VARCHAR(100),
    dt VARCHAR(100),
    doi VARCHAR(100),
    year int,
    month int,
    date int,
    citations int,
    editor VARCHAR(100),
    tagged boolean
);
/*  TABLE articles
id(int), uid, project, author, title, abstract, source, j9, volume, issue, bp, ep, page, dt, doi, year(int),
month(int), date(int), citations(int), editor, tagged(boolean)
*/
CREATE TABLE authors
(
id int,
lastname VARCHAR(100),
firstname VARCHAR(100),
initials VARCHAR(100),
project VARCHAR(100),
uid VARCHAR(100),
pos VARCHAR(100)
);
/*  TABLE authors
id(int), lastname, firstname, initials, project, uid, pos */
CREATE TABLE refs
(
id int,
bibcode VARCHAR(100),
ref VARCHAR(100),
doi VARCHAR(100),
author VARCHAR(100),
year VARCHAR(100),
source VARCHAR(100),
volume VARCHAR(100),
page VARCHAR(100),
citer_uid VARCHAR(100),
project VARCHAR(100)
);
/*   TABLE refs
id(int), bibcode, ref, doi, author, year, source, volume, page, citer_uid, project */
CREATE TABLE keywords
(
id int,
keyword VARCHAR(100),
uid VARCHAR(100),
year VARCHAR(100),
project VARCHAR(100),
type VARCHAR(100)
);
/* TABLE keywords
id(int), keyword, uid, year, project, type */
CREATE TABLE phrases
(
id int,
phrase VARCHAR(100),
isTitlePhrase boolean,
project VARCHAR(100),
uid VARCHAR(100),
year int,
month int,
date int,
freq int
);
/*  TABLE phrases
id(int), phrase, isTitlePhrase(booelan), project, uid, year(int), month(int), date(int), freq(int)  */
CREATE TABLE verbs
(
id int,
project VARCHAR(100),
uid VARCHAR(100),
verb VARCHAR(100),
freq VARCHAR(100)
);
/*  TABLE verbs
id(int), project, uid, verb, freq  */
CREATE TABLE bursts
(
id int,
project VARCHAR(100),
term VARCHAR(100),
weight  double,
start int,
end int
);
/* TABLE bursts
id(int), project, term, weight(double), start(int), end(int) */
CREATE TABLE institutions
(
id int,
name VARCHAR(100),
country VARCHAR(100),
uid VARCHAR(100),
year int,
project VARCHAR(100)
);
/* TABLE institutions
id(int), name, country, uid, year(int), project */


texts: Improve a regression model and feature selection

texts:
I am working on Azure ML Studio and try to create a regression model to predict a numerical value. I will try to describe my features and what I have done until now.

My data with about 3 million rows :

Features:

8 integer features from 1 to 25
2 boolean features with 0 and 1
3 integer features from 1 to 10
2 integer feature from 0 to 500.000 (and 1.000.000 respectively) with about 4.500 unique values
1 integer feature from 20 to 50
1 integer feature from 1 to 15
1 integer feature from 0 to 100

Label:

Integer from 10.000 to 100.000.000 with about 5.000 unique values

What I have done:

Split the dataset to 80% (train) and 20% (test). Then I split the training dataset again to 60% (actual train) and 40% (validation).
Normalize the features with many unique values (4th bullet in the above list)
Train a model of Boosted Decision Tree Regression.
Use the Sweep Parameters module to find the best combination

I tried also Neural Networks, Bayesian Linear Regression, but BDTR gave the best score.

I tried to exclude columns and start with only a few (based on what I think it will affect the model) and then add more columns one by one.

However, the least MSE I could achieved was 1.500.000 (plus I had many negative scored values)

So, I was thinking what other techniques I could use to improve the model.


texts: Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?

texts:
According to this article, "Systems used for intent classification contain the following two components:
Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+SVM.
I'm trying to do the opposite, comparing two different classifiers (RNN and SVM) using BERT's word embedding.
Most Python codes that I found use BERT for the whole intent classification problem which made me confused. Example
I only want to map the words into vectors with BERT and feed the result into a classifier (SVM/RNN).
Does BERT support word embedding and text classification at the same time? Does someone has an explanation? Is what I'm trying to test feasible with Python?
I have a dataframe that has two columns: intent and questions. It's a small dataset.
Thank you!


texts: Speed decay proof for L2 regularization and non-normalizied weight initiation

texts:
I'm working through http://neuralnetworksanddeeplearning.com/chap3.html,
textbook by Michael Nielsen.

While most problems are rather straightforward, I'm very stuck on these two questions:

Sketch a heuristic argument that: (1) supposing $\lambda$ is not too small, the first epochs of training will be dominated almost entirely by weight decay; (2) provided $\eta\lambda \ll n$ the weights will decay by a factor of $ \exp(−\eta\lambda/m)$ per epoch;

To give a self-sufficient context, the weight decay function is:

$w \rightarrow \left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m}
  \sum_x \frac{\partial C_x}{\partial w}$,

and cost function is

$C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln
(1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2$

(Classic logloss)

Weights in the network are initiated as $N(0,1)$.

My intuition for (1) goes as follows: As weight initialisation is non-normalized by size of previous layer, activation values are likely to be either $1$ or $0$ with equal probability (which is pointed out by the author). $y_j$ has $0.1$ probability of being 1. Given that $\frac{\partial C_x}{\partial w} = a^{L-1}_k(a^L_j - y_j)$ for logloss and sigmoid neurons, expected value of $\frac{\partial C_x}{\partial w}$ on the first iteration is $0.2$.
Given that $\lambda$ is arbitrary, we can be sure that
$|\frac{\eta \lambda}{n} w|$ can be larger than $|\frac{\eta}{m}
  \sum_x \frac{\partial C_x}{\partial w}|$

Despite questions being rather informal, I don't think this solution is a good one.

As for the (2), I lack any ideas whatsoever, except driving $\frac{\eta \lambda}{n}$ to zero, but that seems to have zero chance of yielding the required result.

I hope someone can give me a hand in this.
Thank you!

P.S. There is a chance I'm going to the wrong overflow, but I used
https://meta.stackexchange.com/questions/130524/which-stack-exchange-website-for-machine-learning-and-computational-algorithms
question as a guidance for overflow choice.

A note: I had asked similiar question here, but that might not be the best overflow for this question. I'll put a link to whichever get's answered first :)


texts: Getting extra characters while loading data

texts: While loading data using read_sas from sas dataset in Python getting b' along with every string. How can I fix this?


texts: Support Vector Regression trained with data sets

texts:
I am now searching for a long time on the internet and on papers for an answers of simple questions. Am I able to train a Support Vector Regression algorithm with different data sets? If yes, how is the approach called?

I have 10 times the same battery with different usage, temperature and capacity.

Usage and temperature are features (x_i,i) and capacity is the output (y_i,i).
Battery_1 till timepoint n: [x_1,1 y_1,1; ... ;x_1,n y_1,n]
...
Battery_10 till timepoint n: [x_10,1 y_10,1; ... ;x_10,n y_10,n]

Now I want to train my SVR with these sets, where the samples within a set belong together. I want give the algorithm a set of usage and temperature where I don't know the capacity and my SVR should predict it as such : x_d --> y_d.

Thank you very much for your help and input.


texts: Estimating the value of $\pi$ with a Monte Carlo dartboard: $<$ or $\leq$?

texts:
I'm trying to figure out which is the proper way to estimate $\pi$ using the Monte Carlo method randomly distributing points in a square that also contains an inscribed circle.

Some sources say to use the comparison of $\sqrt{x^2+y^2}\le 1$, while others use $\sqrt{x^2+y^2}<1$.

Here's some example code from a wikipedia article:

def monte_carlo_pi(nsamples):
    acc = 0
    for i in range(nsamples):
        x = random.random()
        y = random.random()
        if (x**2 + y**2) < 1.0:
            acc += 1
    return 4.0 * acc / nsamples

Instead of posting a long list of websites that use  $\le 1$  or  $< 1$, I've made the list and stored it on the following websites:
See either: socrates.io or markdown.press or markdownshare for examples using less than and less than or equal to.


texts:
I am running the exact example give in this SVM example of Scikit learn without any modification.
I get the following warning.
<ipython-input-49-ba88740f796f>:14: UserWarning: No contour levels were found within the data range.
  ax.contour(XX, YY, Z, colors='k', levels=[-1, 0, 1], alpha=0.5,

Could anyone suggest how to resolve this? TIA


texts:
I have a general question on applying k-means clustering on the datasets..

How to find the feature/(data column) that separates each cluster of K-means?

may be using scikit-learn  in python ..

Best Regards,
Swati


texts: Image segmentation - handcrafted features vs DNN?

texts:
Currently working on a project that requires multi-class image segmentation to identify distinct anomaly types in various sheet metals.

Have had moderate success with various NN segmentation architectures, however inference performance is far from ideal given the native resolution of the images (even troublesome with extensive down-sampling).

Have also attempted to hand-craft several features based on filter banks of various convolutional kernels and then subsequently applying clustering algorithms to build a sort of 'anomaly database'. I can readily apply this defined filter bank to extract features and compared to the features to those in the labeled database.

Shown in the above image, the center horizontal line needs to be identified as an anomaly. Intuition would lead me to believe simple shadow filters would be able to detect its presence, however given the inherent inconsistencies in the sheet metal - it doesn't appear to be simple (despite human identification being very simple).

I'm not necessarily looking for a perfect solution to solve this, however if anyone was able to provide potential avenues for investigation. I want to exhaust all possible (promising) options of segmentation methods.


texts:
I have a list, which looks like this:

params
['h\x00i\x00', '\x00t\x00h\x00e\x00r\x00e\x00']

Now, all I want is to merge these two elements into the string "hi there", but when I use

print ' '.join(params)

I get

h i  t h e r e

How do I strip out those hex characters to make the string "hi there"?