texts:
When explaining advantage function, it is usually claimed that using a baseline reduces the variance. I have not found any specific reference to justify this.

Is this an application of control variates or something similar?

Could anyone provide some reference or formal justification for the variance reduction?


texts:
I've just started working on an anomaly detection development in Python.

My data sets are a collection of timeseries. More in details, data are coming from some sensors/meters which record and collect data on boilers or other equipments.

As I said before, the data which I have to work with, are timeseries, so a timestamp and the relative value detected by sensor; a value is anomalous when it's bigger or smaller than the others near it; basically a peak.

I need to develop an unsupervised classification model, because I haven't labels for all data.

Another important aspect, is that this data are "season dependent"; in fact a boiler should be has a higher consumptions in winter than summer. Those values must not be considered as anomalies.

Since I've no experince on this topic, I'm here to ask you, what is the best algorithm/approace to solve this problem.

Furthermore, do you know some books or links to suggest?


texts: identifying the primary and secondary keywords in sentense

texts:
want to identify the primary and secondary keywords which are having an impact to sentences or comparison between 2 keywords.
below is the example

India and China has highest population in the world.
travelers are interested in travel Paris instead of Hamburg

in the above example
sentences 1 primary keyword is India and secondary keywords is China
sentences 2 primary keyword is Paris and secondary keywords is hamburg


texts: White circles extraction from a little darker background

texts:
I'm trying to extract three white circles on the top left corner from the image below:

I've tried to use:

clone = img.copy()

# Threshold grayscaled image to get binary image
ret,gray_threshed = cv2.threshold(img,90,100,cv2.THRESH_TRUNC)
cv2.imshow('gray', gray_threshed)

# Find edges
edge_detected_image = cv2.Canny(gray_threshed, 350, 100)

# Find contours
contours, _= cv2.findContours(edge_detected_image, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

contour_list = []
for contour in contours:
    contour_list.append(contour)

# Draw contours
cv2.drawContours(clone, contour_list, -1, (255,0,0), 2)

#Resize image
clone = cv2.resize(clone, (800, 800))

cv2.imshow('Objects Detected',clone)

The result was:

I've also tried sth like:

img = cv2.blur(img,(5, 5))

img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
cimg = cv2.cvtColor(img,cv2.COLOR_GRAY2BGR)

circles = cv2.HoughCircles(img,method=cv2.HOUGH_GRADIENT,dp=1,minDist=2,param1=50,param2=60,minRadius=0,maxRadius=300)

circles = np.uint64(np.around(circles))
for i in circles[0,:]:
    cv2.circle(cimg,(i[0],i[1]),i[2],(0,255,0),2)
    cv2.circle(cimg,(i[0],i[1]),2,(0,0,255),3)

But it showed the error due to the fact that it hadn't found any circles.

Any hints how to solve my kind of problem are welcome. Thanks in advance.


texts: Is it possible to detect drift with real time predictions?

texts:
I have been reading up on detecting data drift and concept drift, I have found this library but it seems all the methods here detect concept drift and take input as if the prediction was correct or not. (Requiring ground truth) is this the correct assumption?
Then I stumbled on Kullback-Leibler Divergence and JS-Divergence. Can I use these methods to detect data drift in real time? (ex: request comes into my models API and then the prediction is made. I then take the features and pass it to the function calculating the drift)
Some of my concerns are do I need the full training data to compare to? As I understand these algorithms need the same size of data to compare, so would I need a data set the same size as my training data? Even understanding what the inputs used to detect data drift vs concept drift vs covariate shift would helpful.


texts:
During linear regression classes in academia, it is taught that including trivial/irrelevant features to the model decreases its ability to predict more accurately.

In fbprophet, there is this function, add_regressor(), which allows us to add additional regressors to the model.

I want to know:
- Is there a way to check whether added parameter/feature actually improves the model or it's actually trivial?
- What should I look for during the process of adding regressors to fbprophet?

Of course I prefer a more intuitive, smart way rather than simply checking the evaluation metric with and without the added regressor, since the improvement can simply be noise and cause overfitting, decreasing the ability to generalize on unseen data.


texts: How to numerically estimate MLE estimators in python when gradients are very small far from the optimal solution?

texts:
I am exploring how to model a data set using normal distributions with both mean and variance defined as linear functions of independent variables.

Something like N ~ (f(x), g(x)).

I generate a random sample like this:

def draw(x):
    return norm(5 * x + 2, 3 *x + 4).rvs(1)[0]

So I want to retrieve 5, 2 and 4 as the parameters for my distribution.

I generate my sample:

smp = np.zeros((100,2))

for i in range(0, len(smp)):
    smp[i][0] = i
    smp[i][1] = draw(i)

The likelihood function is:

def lh(p):
    p_loc_b0 = p[0]
    p_loc_b1 = p[1]
    p_scl_b0 = p[2]
    p_scl_b1 = p[3]

    l = 1
    for i in range(0, len(smp)):
        x = smp[i][0]
        y = smp[i][1]
        l = l * norm(p_loc_b0 + p_loc_b1 * x, p_scl_b0 + p_scl_b1 * x).pdf(y)

    return -l

So the parameters for the linear functions used in the model are given in the p 4-variable vector.

Using scipy.optimize, I can solve for the MLE parameters using an extremely low xtol, and already giving the solution as the starting point:

fmin(lh, x0=[2,5,3,4], xtol=1e-35)

Which does not work to well:

Warning: Maximum number of function evaluations has been exceeded.
array([ 3.27491346,  4.69237042,  5.70317719,  3.30395462])

Raising the xtol to higher values does no good.

So i try using a starting solution far from the real solution:

>>> fmin(lh, x0=[1,1,1,1], xtol=1e-8)
Optimization terminated successfully.
         Current function value: -0.000000
         Iterations: 24
         Function evaluations: 143
array([ 1.,  1.,  1.,  1.])

Which makes me think:

PDF are largely clustered around the mean, and have very low gradients only a few standard deviations away from the mean, which must be not too good for numerical methods.

So how does one go about doing these kind of numerical estimation in functions where gradient is very near to zero away from the solution?


texts: Compare cross validation values of Bernoulli NB and Multinomial NB

texts:
I'm testing the Multinomial NB and Bernoulli NB on my dataset and I'm using the cross validation score to better understand which of the two algorithms work better. This is the first classifier:
from sklearn.naive_bayes import MultinomialNB
clf_multinomial = MultinomialNB()
clf_multinomial.fit(X_train, y_train)
y_predicted = clf_multinomial.predict(X_test)
score = clf_multinomial.score(X_test, y_test)
scores = cross_val_score(clf_multinomial, X_train, y_train, cv=5)
print(scores)
print(score)

And these are the scores:
[0.75       0.875      0.66666667 0.95833333 0.86956522]
0.8637666498061035

This is the second classifier:
from sklearn.naive_bayes import BernoulliNB
clf_multivariate = BernoulliNB()
clf_multivariate.fit(X_train, y_train)
y_predicted = clf_multivariate.predict(X_test)
score = clf_multivariate.score(X_test, y_test)
scores = cross_val_score(clf_multivariate, X_train, y_train, cv=5)
print(scores)
print(score)

And these are the scores:
[0.5        0.5        0.54166667 0.54166667 0.52173913]
0.5

From what I understood from the answer posted here, the first classifier works better because my dataset has lots of features (11k) instead of just 1. However, it's pretty strange that I got 0.5 in the second classifier which is an high value considering the number of features. What are the differences between the classifiers?


texts: Tools and Techniques for Analyzing German Automotive Discussion Forum Posts

texts:
I work for a German online disussion forum around all things automotive, a bit like a “StackOverflow for cars”, if you will.

We would like to train a model using TensorFlow with our high quality content, to be able to evaluate the quality of new content our users post on our platform.

Our ultimate goal is to be able to put a link to the best answer to a question on our discussion forums.

We (two backend Java developers and myself, a JavaScript frontend web developer) are very new to the field of data science and machine learning, currently going through the tutorials from Google and trying to figure out where to start.

Which tools and technologies would you recommend to use for this project?

How can we train a model to suit our needs?

Are there any tutorials that demonstrate how to train a model to use German text as input?


texts: python///TypeError: unsupported operand type(s) for -: 'list' and 'list' ,i write the code and i get this massege

texts:
N = int(input('number of traverse='))
x = []
y = []
for r in range(N):
    x.append([float(input('enter x='))])
for l in range (N) :
    y.append([float(input('enter y='))])

N=N-1
b=int(N)
area = x[0] * (y[1] - y[b]) + x[b] * (y[0] - y[b-1])

for k in range(1, N):
    area = area + x[k] * (y[k+1] - y[k - 1])
    i=i+1
area = .5 *abs(area)**strong text**
```


texts:
I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a binary variable.

Do I need to run feature selection before the model fitting? Does it affect the model performance significantly or is there not much difference if I directly fit the model using all predictor variables?


texts: LSTM with auxiliary data achieves abismal results

texts:
I have a fairly simple LSTM models that achieves ok results.

I wanted to add auxiliary data as appeared in the keras docs. I added only one binary variable. I expected this model to be at least as good, assuming that if it has no information its weights will go to zero. In reality, the model fails and just predicts zeros for my entire test set.
The second model:

More specifically, I'm predicting a time series of human-behaviour, and my binary feature tags whether the time point is in a weekend. I'd expect it capture some of the weekend (black background) reduction, and if not, at least not to ruin the model completely.


texts: Ridge regression - varying alpha and observing the residual

texts:
I am trying to reproduce this figure from Bishop:

Residual vs. Alpha (lambda in figure)

The code is pasted below:

import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model

from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

x_train = np.linspace(0, 1, 10)
noise_10 = np.random.normal(0, 0.3, 10)
y_train = np.sin(2*np.pi*x_train) + noise_10
x_train = x_train[:, np.newaxis]

x_test = np.linspace(0, 1, 100)
noise_100 = np.random.normal(0, 0.3, 100)
y_test = np.sin(2*np.pi*x_test) + noise_100
x_test = x_test[:, np.newaxis]

n_alphas = 200
alphas = np.logspace(-40, -18, n_alphas)

errors = []
for a in alphas:
    ridge = make_pipeline(PolynomialFeatures(degree = 9),
              Ridge(alpha=a))
    ridge.fit(x_train, y_train)
    mse = mean_squared_error(y_test, ridge.predict(x_test))
    errors.append(np.sqrt(mse))

print(errors)

However, the errors array has the same value for all values of alpha. It's taking the first value of alpha = np.exp(-40) and all the other values seem to be the same for all future iterations of the for loop. How can I correct this error?


texts: Multi-class classification v.s. Binary classification

texts:
If a training set has 5 classes including :
"label-A", "label-B", "label-C", "label-D", "others".

But the problem is much simpler, it wants to distinguish each input whether it is belongs to "label-ABCD" or "others". In this case, there are two solutions to solve this problem in my mind.

Solution-1 : Train a 5-classes classifier, when the classifier predicts the input as "label-A" or "label-B" or "label-C" or "label-D", we relabel it as "label-ABCD".

Solution-2 : Train a 2-classes classifier, we relabel the data as "label-ABCD" which is labeled as "label-A" or "label-B" or "label-C" or "label-D". And then it becomes a binary-classification problem.

My question is...

Which way can the model get a better performance in "theorem"?
In real case, this two cases get almost the same performance by CNN classification model, and I am wondering if I adopt a weaker classifier like C4.5, Naive Bayes, SVM...which way will win?

Thanks!!


texts: Where is publically available time-series RCT data?

texts:
Our team is writing a paper about uplift modeling considering time heterogeneity using Machine Learning model and Causal Inference theory; the example would be like, in marketing, a model maximizing gross-profit of an advertisement campaign by properly choosing the reachable customers based on their not only the fixed but also time-series properties (e.g. not only gender, living location and number of purchases, but also when/how much they have purchased). Since we want to examine the pure effect of the advertisement by eliminating the effect of customers who could make a purchase without advertisement, we need to run RCT as well.
We are looking for some public data to verify our models, but so far found only this one and its data is not enough...
Could you share the other websites providing that kind of data?


texts: Classify the input set into categories based on pre-defined rule set

texts:
I'm trying to solve a problem where there are 2 input files given,

Input 1:

A set of strings in the format. All the strings start with "A".

A-B-C-D
A-B-C-3
A-X-Y-Z-4
A-X-5
A-X-P-Q-R
A-X-P-Q-S
A-M-9-P
A-M-10-Q
A-M-N-G-3
A-V-B-H-3
A-J-K-L-3-S-Y
A-U-3-W-E

Here A,B,C,D,X,Y,Z,M.. (All alphabets are keywords and integers are placeholders)

Input 2:

A set of rules and names to each set of rules. Each rule set can have 1 or more rules. All the rule sets start with "A"

Rule 1: (Name: Red)

A-X-^

Rule 2: (Name: Blue)

A-X-P-Q-R
A-X-P-Q-S

Rule 3: (Name: Yellow)

A-X-P-Q-S
A-X-P-Q-R

Rule 4: (Name: Green)

A-M-$-P
A-M-@-Q
A-M-#-P

Rule 5: (Name: Purple)

A-V-B-H-$
* A-J-K-L-$-S-Y
A-U-$-W-E

Where alphabets are keywords, others (such as ^,$,@,#...) are placeholders. In a given rule set, if the placeholder is the same (like in rule 5), it means the place holder values have to be same, only then it belongs to this category. If they are different placeholders, then the values could be different.

Problem Statement:

Take input 1 (set of strings) and find which category the input belongs to, based on the rule sets from input 2. The output could be one or more categories. Also, the rules in the rule set may not appear in the input one after another.

There could be multiple other rules in between. For example, in rule 2, A-X-P-Q-R has to be followed by A-X-P-Q-S. It doesn't mean there cannot be others in between these two.

Example:

The input file could be

A-X-P-Q-R
A-Q-W
A-Y-T-H
A-X-P-Q-S

It still matches with rule 2 and belongs to "Blue"

My Approach:

I'm thinking of two approaches

Using decision trees (or)
Representing Input 1 as a graph and each of the individual rule sets in Input 2 as separate graphs and finding graph isomorphism.

Any thoughts or suggestions?


texts: Is there a safe and simple way to estimate a standard deviation for a next subset?

texts:
In case I receive only standard deviation from a sensor of a value $v$ (that is btw normally distributed) each 4th minute but need to provide a standard deviation $\sigma$ for each 15 minutes is there a safe way to do it.

There are two things that came into my mind:

1) One and safe way is to get the mean, generate possible values using standard deviation of the 4 minute interval for the 15 minutes period (15*60 values). Calculate the $\sigma$ for this period

2) Alternatively one can naively estimate the value of $\sigma$ of the next time interval based on two previous values. For example, use  and  standard deviations to estimate

In case the standard deviation is increasing/descreasing in previous cases  and   it will increase/descreasing in the next time interval on absolute value  -

The first method can be time-consuming/computationally-consuming comparing to the second method. Though the second method may suffer on precision.

Edit 16.04:  Since I'm limited in the amount of data i preferably would use  only the last standard deviation and no mean data

Edit 23.04: There is one more way that bring me to the result very close to 1st way of problem solving.

Let say $\sigma_i$ is based on $n$ observations while $\sigma_{i+1}$ is based on $k$ observations and $k > n$. Then
$\sigma^2_{i+1} = \frac{(n-1) * \sigma^2_i * \frac{k}{n}}{k-1} $

The benefit in this case that you are not dealing with a mean value. I suppose that this solution works well only with normally distributed values.


texts:
Upfront question:

What are some alternative methods for predicting categorical data?

Details:
I routinely process data that is 100% categorical. Almost always, this data is nominal (while, occasionally ordinal data such as 'quarters' may be included). When the time comes to run predictive modeling, I feel boxed-in to utilize only one method: multinomial regression models.
I have yet to have success employing randomForest or decision trees due to the amount of levels for the data (e.g., some variables have upwards of 111 levels in a set of 12000 observations).
I understand that, given the restricted data type, my options will be limited. However, I am wondering if anyone else has created predictive models with (only) nominal data and what method was chosen.

Note: I utilize R [3.6.1] for processing and modeling.


texts: Machine Learning algorithm for detecting anomalies in large sets of events

texts:
Let's start with the following hypothetical preconditions:

There is traffic: normal and anomaly. Each traffic sample contains a list of events (of variable size)
Events happen in order, the possible events set size is ~40000  elements
Should run on relatively small amounts of memory and processing power

Having a traffic sample (of size 1000 events max), what is the best machine learning algorithm, that fits the preconditions, to identify whether it's an anomaly?

Given my limited knowledge in machine learning algorithms, here is what I came up with:

This system can be described very well as a markov process, but there are huge memory limitations in this scenario

1. Reduced Markov Chains

Store frequent pairs of events (that appeared more than 10 times in normal traffic) and then search the some pair there: if it doesn't appear, count as an anomaly. Then use some heuristic to identify if the traffic as a whole is an anomaly.

I called it reduced, because practically, we are using only a chain of two events, any other bigger size would become a huge combinatorial problem and fill any memory it's been given, which is infeasible.

2. Naive KNN

Get all the normal traffic samples (each sample may contain up to 1000 events) and analyze the number of appearances of each event in each sample.
Separate the dataset into 10 parts, compute their means to get the mean frequency for each part (basically, we now have 10 mean-frequency vectors) and use them as positive data-points inside the KNN algorithm

Do the same with anomaly traffic and add 10 data points.
Having the points, we can now use the KNN algorithm regression to compute a score and make a decision.

This is a bit tricky, because the frequency vectors are quite big, have too many of them becomes a problem. A solution would be to implement sparse vectors

Any other ideas, what am I missing?


texts: What are some data sources with strong positive and negative correlations in feature and sample space?

texts:
For example, financial data has in some cases strong positive and negative correlations between different stocks (feature space) and also strong positive and negative correlations between specific dates (sample space), if surprising positive or negative news was released on those dates.

What are other sources of continuous data, such as audio, visual, or data obtained by some measurement instrument or process, that display the above characteristics? See also "transposable data".

Importantly, it seems that in the sample space often negative correlations are non-sensical or non-causal, which limits the types of data sources with positive and negative correlations in both sample and feature space.


texts:
I want to understand the kernel selection rationale in SVM.
Some basic things that I understand is if data is linear, then we must go for linear kernel and if it is non-linear, then others.
But the question is how to understand that the given data is linear or not, especially when it has many features.
I know that by cross validation I can try and feed different kernels and see the output whichever performs best to be selected, but I'm looking for something anyway to have some early indications.


texts: Score for concordance of two classifications with different number of classes

texts:
I am searching for a score to compare two different classifications of the same observations.

I was thinking about Adjusted Rand Index or Adjusted Mutual Information, but the problem is that the first classification splits the observations in 4 classes, the second in 5 classes. If I understood these metrics correctly, the number of classes should be the same between the two classifications (only permutation of the labels is allowed).

The idea is that, if the score is between 0 (no concordance) to 1 (full concordance) I may get a result like this:

# permutation is ok
score( [1,2,3,4], [2,1,3,4] )  =  1

# subset is ok
score( [1,2,3,4], [1,1,2,3] )  =  1

# this should be less than 1 (last item in the second classification
# may be 1,2 but not 3)
score( [1,1,2,1], [1,2,3,3] )  <  1

Is there a similar score to evaluate concordance of two classifications with different number of classes?


texts: Is Annoy a machine learning algorithm to find nearest neighbor ? and is it similar to K nearest neighbor algorithm?

texts: I was researching about  Google universal sentence encoding and i saw that it uses simple neighbor/Annoy to find the nearest vector for semantic-similarity search engine. This is the first time i'm hearing about it.  Is simple neighbor/Annoy another kind of  machine learning algorithm to find nearest neighbor ? and is it similar to K nearest neighbor  algorithm ?


texts: Photorealistic synthetic data for object segmentation

texts:
Let's say that we have a very small labelled dataset for instance segmentation and there is a photorealistic physics engine available that can produce synthetic data for us. Looking on the web I haven't found any paper about a similar issue.
Has anyone tried to use solely a photorealistic synthetic dataset or mixed with real data?
If so, I would appreciate your conclusions or any relevant work that you can cite here.


texts: X Second samples taken on unevenly spaced intervals

texts:
I have dataset of following specification:

512 samples taken at unevenly spaced intervals over the year
Each sample is an 8 second data from sensors with 4ms resolution
Samples are not labeled

For example, I have 5 samples taken on first day, then more then 10 samples take on 5th day and so on.
I want to cluster data to check if I can infer the mode of operation for the machine from single 8 second sample. Also, I want to measure the performance of the component over the year for predictive maintenance.
Currently I want to use self organizing maps for clustering purposes. I am new to this data science and am currently learning. The usual methods use evenly spaced samples. Also each samples in these cases is single input (Like stock value at the end of a day) instead of X second data taken at time Y.
My question is: How do I input such data into any model?


texts: What is the depth of an image in Convolutional Neural Network?

texts:
I am learning cs231n Convolutional Neural Networks for Visual Recognition. The lecture notes introduce the concepts of width, height, depth. For example,

  In CIFAR-10, images are only of size 32x32x3 (32 wide, 32 high, 3
  color channels)

However, in another example,

  a volume of size [55x55x96] has 96 depth slices, each of size [55x55]

What does width 96 means? Does it mean 96 color channels? Why can we have more than 3 color channels?


texts: Tuning Lexicon Sentiment-values Using Machine-learning

texts: I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable with a machine-learning based tool, or by using an optimisation algorithm such as gridsearch?


texts: Python: Plotting equation with the Dirac Delta function

texts:
I am attempting to plot the expected diffraction pattern from a sharp razor which has a formula of:

$$ I(p) = \frac 14 \delta^2(p) + \frac {1}{4\pi^2p^2}$$
where $p = \frac {\sin\theta}{\lambda}$

I initially want to just make a simple list of x values and feed them into the equation to get y values and plot those, like so:

x = [theta for theta in range(-90,90)]

y = [I(theta) for theta in x]

plt.plot(x,y)

However, I'm not sure how to write $I(\theta)$ because I don't know how to write a dirac delta function in python. How would one graph this function?


texts: Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?

texts:
The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed
#treat extreme values in Item_Outlet_Sales
train['Item_Outlet_Sales_log'] = np.log(train.Item_Outlet_Sales)
test['Item_Outlet_Sales_log'] = np.log(test.Item_Outlet_Sales)

plt.figure(1)
plt.subplot(121)
sns.distplot(train.Item_Outlet_Sales)
sns.distplot(test.Item_Outlet_Sales);
plt.subplot(122)
sns.distplot(train.Item_Outlet_Sales_log)
sns.distplot(test.Item_Outlet_Sales_log);

Then use the new target variable (Outlet_Item_Sales):
#creating dummies for the training dataset
X = train.drop('Item_Outlet_Sales', 1) #drop the log target column
y = train.Item_Outlet_Sales_log

X = pd.get_dummies(X)
train = pd.get_dummies(temp_train)


texts: Is it acceptable to make new independent variables out of old ones?

texts:
I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or at least modify the independent variables in a way to potentially get more worthwhile data).

For example, one of the variables has data on a person's birth state/territory. So instead of trying to model using 50+ categories, creating a new variable that would categorize each state by region (North East, Midwest, South, Etc.) might lead to more significant and easier to interpret analysis.

I know the objective, or what we're trying to answer, is important, but I wonder if making brand new variables or significant changes like this is appropriate for an interview assignment?

Thanks!


texts:
Disclaimer: I am very new to using stackexchange so bear with me.

I am trying to gain experience using neural networks and would like to do a few projects involving them. Right now I am trying to train on the cifar10 dataset and built a quick CNN model in keras. I then realized that this model has 2 million trainable parameters, and as a result the kernel restarts before the model is trained on even 1 epoch. I think this is due to a lack of memory (I have 8gb RAM), but this confuses me somewhat because I've usually heard that GPUs were usually the factor that make machines inadequate for deep learning tasks.

The questions:

Is my kernel restarting because of a lack of memory or processing power?
What are my options? If it's memory, should I just buy another 8GB of RAM and plug it into my laptop? If it's the GPU, should I use AWS (I have before) or just build my own desktop? I would really like to engage in the iterative process of improving a model -- building a baseline model, training, tweak the architecture of the NN, train again, read a paper about how people improved results on cifar10, train again, etc., so I'd imagine using AWS all the time could get expensive quick. I probably would not build a desktop right away since I have to make sure the money would be well spent but would still appreciate being pointed towards some resources by someone experienced.
My third option is just reducing the amount of computation my computer has to do by using smaller/less fully connected layers and using more pooling layers. If I went this route, how limited would I be in terms of what projects I can tackle? I'm particularly interested in image processing projects.

I'm just trying to fully understand all my options from someone that's dealt with this issue. Thank you for taking the time to read my post, I really appreciate any advice given.


texts: Are LDA clusters identical across different runs?

texts:
for a given corpus are the Latent Dirichlet Allocation clusters for it is unique in general?
How about the gensim  multi-process implementation of LDA? are there unique or they will be different for every run of code?


texts:
Question
Which GPUs are supported in Pytorch and where is the information located?
Background
Almost all articles of Pytorch + GPU are about NVIDIA. Is NVIDIA the only GPU that can be used by Pytorch? If not, which GPUs are usable and where I can find the information?


texts: Binary Classification without machine learning tools

texts:
I have to write a binary classifier for my company that should be as simple as possible and doesn't use machine learning libraries (and I also should not code too sophisticated algorithms by myself).

I just came up with the following idea: If I used a boxplot of my features and if the feature value of a new sample is bigger than the whiskers of the boxplot, I say the sample is in category A and if it is smaller than the whisker, it is in category B.
Is this a good approach? (probably not) Are there classification algorithms that are simple enough so that it is feasible to code them without libraries?


texts: training model on random samples from a large dataset

texts: I have a huge data set(More than 1 million data points).My dataset is text. i am doing NER on it to identify few entities. if i randomly choose 100 data points from the total data set and train my model(LSTM), will this yield good results? i will be running for 20k random batches. Does this approximate the data properly or do i need to run for more number of batches than the total number of datapoints?


texts:
I am trying to plot results from document (data source: twitter) clustering using python's sklearn. Below is 10-words of words belonging to each cluster:

donated     0.221646
cross       0.034354
red         0.033817
relief      0.030248
every       0.026620
victims     0.023084
efforts     0.021768
redcross    0.021427
wade        0.019810
million     0.018602
Name: 0, dtype: float64

victims      0.030352
help         0.026362
relief       0.020772
donate       0.020332
donations    0.012362
please       0.012352
want         0.012348
sandy        0.010720
new          0.010089
like         0.008957
Name: 1, dtype: float64

affected     0.179101
help         0.054969
please       0.027684
donate       0.026976
red          0.026417
people       0.026100
cross        0.025541
sandyhelp    0.025537
thoughts     0.023171
like         0.021116
Name: 2, dtype: float64

I am able to generate wordcloud by using Amueller's wordcloud .

I wanted to create a scatter plot of these clusters for an instance in above example, indicate the words are in one cluster have red color, other cluster has green, etc. I want those in one plot.


texts: Is this a task of meta-learning or transfer learning?

texts:
I have a task that I am not able to identify if it is of transfer or meta learning. I want to know this, in order to ask help in solving it, because there are some parts that I have not understood.
The task is the following:
We want that our neural network learns to sum weights.
1)Do the training on MNIST, and on CIFAR10 (as support dataset). We want that performance (accuracy) on MNIST, before and after the sum, to be as close as possible. We use CIFAR10 only to try to learn the operation of sum of models. We are interested in the weights of MNIST, we can theta_mnist, and we hope that they have learnt the operation of sum.
The way to do this is the following: have 3 NNs, with the same architecture, but with different weights, the first one for MNIST, second for CIFAR10, and the third one that have weights (for each layer) of theta_mnist+theta_cifar10. The loss function for this network must be CrossEntropy(MNIST) + CrossEntropy(CIFAR10) + CrossEntropy([MNIST+CIFAR10]-MNIST) (this last term of loss should guarantee that the network on MNIST have similar performances before and after the operation of sum.
2)Do the same thing of 1)replacing MNIST with SVHN and again CIFAR10 as support dataset.
3)Building a NN with weights: theta_mnist+theta_svhn, what are the performances?
Now, step by step, I have to start from point 1. I highlighted the parts that I have not clear. In particular, i have not understood, if I have to define a loss and do the training, how is it possible that the network has the theta_mnist+theta_cifar10? Do you think It means to create a network with those parameters and then start doing learning (but i think it is not useful), or doing like a simultaneously training? But I have no clear how is it possible to define weights and at the same time doing training.


texts: What are the elements in a BERT word embedding?

texts:
As far as I understand, BERT is a word embedding that can be fine-tuned or used directly.

With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one vector per word). This was a problem because it did not take homonyms into account. As far as I understand, BERT tackles this problem by taking context into condsideration.

What does this mean for the word embedding itself? Is there still one vector for each word token? If so, how is context taken into consideration? If no, what is the format of the embedding?


texts: Can't get a regression problem to converge

texts:
I am working on implementing a really simple version of YOLO to learn about pytorch and building deep learning models. My dataset consists of images which have two MNIST digits placed somewhere on the image like this:

I am using a CNN to try and predict which quadrant of the image contains a digit (probability between 0 and 1) and then the center coordinate of the digit (two numbers between 0 and 1). So I have an output that is shaped 2x2x3.
It seems that I can do OK predicting the probability of a digit being contained within a coordinate, but when it comes to predicting the center coordinates of the digit, I can't seem to get anything to work. The coordinates tend to just get stuck predicting the center of each coordinate (see an example below where the yellow dots are the predictions regardless of whether or not I predicted a digit there).

My loss function for the center coordinates is MSE, and it will only use a coordinate if there is a digit available in the quadrant.
If anyone is able to give a little bit of direction or tip or they spot something wrong with my network, I would greatly appreciate it.
My code is available here: https://www.kaggle.com/funky15/mnist-object-detection-v3-yolo-w-data-import


texts: How to plot segmented bar chart (stacked bar graph) with Python?

texts:
cat = {'A':1, 'B':2, 'C':3}
dog = {'A':2, 'B':2, 'C':4}
owl = {'A':3, 'B':3, 'C':3}

Suppose I have 3 dictionary, each containing pairs of (subcategory, count). How can I plot a segmented bar chart (i.e stacked bar graph) using Python with x being 3 categories (cat, dog, owl) and y being proportion (of each subcategory)? What I have in mind looks like this:


texts: which Model to apply on panel data where unique id has 6-8 records and total records are 2,000,000?

texts:
I am new to such panel data where I have multiple observation for same ID in different Quarter and I am not sure what kind of machine learning algorithm I can apply.
I have data from Q1-18 till Q4-2020
I have 2,000,000 rows and 200,000 unique id and 20 columns
For each id I have only 6-8 past quarter values, max quarter for each id are 8 quarters and for some id I have only 6 quarters where few quarter value are not available for that id
Below is the basic idea of what my data set look like
Quarter - respective business quarter for that year
Target - is the sales volume in ratio
I am trying to Predict - Target column for 2021 Q1 quarter
I have 8-10 different numeric columns and state , quarter and ID as category columns
I would appreciate if someone could suggest me what kind of modelling could be performed


texts: Why Sckit's fit_transform causes a huge drop in accuracy and all other evaluation metrics?

texts:
Trying to use sc.fit_transform(X), I get a huge drop in accuracy on the same model. Without scaling the values of my dataset, I get accuracy values of 80 - 82%. When I try to scale them, using sc.fit_transform(X), I get accuracy values of 70 - 74%.

What could be the reasons for this huge drop in accuracy?

EDIT:

Here is the code I am using:

# read the dataset file
basic_df = pd.read_csv('posts.csv', sep=';', encoding = 'ISO-8859-1', parse_dates=[2], dayfirst=True)

# One-Hot-Encoding for categorical (strings) features
basic_df = pd.get_dummies(basic_df, columns=['industry', 'weekday', 'category_name', 'page_name', 'type'])

# bring the label column to the end
cols = list(basic_df.columns.values) # Make a list of all of the columns in the df
cols.pop(cols.index('successful')) # Remove target column from list
basic_df = basic_df[cols+['successful']] # Add it at the end of dataframe

dataset = basic_df.values

# separate the data from the labels
X = dataset[:,0:45].astype(float)
Y = dataset[:,45]

#standardizing the input feature
X = sc.fit_transform(X)

# evaluate model with standardized dataset
#estimator = KerasClassifier(build_fn=create_baseline, epochs=5, batch_size=5, verbose=0)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=seed)
#estimator.fit(X_train, Y_train)
#predictions = estimator.predict(X_test)
#list(predictions)

# build the model
model = Sequential()
model.add(Dense(100, input_dim=45, kernel_initializer='normal', activation='relu'))
model.add(Dense(50, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model
history = model.fit(X_train, Y_train, validation_split=0.3, epochs=500, batch_size=10)

There is a part of code commented, as I tried to use the KerasClassifier in the beginning. But both methods end up with much less accuracy (as stated above), when I use fit_transform(X). Without using fit_transform(X) I get an accuracy of 80 - 82%. Without 70 - 74%. How come? Am I doing something wrong? Does scaling the input data not always lead to better (or almost same accuracy results at least) AND primarily faster fitting? Why is this big drop in accuracy when using it?

PS: 'sc' is StandardScaler() --> sc = StandardScaler()

Here is the dataframe used (in 2 photos, because it too wide to make a screenshot in just one photo) with column 'successful' as label-column:


texts: How to train convolutional neural networks on unbalanced datasets of images?

texts:
How can I train convolutional neural networks on unbalanced datasets of images?

My dataset has around 400 classes and the classes have different number of images..


texts: Why multiply by 2 when calculating partial derivatives during backpropagation?

texts:
I'm wondering why we multiple by 2 when calculating partial derivatives.

I'm referencing the 2's that I've circled below, from here.

We also see this in the python implementation,
m_deriv += -2*X[i] * (Y[i] - (m*X[i] + b)), there is a 2.

How can I understand the intuition here?

My math is not strong so apologies in advance if this is a silly question.


texts: Can we train a neural network to tell if an object is present or not in an Image?

texts:
I am new to machine learning, working on object detection, but not interested in the location of the object in the image, so I just want to know is it possible to train such a neural network, if yes, how? (I just want a list of objects present in the image). I am not sure what kind of dataset I would require to build such a network.

edit: I will be more clear on my question. First I know about CNN and the models like VGG16, Inception, Resnet etc. My doubt is that these networks can be trained to identify an object i.e. it will predict which object is present in the image and give the probability. What if I want to train a network to tell whether particular objects are present in the image or not i.e. The output should be yes if any of the objects is present no if not


texts: Categories with the same mean in target encoding

texts:
While doing target encoding it can happen that two categories have the same target mean.

This is bad because there will be no difference in the new feature in it and we will lose some information.

Also, this is potentially harmful to the model, choosing this split in the feature can produce some incongruences.

Is there any way to fix this problem?


texts: On gradient boosting and types of encodings

texts:
I am having a look at this material and I have found the following statement:

For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly
more computationally efficient use an arbitrary integer encoding [also known as Numeric Encoding] for
the categorical variable even if the ordering is arbitrary [instead of
One-Hot encoding].

Do you know some references that support this statement? I get that Numeric Encoding is more computationally efficient than One-Hot Encoding, but I would like to know more about their supposed equivalence to encode unordered categorical variables in Gradient Boosting Methods.
Thanks!


texts: Should i drop simple features after deriving more complex features from them?

texts:
I know for a fact that complex features projects the data into higher dimensions which makes the previously non-separable data linearly separable.
But, Is this not true that these complex features will be highly correlated with the features from which they are derived.
My question is, Should i drop the simpler features from training set?


texts: Is it possible to make a CS:GO Machine Learning AI?

texts: I am not an expert on Machine Learning, Neural Networks or NEAT. In fact, I probably have no clue what I'm talking about. My question is if you can make a learning AI that learns to play complex multiplayer games and possibly outpreform humans. If it is possible, could you also recommend a language or languages to make this AI in? (I know I'll probably have to take a VACation for botting, but it's something I feel like we should try.)


texts: Leaky ReLU inside of a Simple Python Neural Net

texts:
To build a simple 1-layer neural network, many tutorials use a sigmoid function as the activation function. According to scholarly articles and other online sources, a leaky ReLU is a better alternative; however, I cannot find a way to alter my code snippet to allow a leaky ReLU.

I tried logic like if x > 0 then x else x/100 as the activation function, then the same for the derivative.

Is it failing because the output layer cannot have a ReLU? Should I change the first layer to ReLU then add a softmax output layer?

import numpy as np
np.random.seed(1)

X = np.array([[0,0,1],[0,1,1],[1,0,1],[1,1,1]])
y = np.array([[0,1,0,1]]).T

class NN:
  def __init__(self, X, y):
    self.X = X
    self.y = y
    self.W = np.random.uniform(-1, 1, (X.shape[1], 1))
    self.b = np.random.uniform(-1, 1, (X.shape[1], 1))

  def nonlin(self, x, deriv=False):
    if deriv:
      return x*(1-x)
    return 1/(1+np.exp(-x))

  def forward(self):
    self.l1 = self.nonlin(np.dot(self.X, self.W + self.b))
    self.errors = self.y - self.l1
    print(abs(sum(self.errors)[0]))

  def backward(self):
    self.l1_delta = self.errors * nonlin(self.l1, True)
    self.W += np.dot(self.X.T, self.l1_delta)
    self.b += np.dot(self.X.T, self.l1_delta)

  def train(self, epochs=20):
    for _ in range(epochs):
      self.forward()
      self.backward()

nn = NN(X, y)
nn.train()


texts: Identifying sequences of actions required to complete tasks, based on data of completed tasks

texts:
I have a list of about 20 tasks that a machine need to be performed by a machine. Each tasks consists of a sequence of 3 to 5 actions that must be executed sequentially to complete a task. There are a total of 50 different actions.

I want an algorithm to learn which actions to execute to complete the tasks. To learn this I have a dataset of approximately 1000 samples. Each sample consists of a task with corresponding actions.

Given a task $T_n$, the machine needs to come up with the sequence of actions to complete the task.

Sample dataset ($T$ for tasks out of 20, $A$ for actions out of 50):
$T_1$ - $(A_5 \rightarrow A_1 \rightarrow A_3)$
$T_2$ - $(A_7\rightarrow A_12\rightarrow A_4\rightarrow A_1)$
$T_1$ - $(A_5\rightarrow A_3\rightarrow A_{13})$

What is the best way to solve this?

Based on my knowledge of linear regression and classification, I believe that both are not the right approach to solve the task at hand.

EDIT 1:

One way I can think to solve this problem.
The machine doesn't have to come up with the whole sequence instantly.

If a new task has to be done, the machine has an option to choose the next action (first action) from, say, 5 available actions it can perform. After performing this action, it would have, say, 5 more actions and it has to choose it's next action to complete. And go on doing these actions to complete the given task.

Is this kind of a finite state machine?


texts: How to handle “not label Y” in a multi class machine learning problem?

texts:
I have a train data set that comprises information in the form:

  feature 1, ..., feature N, label
1 x1,        ..., xn,        A
2 x1,        ..., xn,        B
3 x1,        ..., xn,        C
  ...
4 x1,        ..., xn,        not A
5 x1,        ..., xn,        not B
6 x1,        ..., xn,        D
7 x1,        ..., xn,        E
  ...

In the test dataset, the labels A, B, C, D, E should be predicted.

I would like to use the information in row 4 and 5 in order to avoid overfitting.

Which techniques / tools can I use to utilize the "not A" and "not B" information in rows like 4 and 5?


texts: How to count the number of times a sequence in a panda (time) series is less than a specific duration


texts:
I have an item-item similarity matrix. e.g. (the matrix is symmetric, and much bigger):

1.00 0.88 0.96 0.99
0.88 1.00 0.99 0.96
0.96 0.99 1.00 0.86
0.99 0.96 0.86 1.00

I need to implement recommender which, for a set of items, recommends a new set of items.

I was thinking about using SVD to reduce the items to n-dimensional space, let's say 50-dimensional space, so each item is represented with a vector 50 numbers, and similarity between two items is calculated by cosine similarity between two 50-dimensional vectors.

For a base set of items (which can get quite big), I hope I could calculate an average of their vectors, and use it for search.

Is this a good idea? What is this procedure called? And can it be done in Mahout?

EDIT:

This is my code so far:

ItemSimilarity similarity = new LogLikelihoodSimilarity(model);
Matrix m = new DenseMatrix(NUM_ITEMS, NUM_ITEMS);
// copy similarities to a matrix
for (int i = 0; i < NUM_ITEMS; i++) {
        double[] similar = similarity.itemSimilarities(i, range(NUM_ITEMS));
        for (int j = 0; j < NUM_ITEMS; j++) {
            m.setQuick(i, j, similar[j]);
    }
}
Matrix v = new SingularValueDecomposition(m).getV();
Matrix reduced = v.viewPart(0, NUM_ITEMS, 0, 50);

The problem is, SVD is taking forever for NUM_ITEMS > 30. I don't know if there is an issue with data, or with SVD implementation I'm using. The matrix m is symmetrical, could that be an issue? I tried googling "demean matrix mahout" with no results. How should I preprocess it for SVD to work faster? I will need NUM_ITEMS to be about 20.000 - 40.000 in the future. Is this reasonable size for SVD?

EDIT 2:

The problem was the matrix contained a few NaN values, that's why SVD was taking infinite time. After replacing these with 0.0 it works fine for 1000 x 1000 matrix. And my recommendations are working like a charm. I'll still need compute SVD of 20x more rows and columns. If anyone knows what's the easiest way to compute (approximate) SVD of 20.000 x 20.000 dense matrix, probably through some cloud parallel service (?), please let me know.

PS. Thanks for help!


texts: Process and clean a huge .tar file with millions of small XML files

texts:
I have one huge .tar file (~700GB) which includes millions of small XML files (<1MB each). The dataset has a lot of unnecessary information that I don't need in the project I am working on. What I try to do is to read all those XML files, remove the info that I don't need and save the new dataset as a CSV (which I expect to be around a dozen of GBs).

I tried to unpack the .tar file, to read and clean each of them with Python, but after 24hours of unpacking, I wasn't even at half of the process.

I used this command

tar -xvf myfile.tar

How I should proceed with this? What is the fastest way to read, clean and save the new data?


texts:
I am learning about the difference between categorical, ordinal and numerical variables.
From what I understand:

Categorical variables have 2+ categories without any intrinsic order.
Ordinal variables have clear ordering but the intervals are not necessarily equally spaced.
Numerical variables are ordered with equally-spaced intervals.

I am stuck on trying to classify these two situations:
1. What level of commitment are you prepared for?

Multiple times per week
Once per week
A couple times per month
No more than once per month
Level of commitment is not a factor.

2. What level of commitment are you prepared for?

9-11 times per month
6-8 times per month
3-5 times per month
0-2 times per month
Level of commitment is not a factor.

My guess would be #1 is ordinal and #2 is numerical. However, the "Level of commitment is not a factor" is throwing me off. Can anyone help clarify this for me?


texts: Dealing with high frequency tokens during masked Language modelling?

texts:
Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency
Sample Sequence:-
<tok1>, <tok1>, <tok4>, <tok7>, <tok4>, <tok4> ---> here tok4 is very frequent in this sequence

So if I mask some tokens and get the model to train to predict those masked tokens, obviously the model will gain a bias in predicting <tok4> due to its statistical frequency.
Since <tok4> represents important information, 'downsampling' (or removing those frequent tokens) would not be preferred and I would love to have my sequence as intact as possible.
How best should I deal with this? Is there any already established method that can counter this problem?


texts: How to best visualise two sets of data over several years?

texts:
I have a set of data on crime statistics by police force and year, and another set of data showing police workforce numbers over the same period. I am trying to determine the best way to show this data. I have tried three alternatives but looking for preference out of these three or altnernatives that would show the data better (using Tableau but it's the visualisation more than the tool that matters).
First, i created a pair of charts shown one above the other so the X axis is common with separate Y axis.

Then i combined these two into a dual axis chart

And my last attempt was to use a KPI used by the police in some of their reports - the crime rate per 1000 population and create a similar metric for police staff per 1000 population.

So for 2020, this is showing on average across England and Wales, there were 86 crimes per 1000 people and 3.34 police staff.  Dropdowns on the dashboard of each allow changing this to a specific police force and to specific crime groups. The crime rate per 1000 dont match up with the police force reported numbers as they seem to use just the census 2011 figures whereas i have rolledup the population estimates per year (census 2011 + births + immigration - deaths - emmigration). It's still an estimate but should be more accurate than using 10 year old figures.
Any suggestions on how to better present this data ? Thanks in advance.
The full dashboard (still work in progress and draft so apologies for the performance until i determine the right data vis), is on tableau public
https://public.tableau.com/profile/trevor.north#!/vizhome/police-reported-crime-workforce-population-public-v1_12/PoliceWorkforceandReportedCrime-EnglandWales-2007-2020
After some further feedback, i have created two more versions.  One without the map

And one retaining the map but removing one of the panes (that showed %change of workforce).

There is a problem with the map.  I used colour to show the number of offences so London shows as darker colour.  Without modifying the data, all the rest were at other end of the scale.  To rectify, i used a LOG function on the number of offences.  This then shows more colour variation between the different police force regions.  But i now cannot show the legend since that shows a range upto about 7000 whereas the number of offences is over a million in London.  Think i may have to create an information button and an extra layer and manually include a legend on that for the map. Is there a better way ?


texts:
I was wondering what I should be looking into if I want to measure the similarity between a paragraph and a corpus of text.

For example, given a paragraph of text and the entire corpus of Data Science StackExchange questions, how would you determine whether that paragraph of text could be classified as a Data Science question?


texts: how to calculate mean average precision of custom object detection algorithms in python

texts:
I know that for calculating mean average precision first we must have ground truth files for each image in dataset. I am following this
tutorial to detect whether a person has mask on its image or not. I know that It is a classification problem. I have found one github tutorial which has comprehensive details about how to calculate MAP for object detection. But i am totally confused. Because it says that you need to generate ground truth files for each of the image in dataset. As far as I know ground truth files is original area of the object which we want to detect.
After getting the ground truth files we have to calculate IOU of the images and then take Mean of all classes in this case we have two class

either a person a mask
or it does not have a mask.

Kindly clear my confusion and tell me if i am wrong somewhere in my understanding.


texts: I am trying to implement Isolation forest for anomaly detection but I am not able to understand and visualise the decision function

texts:
I am trying to implement Isolation forest for anomaly detection but I am not able to understand the scores. Also, I want to plot and visualise the decision function that my Isolation forest is creating.

Below is the code that I have written :

def IForest(df):
    clf = IsolationForest(n_estimators = 300, contamination = 0.01, behaviour = "new")
    clf.fit(df)
    anomaly_score = clf.decision_function(df)
    anomalies = clf.predict(df)
    print("\n Decision function for Isolation forest")
    print(anomaly_score)
    print("\n The label -1 suggests that the point is anomolous and 1 suggests that the point is normal")
    print(anomalies)
    anomolus_dp = data[np.where(outlier == -1, True, False)]
    print("\nThe data has {} anomolus points and the data points are the following for  : ".format(len(anomolus_dp)))
    print(anomolus_dp)

The output that I am getting is  the following :

On what basis is  this score -0.01010126 is classified ? It is not the probablity of a data point to be considered an anomaly and why is -1(suggesting anomaly) assigned to it, then how can I get the probablity of a a data point to be considered anomlous ?
Also, How do I understand which variable(s) is contributing for this data point to be classified as anomaly ?


texts: What kind of non linear regression model to use when there is very low correlation between independent and dependent variable

texts: I have a data set with one independent variable(X) and one dependent variable (Y). Since my dependent variable is continuous that's why I am trying regression modeling. As can be seen in the image there is no clear correlation happening between my indep and dep variable. I have already tried linear and polynomial regression but it is giving very poor results. I am stuck and looking for some expert advice on what to do? which model to try


texts: How to reduce position changes after dimensionality reduction?

texts:
Disclaimer: I'm a machine learning beginner.

I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing their positions after the modification and updating the 2D-plot. The logic already works, but each iterative visualization is very different from the previous one even though only 1 out of 28.000 features in 1 data point changed.

Some details about the project:

~1000 text documents/data points
~28.000 tfidf vector features each
must compute pretty quickly (let's say < 3s) due to its interactive nature

Here are 2 images to illustrate the problem:

Step 1:

Step 2:

I have tried several dimensionality reduction algorithms including MDS, PCA, tsne, UMAP, LSI and Autoencoder. The best results regarding computing time and visual representation I got with UMAP, so I sticked with it for the most part.

Skimming some research papers I found this one with a similar problem (small change in high dimension resulting in big change in 2D):
https://ieeexplore.ieee.org/document/7539329
In summary, they use t-sne to initialize each iterative step with the result of the first step.

First: How would I go about achieving this in actual code? Is this related to tsne's random_state?

Second: Is it possible to apply that strategy to other algorithms like UMAP? tsne takes way longer and wouldn't really fit into the interactive use case.

Or is there some better solution I haven't thought of for this problem?


texts: Prime model with mock data, good idea or not?

texts:
I am currently working on text classifier with some pretty unique characteristics. The data is composed of about 2K categories but 98% of the data lives in just one of those 2K categories. However, our main problem is that we have a list of additional categories but have yet to see any transactions that fall into those categories (we know they will eventually).

Question:

1) Does it make sense to "prime" the model with these additional categories? For example, I could just add examples of these categories to my training set.

2) If priming is an acceptable are there any best practices or guides in doing so? I am wondering if I should just add the additional transactions to my training corpus, randomize them with other text, how many examples of the training set they should represent, etc..


texts: exporting the model parameters after training and testing

texts:
I want to construct a python micro service to generate predictions based on inputs from web via an API.
I have trained a logistic regression model, but need to create a python script to run in AWS lambda or a container with minimum set of libraries.
How do I see and possibly export the model parameters?


texts:
I'd like to model the evolution of the sales of a store.

Here are the data I have :

Customers are aggregated into monthly cohort depending on the date of the first purchase
eg : customers who did their 1st purchase in Jan 2015 are in the cohort 1, customers who did their 1st purchase in Fev 2015 are in the cohort 2.
Sales : aggregated for each Month * Cohort (cohort 4 is not visible because there is no line for only 1 record)

if we put all the cohorts at the same origin:

Question: How to predict the spending of the next 2 months, ie :

the spending of the cohort 1 in its aging 5 and 6
the spending of the cohort 2 in its aging 4 and 5
the spending of the cohort 3 in its aging 3 and 4
the spending of the cohort 4 in its aging 2 and 3

But also for the cohort who don't exist yet :

the spending of the cohort 5 in its aging 1 and 2
the spending of the cohort 6 in its aging 1

I have 2 methodologies in mind, but don't know if they're good:

Polynomial regression by using 2 parameters:

aging: because there is a clear evolution over the time, up then down.
cohort number: because more recent cohort seems to spend less.

Time series : I didn't perform time series recently I'm a bit rust so I would prefer to use something else but I think it could fit with this problem.

Indeed it seems obvious that yt-1 t in yt-2... are good predictors of yt but how to predict spending of future cohort with no history ?


texts: What is the best tutorial to quickly learning machine learning in R

texts: I want to learn machine learning methods, preferably high-performance methods, by R in the quickest way possible. I am totally familiar with R but very less experience with ML packages. I know some resources like books but what I intend by opening this topic is to find the quickest way possible. I also need to see some examples in the tutorial and I DO NOT need any theory.


texts: How to process categorical variable having lots of unique values in linear regression?

texts:
I have House Price dataset and I am using linear regression to predict the house price.
while data preprocessing I found a variable called "Location" and it have around 342 unique value. For performing regression I have to convert this Location variable to numerical value.
So how do I handle this "Location" variable? or will it be fine to delete this variable?


texts:
In addition to our list of publicly available datasets, I'd like to know if there is any list of publicly available news datasets/crawling APIs. It would be very nice if alongside with a link to the dataset/API, characteristics of the data available were added. Such information should be, and is not limited to:

the name of the news network / news aggregator;
what kind of news information it provides (title, snippet, full-article, date, author, url, ...);
whether it allows for crawling its contents via an API (and rate: 10/min, 1k/month, ...);
whether it simply provides a snapshot of the whole dataset.
news sources covered
date range of news content coverage

Any suggestions and further characteristics to be added are very welcome.


texts: API for Company Data Enrichment Suggestions

texts:
I'm looking for API suggestions for enriching data on companies. Currently I use the Crunchbase API to look up a company's name or domain and I am trying to gather the domain/name (if I don't already have both), contact email (this one is a long shot), and the location of their headquarters.

This works incredibly well if Crunchbase has the company in their API, but I'd say this only happens about 25% of the time.

I'd love to get some suggestions on some free APIs that I could use along with Crunchbase. I'd also love to see if anyone has had positive or negative experiences with paid APIs!


texts: String similarity algorithms for string containment (rather than string equality)

texts: I saw that there are a lot of string similarity algorithms to whether two strings are the same. I  have a slightly different problem - I get two strings "a" and "b" and I need a similarity algorithm to whether "b" contains "a" ("b" is likely to contain "a" with some "error"). I haven't found any algorithms to solve this problem and I'd be happy to hear about some (if they exist).


texts: How to select between models when AUC scores are similar?

texts:
I use two machine learning algorithms for binary classification and I get this result :

Algo 1 :

 AUC- Train : 0.75      AUC- Test: 0.65          big Train / overfitting

Algo 2 :

 AUC- Train : 0.72      AUC- Test: 0.65          small train / small overfitting

Which one is better?


texts: Syntax of the inputs for LightGBM binary build (without python API)

texts:
I built the LightGBM binaries from https://github.com/microsoft/LightGBM. The build was successful, but the python API did not work for me. I am using the light gbm binary directly to run for time being.

I tested the binaries on LightGBM/examples/regression. The train and predict work fine using

lightgbm config=train.conf
lightgbm config=predict.conf

The outputs are _model.txt and _predict_result.txt

The problem I am facing is I cannot make out which in the input file regression.train is the target variable.

Also, I am looking for some documentation that explains how to run LightGBM without python API.

Thanks for the help.


texts: Different methods for clustering skills in text

texts:
Consider a talent pool in which each member has some set of skills. Some of these talent are submitted to orders as potential candidates of which one is selected. It is reasonable to assume that the submitted talent have some dominant thing in common in their skill sets (let's call it a segment) that qualifies them for the order. Example segments are "front end web-designer" or "brochure / sprint designer".

Given the total set of skills over all of the talent submitted to an order (like 2-5 with say 10 skills each, so 20 - 50 skills total), I am looking for the dominant segment. Then, I am looking for the dominant segment for each individual talent.

My plan is to use latent Dirichlet allocation (LDA) such that the skills of all the talent submitted for an order are a "document" that contains some segments or "topics" with some probability. Likely, there will be one or two dominant topics depending on the total topic number. I will then use this model to predict the dominant segment for each talent where the individual talent skill set is a "document" with some segments or "topics" within.

I am curious if anyone has feedback about my use of LDA or other ideas about how I might go about discovering these segments?


texts: In my project I use 2 MLP ANNs: for classification and for prediction. Basic training dataset is stored in database (MS SQL Server). But we need to make future incremental learning possible. It means that ANN is used in industrial realtime system, operator marks observations in database as objects of a particular class and ANN must take it into account. How can I do it?


texts:
I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are:

Apache Flink
Google DataFlow
Apache Storm
SAS

Can any of these can be considered as a complete alternative/better than Spark? Or Spark is the only comprehensive Big Data analytics/machine learning engine there is these days?


texts:
I just saw this video on Youtube.
On what principle did Google's DeepMind learn to walk?
Was it Q-Learning or a Genetic Algorithm or Policy Gradient?


texts: Minimizing an upper bound of objective function

texts:
In many machine learning problems, you often have an objective function (e.g., cost, loss, error) that you want to minimize.  Instead of directly minimizing this objective function, I sometimes see papers that derive the upper bound of the objective function, and then minimize the upper bound version instead.  The reason for this seems to be that the original objective cannot be obtained by the data we have at hand, or just hard to minimize for some reasons.

My question is, does it make sense to minimize the upper bound?  Are there guarantees that they lead to the same solution?  My concern is that the minimizer can obviously become different in the two cases.  If this is the case, do we know how close the two can be?  Are there good textbooks or papers that explain these issues?

Thank you very much for your help!


texts: Can I use sentiment analysis techniques and text processing for detection social bot in online social networks?

texts: I want to study in field of " Detecting fake account on online social network by content/sentiment analysis " for Master's Thesis.is this possible?


texts: How to reduce overfitting in a pre-trained network

texts:
I have a custom dataset with 10 classes and I am using a pre-trained resnet18 model from torch-vision. I can clearly see it's over-fitting because: the model is trained for 75 epochs with a batch size of 4 and from epoch 30 the validation accuracy stops increasing and the training accuracy keeps increasing.
Things I did to improve generalisation performance and reduce overfitting:

I normalized my data with a calculated mean and std on my training data.
I added random rotation, random horizontal flip and random vertical flip as data augmentations.

The above were the results after all of these, I can't think of other regularization techniques that produce regularization effects without changing the architecture of the model which will completely contradict the purpose of a pre-trained network, and I am using one because I only have around 300 images in my each class.


texts: Creating new column in dataframe based on conditions in 2 other columns

texts:
I would like to create a new column in my dataframe based on values from both the gender and experimental_grouping columns.

As I have it written below, the column df['group_gender'] has 'control_m' in every row.

Any ideas on why the if statement isn't working/a better way to accomplish this goal?

def group_and_gender(group):
    for i in group:
        if i == 'Controls':
            for i in df['gender']:
                if i == 'M':
                    df['group_gender'] = 'control_m'
                else:
                    df['group_gender'] = 'control_f'
        else:
            for i in df['gender']:
                if i == 'M':
                    df['group_gender'] = 'dp_m'
                else:
                    df['group_gender'] = 'dp_f'

group(df['experimental_grouping'])


texts: How to calculate KL-divergence between matrices

texts: Given there are two matrices of dimensionality 100x2 with absolute values ranging from -50 to +50. Is it possible to determine the kl-divergence by applying the entropy algorithm from scipy.stats to the two flattened vectors of size 200?


texts: Should you cluster before performing collaborative filtering?

texts:
So I am building a recommendation model using customer and product information. This will be done via implicit, that is, a customer has a product or not as we don't have rating information about products.
I want the model to recommend a product using collaborative filtering so I am thinking of using a recommender model like Matrix factorisation/ SVD etc. I will perform multiple tests/experiments to determine the best algorithm to go with.
Should I perform clustering first to determine similar customers based on their demographic and geographic information before the recommender? Then subsequently should I train a recommender model for each cluster?
Would there be any benefit in this approach or would the recommender model be able to naturally determine similar customers?


texts: What is the input dimension for this keras model?

texts:
I'm doing a tutorial, where I have to evaluate the sentiment of IMDB reviews, positive or negative.
I first created an index for each word and then replaced every word in each review for each respective index.So now I have a list of lists of reviews, with 18000 reviews but every review has different lengths, what input_shape should I pass to the dense layer? I've tried padding with zeros so all the inputs are of the same length but I can only get 0.5 accuracy everytime.

training_reviews = list()
for review in reviews:
    indices = set()
    for word in review.split(" "):
        if(word in word2index.keys()):
            indices.add(word2index[word])

    training_reviews.append(list(indices))

model = Sequential()
model.add(Dense(512, input_shape=))
model.add(Activation("sigmoid"))

model.compile(loss="categorical_crossentropy", optimizer="adam", metrics = ['accuracy'])


texts: Use Feature Constructor in Orange to extract number from string?

texts:
I would like to use the Feature Constructor to extract numbers from a column of strings.

 Input          Output
S.c 3224   ---> 3224
South      ---> 0
322523     ---> 322523
S-1c 4     ---> 14
-1.37      ---> 137

Internal Orange 3.7.0 Documentation says For continuous variables you only have to construct an expression in Python.  Great!

I found information about allowed functions in the feature constructor code:

#Only expressions with no list,set,dict,generator comprehensions
#are accepted.
<...>
__ALLOWED = [
    "Ellipsis", "False", "None", "True", "abs", "all", "any", "acsii",
    "bin", "bool", "bytearray", "bytes", "chr", "complex", "dict",
    "divmod", "enumerate", "filter", "float", "format", "frozenset",
    "getattr", "hasattr", "hash", "hex", "id", "int", "iter", "len",
    "list", "map", "memoryview", "next", "object",
    "oct", "ord", "pow", "range", "repr", "reversed", "round",
    "set", "slice", "sorted", "str", "tuple", "type",
    "zip"
]

Based on the list, I was hoping to use an expression like this:
int(filter(str.isdigit, inputStringColumn) or 0)

...but that is an "Invalid Expression" (no further feedback)

How can I extract a number from a string?


texts: How to Identify p (lag order) for ARIMA Model in Python

texts:
here is my auto correlation plot.
Generated by the following python code.

from pandas.plotting import autocorrelation_plot
from statsmodels.tsa.arima_model import ARIMA

def display_corelation(series):
    autocorrelation_plot(series)
    plt.show()

I know i can pass 1 or 2 in p by looking into the plot for the ARIMA Model.
My question is how i can generate the p value,

how to calculate the lag order from some series by pandas or any library instead of plotting currently at the moment ?

model = ARIMA(history, order=(1,1,0))


texts:
I have prepared Jupyter Notebook with some findings and I shared it with other team members through GitHub to get their feedback in a written form. It used to work like this when working together on a piece of code but does not work for Jupyter Notebook. In GitHub that would mean commenting on HTML or JSON level (internal markup for .ipynb files), not on the document level. An alternative would be for team members to clone the repo and puts inline comments in the document. That's an additional effort for other team member I would like to avoid.

What is the way you collaborate, peer review and provide feedback when working on Jupyter Notebooks?


texts: Is SuperLearning actually different to stacking,or are they essentially the same thing?

texts: Articles which use the terms 'stacking' and 'Super Learner' often seem to use the terms interchangeably. Is the Super Learner algorithm a specific form of the more generic stacking concept, or is Super Learner essentially the same thing, and at some stage one of these terms is likely to become redundant?


texts: Pearson's Coefficient for classification problem

texts: Can we use Pearson's Correlation Coefficient to calculate the correlation between features and target in the case of a classification problem?


texts: How should a decision tree handle an attribute that can be anything?

texts:
Say I have AttributeA that can take values A1, A2, A3, AttributeB that can take values B1, B2, B3, etc. and I know ahead of time that my classification table looks like

AttributeA | AttributeB | AttributeC | Classification

A1 | B1 | anything | Class 1

anything | B2 | anything | Class 2

A3 | B1 | C2 | Class 3

A2 | anything | C3 | Class 4

...

I'm curious how I would modify a decision tree to handle attributes that can take on any value. One idea I had was to change single rules with "anything" into multiple rules where every possible value of that attribute is explicitly stated. For instance, the rule A1 | B1 | anything | Class 1 could be changed into the three rules:

A1 | B1 | C1 | Class 1

A1 | B1 | C2 | Class 1

A1 | B1 | C3 | Class 1

I'm sure this would work, but I'd like to see if there are any existing decision tree implementations that can handle "anything" or "does not matter" entries.


texts: Tell me if there are any w2v models that do not require a dictionary. So, everything that I found in torchtext first wants to know the dictionary build_vocab. But if I have a huge body of text, I would like to have a model that works at the level of phrases. But I did not find one.


texts: How to customize a objective funtion in xgboost?

texts:
I am new to xgboost and not so good at math.
I want to use a self defined objective function, here is the expression:
obj = (y * y_pred).sum() / (y_pred).abs().sum()

y is a n-dimensional vector.
I have some experience for tensorflow, in tensorflow,  I just need to define the function, it will calculate the derivatives for me automatically.
But in xgboost, the customized obj function needs to return the first and second derivatives.
I dont know how to take derivatives for abs(). Is there good method can take derivatives?
Can you help on this?


texts: How to handle features containing strings in XGBoost in AWS Sagemaker

texts:
How can i handle the string containing spaces and colons as a feature for my xgboost classifier model?
AWS Sagemaker requires the input in csv format, I don't know how to convert the string to the suitable format using CountVectorizer etc


texts: running scikit-learn on Sypder with Python 3.5

texts:
I am trying to print a accuracy score and get this message:

<function accuracy_score at 0x0000018A76046840>

So I am wondering if I am missing a package or is something installed incorrect?

Code:

from sklearn import datasets
from sklearn.metrics import accuracy_score
from sklearn import svm
iris = datasets.load_iris()
X = iris.data
y= iris.target
#print (iris.keys)
print('**********')
#print (iris['feature names'])
print('**********')
#print (iris)
#print (iris['data'].shape,'legth: ',len(iris['target']))
clf = svm.SVC(gamma=0.001, C=100)
clf.fit(X,y)
y_hat=clf.predict(X)
accuracy_score(y, y_hat)
score = accuracy_score
print (score)


texts:
I have a pandas data frame like this:

(index) 0           sie

0       1997-01-01  11.2
1        1997-01-03  12.3
2        1997-01-04  11.5
...
12454    2017-02-01  13.2

I would like to use SVM to predict the future values of the sie. How can I implement python code to predict these values?

I am doing something like this:

model = svm.SVR().fit(df[0],df['sie'])

But it is giving me this error:

ValueError: Found input variables with inconsistent numbers of samples: [1, 12455]

Although both df[0]anddf['sie'] have same shape of (12455,)

Note: I don't have continuous data (some dates, in between, are missing), also values in 0 are datetime.date() objects.


texts:
I have the following vector
my_vector=[0.059; 0.223;0.197;0.176; 0.173; 0.171; 0.0421; 0.209; 0.556;0.252;0.198; 0.255; 0.130; 0.176; 0.110; 0.0845; 0.270; 0.192; 0.199; 0.348]

I want to change the vector in such a way that the output derived vector will have the highest possible distribution change. This means the distribution of the two vectors should be highly different (say have large KLD values) but their distribution should be still from the same family (say the F-test or two-sample K–S test will not reject the hypothesis that they are from the same distribution).

How could I generate the new vector from the base vector while I meet the constraint (having "highest" "possible" change in distribution)? I need a machine learning (data-driven) method to achieve such a goal.

Thanks in advance for any help and recommendation.


texts: Rolling average: when is it possible to consider it?

texts:
I would like to know if I can consider rolling average to predict the future trend of sells.
I collected data from January 2020 to March 2020, day by day, on sells in a shop and I would like to run some analysis.
I was considering rolling average on multiple periods (5 - 10 days). Do you think it might make sense?


texts: In a SVD with user/video bias, why is the UV contribute so small?

texts:
I'm testing a SVD-based collaborative filter on my data set, in which the label, $r_{ij}$, is a real value from 0 to 1.

Like the many papers suggested, to have a better performance, instead of using $ \hat{R} = U \cdot V^T $ directly, I use $\hat{R} = \mu + B_u + B_v + U \cdot V^T $, where $\mu$ is the average rating, $B_u$ is the bias of user, and $B_v$ is the bias of item.

Thus, this model corresponds to a loss function:
$\min_{B_u, B_v, U, V}  = ||I\circ(R-\mu-Bu-B_v-U\cdot V^T)||_F^2 + \lambda (||B_u||_F^2 + ||B_v||_F^2 + ||U||_F^2 + ||V||_F^2)$

where I is the masking matrix in which $I_{ij} = 1$ if $R_{ij}$ is known, and $||\cdot||_F$ is the frobenius norm.

Then, I solve this by gradient descent, it seems to work fine, and the test RMSE is 0.25.

However, when I investigate the contribution of each part in predict function $\hat{R} = \mu + B_u + B_v + U \cdot V^T $, I notice that, $\mu$ is about 0.5, $b_u$ and $b_i$ are about $\pm0.3$, but the part of $ U \cdot V^T $ is quite small, normally about $\pm 0.01$.

Why does this part contribute so small? Since this part is the actual part where the collaborative filter works, I expect it to contribute more in prediction.


texts: One hot encoding alternatives for large categorical values

texts:
I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns?
I found this interesting link.
But they are converting to class/object which I don't want. I want my final output as a data frame so that I can test with different machine learning models? Or, is there any way I can use the generated matrix to train the other machine learning models other than Logistic regression or XGBoost?
Is there anyway I can implement it?


texts: Q Learning Neural network for tic tac toe Input implementation problem

texts:
I've recently become interested in machine learning, specifically neural networks, and after creating ones to solve basic problems such as XOR and Sin and Cos graphs, however i am now looking into reinforcement learning and specifically q learning with neural networks, to try this out and see if i could implement this, i used a q learning neural network for tic tac toe. However I am unsure as to why my program does not learn correctly. One idea I have is that i have implemented the input neurons incorrectly.I am using visual basic console application so just let me know if you need me to post any code

My view on how to implement this:

Use 18 input neurons and have the first 9 being the state before placing a move, and the next 9 being the state after placing a move.

another question I have in terms of when you teach the neural network, would you feed it the old state and then the state after both you and the opponent have made a move?al networks, so they can play against each other and teach themselves

Second Problem

This next potential problem i have is how I use my threshold and exploration values to choose whether to pick a random action or one the neural network chooses. I increase my threshold linearly from 0 to 1 throughout the iterations. while exploration is a random value between 0 and 1. is this correct? or is there a better way of doing it?

Any feedback would be greatly appreciated, and if anyone has any problems with my question such as it being unclear, not making sense or anything else please let me know so i can fix it

thank you for all who take the time out to try and help.


texts:
Here's the description about the data usage for ILSVRC 2016 of ImageNet. I've interpreted it with the table as follows,

Data          Image        Label    Changed    Release_Time
-----------------------------------------------------------------
training      1,200,000    1,000    no         always public
validation       50,000    1,000    some       upon registration
test            100,000        0    some       final evaluation

My question is specific about the test data. Is it chosen from the same set of 150,000 images since ILSVRC 2012, although randomly ? If so, most of the test data has been made public before the final evaluation phase of the challenge, how the result could remain fair enough ?


texts: After choosing top models in classification? Can I apply it on the rest of my dataset

texts:
I am working with a corpus that has 5 datasets in product reviews (A, B, C, D and E), mine is a text classification problem and I need to find the best 5 top models in terms of classification performance (F1).

I started with collection A: the mp3 reviews,  Because it has the largest numbers of documents (900: yes, 750: No).

I trained the data using 10-fCv using different algorithms and pre-processing tasks, got the weighted results for all experiments.

I chose the top 5 models and I want to apply them to the rest of the corpus: B, C, D and E (other products' reviews).

My plan is to run 10-fCv and get the results for all the collections and compute the Micro-average for precision, recall and F1.

Is this the right way to choose a model for a large collection?


texts: tensorflow pseudo inverse doesn't work for complex matrices!

texts:
The Tensorflow documentation here says that:

  tf.linalg.pinv is ''analogous to numpy.linalg.pinv. It differs only in
  default value of rcond''.

  However, tf.linalg.pinv requires the matrix to be float type while
  np.linalg.pinv could be used with complex matrices.

I was wondering why they would only create it for float types and if there is a straightforward way to modify tf.linalg.pinv to be used with complex matrices.


texts: Implementing a neural network: low accuracy

texts:
I'm training my neural network implementation on the MONK's problems data set. I read that it can be possible to achieve almost a 0 loss with the hyper-parameters I'm using (chapter 9). These are my results on training set after 90 epochs, they aren't good as you can see:

Training error after 0 iterations: 0.26427015361
Training error after 20 iterations: 0.231331617693
Training error after 40 iterations: 0.216396390383
Training error after 60 iterations: 0.198107908553
Training error after 80 iterations: 0.166181613583
TrErr: 0.155389641876

And here is the code of my neural network. I think the code is quite simple and readable for a expert, tell me if you need any clarification.

# paramaters
etas = [0.1]
hidden_dim = 2
alpha = 0.5
reg_lambda = 0.0

input_dim = 17
output_dim = 1

# read and parse MONK data
def parse_data(text):
    with open(text) as f:
        content = f.readlines()
    t = np.zeros((len(content), output_dim))
    X = np.zeros((len(content), input_dim))
    for i,val in enumerate(content):
        s = val.split(" ")
        t[i] = float(s[1])
        for idx in xrange(17):
            X[i][idx] = 0.0
        j = -1
        # |0 |1 |2 |3 |4 |5 |6 |7 |8 |9 |10|11|12|13|14|15|16
        # |a1|a1|a1|a2|a2|a2|a3|a3|a4|a4|a4|a5|a5|a5|a5|a6|a6
        X[i][int(s[2])+j] = 1.0 # a1
        j = 2
        X[i][int(s[3])+j] = 1.0 # a2
        j = 5
        X[i][int(s[4])+j] = 1.0 # a3
        j = 7
        X[i][int(s[5])+j] = 1.0 # a4
        j = 10
        X[i][int(s[6])+j] = 1.0 # a5
        j = 14
        X[idx][int(s[7])+j] = 1.0 # a6

    return (X, t)

def sigmoid(x):
    output = 1/(1+np.exp(-x))
    return output

def sigmoid_output_to_derivative(output):
    return output*(1-output)

# initialize weights
W0 = np.random.uniform(-0.7,0.7,size=(input_dim,hidden_dim))
W1 = np.random.uniform(-0.7,0.7,size=(hidden_dim,output_dim))
b0 = np.zeros((1, hidden_dim))
b1 = np.zeros((1, output_dim))
dW0 = 0
dW1 = 0

for j in xrange(90):
    # forward propagation
    l0 = X
    l1 = sigmoid(np.dot(l0,W0) + b0)
    l2 = sigmoid(np.dot(l1,W1) + b1)

    # back propagation
    l2_error = t - l2
    l2_delta = l2_error*sigmoid_output_to_derivative(l2)

    l1_error = l2_delta.dot(W1.T)
    l1_delta = l1_error*sigmoid_output_to_derivative(l1)

    if (j% 20) == 0:
        print "Training error after", j, "iterations:",np.mean(np.square(l2_error))

    dW1 = eta*(l1.T.dot(l2_delta) + reg_lambda*W1) + alpha*dW1
    dW0 = eta*(l0.T.dot(l1_delta) + reg_lambda*W0) + alpha*dW0
    db1 = eta*(np.sum(l2_delta, axis=0))
    db0 = eta*(np.sum(l1_delta, axis=0))

    W1 += dW1
    W0 += dW0
    b1 += db1
    b0 += db0


texts: Forecasting time series outside the training/test set

texts:
I am trying to predict some time series based on precedent values using LSTM.

I have pretty good results when I compare the predicted time series with the test set (0,18% error)

I just miss how to forecast outside the interval of data ^^'

I have to admit that I used a point by point prediction method that looks like this:

def predict_point_by_point(model, data):
    predicted = model.predict(data)
    predicted = np.reshape(predicted, (predicted.size))
    return predicted

I then, I used it to override the predict function.
maybe the original function could have nailed the prediction to have a future time series? maybe the point by point isn't that bad neither?

I mean; how could I predict, some precise interval of time series (3months for example) without just reffering to the test set?

Example: the test set starts 01/01/2018 and ends 01/12/2018 and I want to predict 4 months from 02/12/2018

Thanks in advance for your help


texts: why we sample when predicting with Recurent Neural Network

texts:
I trained a Recurrent Neural Network to predict the next word in a sentence.

I trained and now I want to predict, but there is something I am not getting well.

I saw it in many tutorials even in the Keras documentation.

When we want to predict, we get the probabilities from the predict function and we pass it to the sample function bellow.

def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)

I am not getting what this function is doing.

I know that :

My predictions are an array of probabilities and from that, the word indice is the word with the largest probability.

From the function I can see that they are doing log, and then exp, that is what I am not getting.

Why are they doing that?

I saw this in many tutorials but no one explained it well.

Probably I should go back to my statistics and probabilities courses.

Please point me to the right direction.


texts:
I am trying to understand neural networks and how they work, by programming my own one from scratch in nodejs. Currently, i managed to build a network, that has weights, layers and neurons. I also understood what an activation function is and i am using the sigmoid function.

I've now come to the point of back propagation with the gradient descent algorithm. My Problem is, that i only know 10th grade Math and every tutorial/explanation i managed to find, uses complex functions and math which i cannot manage to understand.

I would really like to finish this project and get back propagation to work, so if someone could explain me how to use the back propagation without using too complex math (e, derivatives, functions, etc.) or explaining the complex math it would be greatly appreciated.

My source Code: https://gitlab.com/milan44/node (Training is happening in Network.prototype.train)


texts: Linear regression with a fixed intercept and everything is in log

texts:
I have a set of values for a surface (in pixels) that becomes bigger over time (exponentially). The surface consists of cells that divide over time. After doing some modelling, I came up with the following formula:

$$S(t)=S_{initial}2^{t/a_d},$$

where $a_d$ is the age at which the cell divides. $S_{initial}$ is known. I am trying to estimate $a_d$. I simply tried the $\chi^2$ test:

# Range of ages of division.
a_range = np.linspace(1, 500, 100)

# Set up an empty vector to store the chi squared value
chi_sq = np.zeros(len(a_range))

# Iteration through division ages
for i in range(len(a_range)):
    # Compute the expected value at each time point.
    expect = cell_area[0] * (2**(time_range/a_range[i]))

    # Compute chi squared
    chi_sq[i] = np.sum((cell_area - expect)**2)

# Plot chi squared test
plt.plot(a_range, chi_sq, '.')
plt.yscale('log')

# Labelling
plt.xlabel('division age [min]')
_ = plt.ylabel('$\chi^2$')

but the minimum is always at the upper bound of the age range, depending on what I set this range to be. It doesn't seem right. So I linearized the model:

$$ln(S(t))=ln(S_{initial})+(1/a_d)ln(2)*t,$$

which is now just simple linear regression with a fixed intercept.

Questions:

Why didn't the first method work?
Are there any resources on how to implement the above regression in Python? I'm new to this and everything I found was very simple, but I don't know how to deal with logs and fix the intercept.


texts:
I have used

nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features)

distances, indices = nbrs.kneighbors(item_features)

to find some suggested items based on some attributes in the dataset.
I really find the suggested items but I need to see if these items are really the more nearest items.

I tried this function to find the similarity between distances of A with each items  B or C or D
spatial.distance.cosine(distances of A,distances of B/C/D)

One output for my code is

B with 0.0004864387565900463
C with 0.003675794744087746
D with 0.05855020953442236
E with 0.0048087457894999686

I thought the items must be orderd from the neighrest but E is break the rule.
Is those distances in above is logic or there is a wrong.

How can I make sure my algorithm work well???


texts: How do I create a Dataset with a single feature?

texts:
I am trying to use Python and tf.keras to build a neural network differential equation solver. I have normal python code that does the computation without issue, but I want to convert to Tensorflow to take advantage of the additional features.
My problem is pretty simple:

I select a set of n training points in one dimension. For example, if n=11, the training points could be:

[0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

Each training point should result in a label of 0. In other words, the mathematical equation being examined, evaluated at the training point, should evaluate to 0.

Since I am new to TF, I based my early attempt on the standard Iris classification example at:

https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough
That example , as described, works fine.

To adapt the example to my problem, I modified the example so that my CSV file for the data is as follows:

11,1
0.0, 0.0
0.1, 0.0
0.2, 0.0
0.3, 0.0
0.4, 0.0
0.5, 0.0
0.6, 0.0
0.7, 0.0
0.8, 0.0
0.9, 0.0
1.0, 0.0

Note that no output labels are provided in the first line.

I ingest the training dataset normally:

# Read the training data, and convert to a TensorFlow Dataset.
column_names = ["x", "G"]
feature_names = column_names[0]
label_name = column_names[1]

print("Features: %s" % feature_names)
print("Labels: %s" % label_name)

train_dataset = tf.data.experimental.make_csv_dataset(
    "./input.csv",
    11,  # Batch size of 11 since there is only 1 batch, right?
    column_names=column_names,
    label_name=label_name,
    num_epochs=1
)

I create a simple 1-10-1 network as follows:

# Create the neural network.
model = tf.keras.models.Sequential()

# Use 10 nodes in the hidden layer.
H = 10
model.add(tf.keras.layers.Dense(H, activation="sigmoid", input_shape=(1,)))

# Add the 1-node output layer.
model.add(tf.keras.layers.Dense(1, activation="sigmoid"))

Following the Iris example, I then do a dry run of the initialized model to make sure the output has the correct structure:

predictions = model(features)

At this point, the exceptions start. The bottom of the stack trace is:
~/anaconda3/envs/tensorflow/lib/python3.7/site-packages/six.py in raise_from(value, from_value)
InvalidArgumentError: In[0] is not a matrix [Op:MatMul]

I know this is not the best way to approach the problem at hand, but I am trying to move forward from a known working example. I am pretty sure this error is arising because my "features" vector is in fact a scalar, and/or the fact that there are no output "labels" to speak of. Does anyone have any suggestions on how I can restructure this Dataset to do what I need? I am sure it can be done, and I am sure the solution is simple - I am just not experienced enough yet with TF and tf.keras to get this to work.
Thanks in advance,
Eric Winter


texts: Can the performance of a CNN be dependent on the train-test-val split random seed?

texts:
I am doing multi-class classification and comparing the effects of 2 image enhancement techniques (IET).
IET 1 performs better than IET 2 at random seed x (for train-test-val split)
IET 2 performs better than IET 1 at random seed y (for train-test-val split)
Is this normal, or am I doing something wrong?