texts:
I'm currently analyzing Korean social media text. The below are the steps of the analysis.

Collect/crawling text data from social media (e.g. Twitter, Facebook), which are related to specific topics.
Analyze the data by using BERT. This includes text classification and sentiment analysis.

However, I've faced some complicated problems.
In step 1, I searched with some relevant keywords to the topic, but indeed, the collected data are not related to the topic. Some of them are related to the topic but some of them are not. In this case, how can I get these topic-related data? Should I annotate the data and training a model to classify the relevant/non-relevant data? Is there any unsupervised approach such as topic modeling? Is there any method that lets me collect topic-related data?
In step 2, I actually got documents, sentences, or paragraphs rather than just one sentence. Therefore the topics and sentiments are mixed in the documents and I'm really struggling to annotate them. I tried to separate the documents into smaller ones but there are no thresholds to distinguish them since they are social media text. Even if I could, this makes another problem which is the context of the document is disappeared and this kills the topic or sentiments.
Here's the summary:

Collecting steps: Collect data related to a specific topic

Problems: The collected data are actually not related to the topic

Analyzing steps: Analyze (classification) the data by using BERT

The labels are mixed in the collected data. How can I split them into some meaningful pieces (big enough to maintain the context of the documents, small enough to analyze them easily)

Thank you.


texts: CycleGAN: Generator losses don't decrease, discriminators get perfect

texts:
So I´m training a CycleGAN for image-to-image transfer.

The problem is: while the discriminator losses decrease, and are very small now, the generator losses don't decrease at all.

The generator loss is:

1 * discriminator-loss + 5 * identity-loss +
10 * forward-cycle-consistency + 10 * backward-cycle-consistency

Somehow the discriminator models, although not being that big, become very strong quite fast, while the generators don't seem to learn much at all.

What do I have to change to resolve this problem and make the generators more effective?


texts:
I am attempting to use the tm package to convert a vector of text strings to a corpus element.

My code looks something like this

Corpus(d1$Yes)

where d1$Yes is a factor with 124 levels, each containing a text string.

For example, d1$Yes[246] = "So we can get the boat out!"

I'm receiving the following error:

"Error: inherits(x, "Source") is not TRUE"

I'm not sure how to remedy this.


texts:
When a variable is created, it is added to collections to allow sharing and reusing. However, this seems like it's a potential for "memory leaks" (strictly speaking, it's not, since there still exists a reference). How should I deal with this?

If I do something like:

import tensorflow as tf
g = tf.Graph()
with g.as_default():
    tf.get_variable('myvar', shape=(2, 2))
del g

will myvar be garbage collected?


texts: Imagining a Linear Regression model with more than 3 dimensions

texts:
I'm just getting started with Machine Learning and this is really bugging me now.
Assuming we could use more than 2 feature variables to train a Multiple Linear Regression model, how can we imagine the geometry of the model?
Let me elaborate.
For a linear regression model with a feature variable and a response variable, we come up with an equation of a line (y = mx + c) and for a multivariate linear regression model with 2 predictor variables, we can figure out the equation of a plane (a(x-x1)+ b(y-y1) + c (z-z1) = 0).
How can we imagine the model in 4 dimensions or more, since we are humans and as a layman, we cannot imagine higher dimensions like physicists?


texts:
We are working with a complex application i.e. a physical measurement in a lab, that has approximately 230 different input parameters, many of which are ranges or multiple-value.

The application produces a single output, which is then verified in an external (physical) process. At the end of the process the individual tests are marked as "success" or "fail". That is, despite the many input parameters, the output is assessed in a boolean manner.

When tests fail, the parameters are 'loosened' slightly and re-tested.

We have about 20,000 entries in our database, with both "success" and "fail", and we are considering a machine learning application to help in two areas:

1) Initial selection of optimum parameters
2) Suggestions for how to tune the parameters after a "fail"

Many of the input parameters are strongly related to each other.

I studied computer science in the mid-90s, when the focus was mostly expert systems and neural networks. We also have access to some free CPU hours of Microsoft Azure Machine Learning.

What type of machine learning would fit these use-cases?


texts: Can SMOTE be applied over sequence of words (sentences)?

texts:
I have a highly unbalanced text classification data. I am trying to over-sample through SMOTE.

I have a doubt that applying SMOTE over sequence of word indices will give me valid data points or not (since SMOTE used nearest neighbor like method)?


texts:
I am curating a large quantity of data from different sensors. If I know that a particular sensor was broken or poorly calibrated for a particular time range, what would be a useful way of annotating the data to make it clear that the data are of poor quality and / or have known errors?

I am thinking a set of key:value pairs (like quality:error, description:'sensor was broken') that I can store in json, yaml, image header (e.g. exif) metadata or similar. If there are any standards or conventions for this type of annotation I would like to adopt them.


texts: Is there any inductive Graph Variational Auto Encoder?

texts:
I have been reading about how we can model a Variational AutoEncoder (VAE) into a Graph Variational AutoEncoder (GVAE) where the decoder reconstructs the adjacency matrix.
I presume that this approach makes the GVAE completely transductive because an unseen node will not be present in the adjacency matrix during training and we cannot dynamically change the dimensions of the matrix either. If we had to make it inductive then what approach can be taken? Is there already any work done in this regards? I am unable to find any appropriate literature so will really appreciate some help here.


texts: Importance of variable to implement in Logistic Regression model in Python sklearn?

texts:
Logistic regression using Python sklearn

In Logistic Regression model we can implement variables which meet 3 conditions:

independent variables have to be correlated with dependent variable
independent variables have NOT to be correlated with each other
only relevant variables can be added to the model, otherwise it does not make sense

and both 1. and 2. point we can achieve using Pearson correlation for example:

-- correlation between independent variables and dependent variable (targetVariable

1. CORREL = data.corr().sort_values('TargetVariable')

--heatmap which shows lvl of Pearson correlation between independent variables (the closer 0 the lower the correlation)

2. sns.heatmap (data.corr(), cmap="coolwarm", annot=true)

But when I choose appropriate variables after 1. and 2. step how can I meet the 3. step and choose which variables (which are correlated with independent variable and not correlated with other independent variable) are important and I should use it in model and which are not important and I should out them? Maybe p-value and delete all with p-value higher than 0,05? But how?

Please give me the example code to meet 3. step.


texts: Autoencoder for anomaly detection from feature vectors

texts:
I am trying to use an autoencoder (as described here https://blog.keras.io/building-autoencoders-in-keras.html#) for anomaly detection. I am using a ~1700 feature vector (rather than images, which were used in the example) with each vector describing a different protein interaction. I have a "normal" category of interactions on which I train the AE, then I feed it new vectors and use reconstruction error to detect anomalous interactions.

Adjusting my threshold so I get a true positive rate of 0.95, I get a false positive rate of 0.15, which is rather high. When I trained xgboost on the normal and anomalous vectors (using both types of interactions in training and testing) I was able to get precision of 0.98 **.

Does that mean that my model (or indeed my approach of using an AE) is ineffective, or maybe this is the best I could hope for when training an anomaly detector rather than a 2 category classifier (that is, xgboost in my case)?
How should I proceed?

** Of course, this is merely a sanity check, and cannot be used as the solution.  I need the model to detect anomalies that can be very different from those I currently have - thus I need to train it on the normal interaction set, and leave anomalies for testing alone.


texts: What pre processing should I use on data to feed into a CNN?

texts:
I have a dataset of shape 105 x 501 x 266 where 105 is the number of data and 501 x 266 is the shape of 1 data i.e. The labels_dataset is of shape 105 x 1.

Each value of the 501 x 266 matrix is a complex number.

So it essentially becomes 501 * 266 * 2(real and imaginary part of the number)

And now I have to feed this data to a CNN.
I 'm new to training networks. So need to know whether my data is in best possible form for the CNN or not.

I've printed out max, min, sd, mean of real part, imaginary part and magnitude of the dataset for more info:

max real = 0.186396, min real = -0.204375
max imag = 0.166608, min imag = -0.159017
max abs = 0.219019, min abs = 2.33527e-10
mean real = 4.01718e-10, complex = 6.79294e-15, abs = 8.82916e-05
std dev real = 0.000442753, complex = 0.000400677, abs = 0.000590573

Is this a good form of data for input to a CNN. What are the options to make it more suitable?


texts: F - measure derivation (harmonic mean of precision and recall)

texts:
We can define the F - measure as follows:

  $F_{\alpha}=\frac{1}{\alpha \frac{1}{P}+(1-\alpha)\frac{1}{R}} $

Now we might be interested in choosen a good $\alpha$. In the article  The truth of the F-measure the author states that one can choose the conditions:

  $\beta=R/P$, where $\frac{\partial F_{\alpha}}{\partial P}=\frac{\partial F_{\alpha}}{\partial R}$

and then we obtain $\alpha=1/(\beta^2+1)$ and

  $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R} $

It is said that The motivation behind this condition is that at the point where the gradients of E w.r.t. P and R are equal, the ratio of R against P should be
a desired ratio $\beta$. I understand that the condition will garantify that the user is willing to trade an increment in precision for an equal loss in recall. But I do not get why the equality of both partial derivatives correspond to these hypothesis. I would rather understand when one partial derivative equals the other partial derivative multiplied by minus one. Could anyone explain me why the desired condition (condition in words) correspond to this equality (condition in math terms)?

EDIT:

Well, we could do the following:

  $\partial F=\frac{\partial F_{\alpha}}{\partial P}\partial P+\frac{\partial F_{\alpha}}{\partial R}\partial R$.

And since we want for $\partial P=-\partial R$ that $\partial F=0$, we obtain easily the condition.

But I have one problem with this: Since $\frac{\partial F_{\alpha}}{\partial P}/\frac{\partial F_{\alpha}}{\partial R}=1$, and the fact that the gradient is perpendicular to each level curve (https://ocw.mit.edu/courses/mathematics/18-02sc-multivariable-calculus-fall-2010/2.-partial-derivatives/part-b-chain-rule-gradient-and-directional-derivatives/session-36-proof/MIT18_02SC_pb_32_comb.pdf) we would have that the level curve must have $m=-1$. Nonetheless, when I calculate the level curve for some constant $c$ I get the result,

$R(P)=\frac{c(1-\alpha ) P}{P-c\alpha}$,

which clearly is not a linear funcion with $m=-1$. What am I missing?


texts:
When using/building a system for Entity Linking, is there a well-known measure for "ambiguity degree" of an entity?

Some approach to compare named entities regarding how difficult to disambiguate?


texts: List of samples that each tree in a random forest is trained on in Scikit-Learn

texts:
In Scikit-learn's random forest, you can set bootstrap=True and each tree would select a subset of samples to train on. Is there a way to see which samples are used in each tree?
I went through the documentation about the tree estimators and all the attributes of the trees that are made available by Scikit-learn, but none of them seems to provide what I'm looking for.


texts: HR Machine Learning: Treating/ Standardizing Part Time Employee Sums To Their Full Time Equivalents in Attrition Modeling

texts:
My data set consists of a subset of employees. Each employee has general HR information (typical standard hours, department, site, etc) along with punch card data which gives a clear picture of the types of hours the associate has worked over the past quarter (Night Shift, Time off, etc.)

The data set described above is being fed into tree-based models to make binary predictions related to employee attrition.

My question involves how to scale punch-card data, so models don't overlook part-time employees. In the past, we have loaded total hours worked by employees assuming that after a certain hours threshold is crossed, an action takes place(example, if an employee works over 40 OT hours in the last quarter, they are more likely to x.)  We have come to realize that because our data set consists of full and part-time employees, this logic may be flawed. Associates who work 40 hours a week will always have more hours worked than associates who work 24 hours in a week; thus, they are more likely to reach these thresholds.

Our data set isn't big enough to split into models for full and part-time employees.

Do we need to make changes to the way we are loading hours data into our models? Should we be scaling part-time employees hours worked to their full-time equivalent? Should we use a ratio of hours worked to total hours to change what we measure from hours sums to percentage of hours worked? Loading in hours as rates would change the logic behind each threshold to something like if 25% of total hours are overtime hours, an employee is more likely to x?

If possible, please, provide links to any relevant research papers where a similar problem is handled.


texts: Who invented the concept of over-fitting?

texts:
I list the references that I found so far.
Shortly, the first appearance of the term was in 1670, first appearance in in close meaning was in 1827, first appearance in a biological paper was in 1923 and first appearance in statistics was in 1935.
However, the references indicate that there are gaps in this chronology.
Earliest reference I found was The flying pen-man; or, The art of short-writing by William Hopkins (teacher of stenography.) in 1670.
However, it is a table of words with "overfit" as one of them and the context or meaning is not clear.
I check the other references before 1800 and those were transcription errors, "over, fit" or "overfet".
The earliest appearance in context I found was in "The Enigmatical Entertainer and Mathematical Associate ..., Issue 1" in 1827 saying:
"I , though for poetry not overfit , 7 Will boldly dare to lay down rules for wit ;" hence the meaning of overfit here is "fit too well" but in a positive meaning.
It seems that Darwin's "On the Origin of Species" in 1859 made the concept of fitting popular.
Bray, Charles I. "Fitting livestock for show." Bulletin (Colorado Agricultural College. Extension Service); 171A (1923). : "They believe
that these are almost equally valuable, and that there is less
danger of over-fitting the pigs than with whole milk."
The 1934 paper Twenty-Two Years of the Eastern Percheron Futurity mentions the penalty of over-fitting yet regarding stallions.
The Quarterly Review of Biology Sep 1935 Volume 10, Number 3pp. 341 – 377, saying : "Perhaps we are old fashioned but to us a six-variate analysis based on thirteen observations seems rather like overfitting".
So it was well known and maybe a bit old-fashioned during 1935 already.
First statistics reference to over fitting I found appears in "Tests of Fit in Time Series", 1952.
Yet it says: " Such a graduation implies a gross
overfitting, but this can be allowed for."
Using overfitting as a well known concept.
The wikipedia article on overfitting references to Oxford dictionary entry that claims: "Origin 1930s; earliest use found in Quarterly Review of Biology. From over- + fitting."
It is possible that the reference is The Quarterly Review of Biology Sep 1935 Volume 10, Number 3pp. 341 – 377, yet I couldn't verify it.
I couldn't find the paper in Google Scholar.
Looking in Google ngram viewer it seems that the usage started around 1885.
Looking at Google book for that period shows references about sheep.
I'll be happy to learn that sheep are the origin of a central concept in AI but I'd like to see references.
Reference not related to sheep are welcome too.


texts: Can the learning rate be considered both a parameter AND a hyper-parameter?

texts:
Here is my understanding of those 2 terms:

Hyper-parameter: A variable that is set by a human before the training process starts. Examples are the number of hidden-layers in a Neural Network, the number of neurons in each layer, etc. Some models don't have any hyper-parameters, like the linear model.

Parameter: A variable in which the training process will update. For instance, the weights of a Neural Network are parameters as they are updated as we train the network and there is no human intervention on the process. Another example would be the slope and the y-intercept in a simple linear model.

Having said that, what would the learning rate parameter ($\eta$) be?

$$
\Theta_{i+1} = \Theta_{i} + \eta \nabla J(\Theta_{i} )
$$

My understanding is that $\eta$ is set before the training starts to a large value but then, as the training progresses and the function gets closer and closer to a local minimum, the learning rate is decreased. In that case, doesn't the learning parameter satisfy both the definitions of a parameter and of a hyper-parameter?


texts: Python Neural Network library/program with simple installation? (Solved?)

texts:
Edit: I found a solution to my problem; there's a scipy installer that doesn't require openBLAS, and that seems to have worked.

I'm trying to find a neural network library for Python, but nothing I've tried has been what I've wanted; the libraries a lot of people seem to be recommending (e.g Keras, scikit-learn) require scipy, which won't actually install on my computer.  (I think it requires openBLAS, which doesn't seem to have a tutorial for installation.)  I did find a program that looks like it will work, but it has a maximum of one hidden layer.  Can someone point me to a library that actually works, or show me how to install scipy?


texts: Are stacked NN the second generation of NN?

texts: Spiking Neural Networks are said to be the NN's third generation. Feed-Forward NN are the first. What is the second Generation? Stacked NN?


texts: Entropy calculation for MNIST dataset to form classification decision tree

texts: I am trying to get classification tree in R. I am using MNIST (handwritten digit dataset) to train my decision tree. The dataset has 28*28 pixel box. Pixels are organized in row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black). I have to calculate entropy to get best attributes among 784 attributes (pixel0 to pixel783). Can anyone suggest, how to calculate entropy or information gain for these dense attributes?


texts: Clustering customer dataset to find customer patterns

texts:
Say I have a dataset, which consists of n customers. A customer is represented by a name, an order, an orderAmount and a timestamp (time of day) for when the order was placed. It could look like this:

name     |order|orderAmount|timestamp
customer1|tv   |1          |08:30
customer2|hifi |1          |12:00
customer3|hifi |3          |12:30
customer4|tv   |2          |09:30
customer5|cd   |10         |10:00
customer1|tv   |2          |11:30
...      |...  |...        |...

What I'm interested in is clustering these customers, so it would be possible for me to clearly differentiate between them. By looking at the above set, it's clear that there's a difference in what, how much and when they buy, but I would like to automate this process, and I assume clustering is one way to do it, but please correct me if I'm wrong.

One thing in particular I'm not sure of is how I would represent this dataset in a "customer-matrix". I know, for example, a distance-based clustering method such as KMeans requires some sort of normalized input, but how would I get started on that with a dataset such as this?

When/if I can produce good clusters, I imagine I could assign labels to these clusters and use these labels to train a classifier?

I'm a beginner to data science, so there might be a step or two missing in my process so please bear with me.


texts:
I have a .csv file with data in the following form:

moment_1;moment_2;moment_3;force_x;force_y;force_z;...
-0,02131267;-1,6032766088;5,9906811787;5,40010285;0,0203;86,44227467;...
2599;-1,70091039344;-1,3044809;-0,0406673590;-2,60896180797;43,2334;...

The file is very large and I need to put it in an interactive visualization, that's why I need to reduce the data points without changing the overall structure too much.

Many data points are very close to each other as seen in the following image:

My approach was to define a threshold and filter all points which have a distance to the previous point lower than the threshold. But I think that's not an optimal solution because, when I remove one index, I need to remove it from the other data array too, otherwise the structure is changed.

Are there better approaches?


texts: Sentiment analysis of tweets (Train model on a labelled dataset and use on some other unlabelled data)

texts:
I have a huge amount of tweets on a particular topic say 'ABC' and the data is not labelled. I want to perform multi-class sentiment analysis of these tweets. I tried many unsupervised clustering techniques like Kmeans, DBScan, Agglomerative clustering from sklearn but the max silhoutte score that I have reached is 0.31 and the kmeans gives large negative score. I have performed cleaning and encoding of tweets using Bert embeddings, Word2Vec but nothing seems to change.
Suppose I used some other labelled multiclass dataset and build a classifier and then use that classifier to identify sentiment in my target data, will it be good enough? Is this approach correct and logical?
I have found these general speech datasets. Will they suffice my purpose of getting correct sentiments for the "ABC" tweets dataset?
I found this another emotion dataset related to tweets.


texts: How to obtain with a recurrent neural network the Xor function using keras?

texts: I'm trying to implement a model  of recurrent neural network to solve the XOR problem, but I am not still able to do that. Any hints?


texts: Why we need to have the test set remains consistent across multiple runs?

texts:
In the book "hands-on machine learning with scikit-learn and tensorflow: concepts, tools, and techniques to build intelligent systems" , more specifically in chapter 2 , the writer is teaching us how to create a test set. He mention that we need to keep the test set consistent across multiple runs . To do so he mentioned the following :

A common solution is to use each instance’s identifier to decide
whether or not it should go in the test set (assuming instances have a
unique and immutable identifier). For example, you could compute a
hash of each instance’s identifier and put that instance in the test
set if the hash is lower or equal to 20% of the maximum hash value.
This ensures that the test set will remain consistent across multiple
runs, even if you refresh the dataset. The new test set will contain
20% of the new instances, but it will not contain any instance that
was previously in the training set.
My question is why it is important to do so ? why we don't like to generate different test set for each run ?


texts: How to reduce time R takes for model building

texts:
I am building machine learning algorithms in my laptop. It has i3 procesor and 16 GB RAM. Despite using multiple cores(3 out of 4), it takes 2 days to run all the techniques that i am trying to run an obviously data is huge (close to 1.3 million rows and 20 variables).

Is there any way to reduce this time required for running algorithms into fraction of the time it takes currently? Lets say some hours instead of days? I have heard from a friend who has computer science background that spark takes less time than stand alone R. Not sure if Spark can reduce the analysis time from multiple days to some hours. I am open to suggestions and solutions(preferably open source). Thoughts?

I am sure a solution for this must exist as R is pretty old and some genius would have found a way to solve this painful problem.


texts: Time series forecast: when should I use the additive or the multiplicative method?

texts:
In order to proceed with a time series forecasting, the data set has to be stationary. Stationarity can be determined using a number of packages, the most famous (as far as I could understand) is statsmodel.

What I was not able to pick to date is when I have to use one method vs the other (additive or multiplicative).

Any simple explanation?


texts:
Can anyone explain the meaning of this line:
"Deep networks have been shown to learn representations
in which interpolations between embedding pairs tend to
be near the data manifold".
Reference: Section 4.3 of the paper Generative Adversarial Text to Image Synthesis


texts: Machine Learning/Text Analysis/Algorithm Pointers

texts:
I'm looking for a reference or point in the right direction since I'm not too familiar with machine learning or algorithms. I'd prefer to work in R, but I could also do Python. Any reference to a specific package I could use would be great.

So I have a list of Senate bills and their descriptions that are one or two sentences long. I have a subset of bills that also have accompanied bill types (ie "budget", "immigration", etc). I want to create an algorithm that will assign a bill type(s) to the other subset of bills that don't have bill types.

I would want to be able to do this by using the underlying relationship between descriptions and bill types for the first subset and apply it to the second so that I can predict bill type from bill description. Does this make sense? Any advice would be welcome!


texts: Inverse Transform error on OrdinalEncoder

texts:
I am not sure if the data type is correct to use inverse_transform. Getting an IndexError "index 2913 is out of bounds for axis 0 with size 5"
Reading from a csv:
df = pd.read_csv("pris.csv", usecols=['judge', 'plea_orcs', 'prior_cases', 'race', 'pris_yrs'])
Transforming data:
oe = OrdinalEncoder()
df[['plea_orcs']] = oe.fit_transform(df[['plea_orcs']])
df[['judge']] = oe.fit_transform(df[['judge']])
df[['race']] = oe.fit_transform(df[['race']])

X and y for sklearn:
X = df[['plea_orcs', 'judge', 'race', 'prior_cases', 'pris_yrs']]
y = df[['to_prison']]

Raises IndexError:
print(oe.inverse_transform(X[['plea_orcs']]).reshape(-1,1))


texts: deep learning for non-image non-NLP tasks?

texts:
So far there are many interesting applications for deep learning in computer vision or natural language processing.

How is it in other more traditional fields?
For example, I have traditional socio-demographic variables plus maybe a lot of lab measurements and want to predict a certain disease. Would this be a deep learning application if I have lots of observations?
How would I construct a network here, I think all the fancy layers (convolutional etc.) are not really necessary?! Just make it deep?

On my specific data set, I tried some common machine learning algorithms like random forests, gbm etc with mixed results regarding accuracy.
I have limited deep learning experience with image recognition.


texts: i have to classify an sms into categories like educational, bank related,etc. Is this a problem of topic modelling or text classification?

texts:
If this is a problem of text classification, is a similar dataset available or i have to make one on my own??
I have a dataset of emails for classification into spam/ham.


texts: how can I convert tensor shape to numpy array?

texts:
This is what my data looks like:

I want it to be numpy array, instead of a tensor, so that i can convert it to a dataframe. Is there a way to do so?
thanks in advance!


texts:
i have a dataframe with 34154695 obs. in a dataset a Class variable with value 0 indicate "not purchased" and 1 indicate "purchase".

> str(data)
'data.frame':   34154695 obs. of  5 variables:
 $ SessionID: int  1 1 1 2 2 2 2 2 2 3 ...
 $ Timestamp: Factor w/ 34069144 levels "2014-04-01T03:00:00.124Z",..: 1452469 1452684 1453402 1501801 1501943 1502207 1502429 1502569 1502932 295601 ...
 $ ItemID   : int  214536500 214536506 214577561 214662742 214662742 214825110 214757390 214757407 214551617 214716935 ...
 $ Category : Factor w/ 339 levels "0","1","10","11",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ Class    : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...

I am facing difficulties finding a solution to plot a histogram of the number of purchase per week, per day and time wise purchase based on class value = 1 and wanna output like this.

Could someone please inform how I should proceed??
Really, thank you for any help and suggestings.

Kind regards


texts: Is it right to maintain the train distribution in test set for unbalanced data?

texts: If the training set was unbalanced the chances are the model will be biased. But if the data distribution in the test set is the same distribution as the train set, this kind of bias is not going to affect validation accuracy. But my question is if this is the right thing to do? It's not cheating? What if we want to use the model for commercial business that we have no idea how the distribution of data would be? In this case, what is the right thing to do?


texts:
I apologize that this has been asked and I feel that it may be obvious, but I am wondering exactly what the meaning of the numerical value below from clusterdump:

Top Terms:
   monkey       =>  0.8170868432876803

I believe that to the be center of the centroid.  But if the term vectors were created with term frequencies, could one interpret this as the average occurrence of the "monkey" in the documents that are considered part of the cluster? In this case, "monkey" would appear in 82% of the docs in that cluster or more likely that the average count of monkey is .82?

Looking further, I see words like so:

Top Terms:
     zebra    =>   3.432595573440644

So it is best to interpret this as the average count of "zebra" in the set of docs...

And given the radius values, one could consider that the range of percentages of "monkey"?

mahout seq2sparse -i out/sequenced \
    -o out/sparse-kmeans -wt TF --maxDFPercent 100 --namedVector

...

mahout kmeans \
    -i out/sparse-kmeans/tf-vectors/ \
    -c out/kmeans-clusters \
    -o out/kmeans \
    -dm org.apache.mahout.common.distance.CosineDistanceMeasure \
    -x 10 -k $i -ow --clustering

When one uses tf-idf weighting, it may be best to normalize the output weights by creating a proportion of evidence via Wi=Wi/sum(W)
Is that a good idea? (Some Python LDA libs do this.)

Thank you.

References

https://mahout.apache.org/users/clustering/cluster-dumper.html


texts: How to quantify the performance of the classifier (multi-class SVM) using the test data?

texts:
I am working on a traffic sign recognition code in MATLAB using Belgian Traffic Sign Dataset. The dataset consists of training data and test data.

I resized the given images and extracted HOG features using the VL_HOG function from VL_feat library and then trained a multi class SVM using all of the signs inside the training dataset.

I am particularly interested detecting 8 signs. Question 1: Am I suppose to train the model using only the training set of these 8 signs or train the model with the entire training dataset and ignore the signs that I am not interested in at the detection stage?

Upon training the multi-class SVM, I want to test the classifier performance using the test data. Question 2: Can someone guide me through how to do this? I found something similar to what I am looking for available here. It would be great if I can obtain quantitative data such as a confusion matrix in percentage form.

Thanks!


texts:
I wanted to compute the Area under the ROC curve for a logistic regression model in the context of binary classification. For that I computed, for a list of thresholds, say 0.1 0.2 ... 0.9, the TPR and FPR. I have thus
 (TPR_threshold1, ... TPR_thresholdN)
 (FPR_threshold1, ... FPR_thresholdN)

Can I just do
TPR_threshold1*FPR_threshold1 + ... + TPR_thresholdN*FPR_thresholdN

in order to compute the area under the roc curve or do I need some more elaborate mathematical modeling?
Thanks a lot!


texts: How to check if audio samples have only noise or are silent?

texts:
I have a wav file I want to split into frames in order to feed it into a machine learning model. The problem is that the audio has silence with some noise at some points.

My problem is that I do not want to include frames with no sound (or only noise) in my dataset.

One solution I believe is to use a model for speech recognition or something similar to do classification and see if a frame includes only silence or noise.

However, I am searching for a solution that will not rely on machine learning but mostly on signal processing techniques or some other pre-processing method.

So, how could I exclude these frames with only silence or noise?


texts: Deep Q-learning, how to set q-value of non-selected actions?

texts:
I am learning Deep Q-learning by applying it to a real world problem. I have been through some tutorials and papers available online but I counldn't figure out the solution for the following problem statement.
Let's say we have $N$ possible actions in each state to select from. When in state $s$ we make a move by selecting an action $a_i, i=1\dots N$, as the result we get a reward $r$ and end up in a new state $s^\prime$. In order to update the Neural Network with this experience $(s, a, r, s^\prime)$, the only ground truth q-value that we have is for action $a_i$. In other words we do not have any ground truth q-values for all other possible actions ($a_j, j=1\dots N, j\neq i$). Then how should we feed this training data sample to the Neural Network?
Following are the possibilities I was thinking about,

Set other q-values to don't cares. In this scenario we do not update the weights of the last hidden layer that connects to the output values of all $a_j$'s. However, due to interconnections between the earlier layers, any update to the weights will affect the q-values of $a_j$'s.

Set other q-values to the current predicted values by the Neural Network. This makes the error for those q-values zero, but like above solution the change in the weights will affect the $a_j$'s q-values eventually.

Use one Neural Network for each possible actions. This perfectly makes sense as the solution to me, but Deep Q-learning uses one network to predict the q-values of all possible actions (regardless of having two networks, one for policy and the second as target).

If someone who has experience and knowledge to help me with this to understand, I would highly appreciate it.


texts:
I have a dataset consist of 8 columns and 15600 rows with the following columns:-

1.Entry_academic_year which have 5 discrete value (2558,2559,2560,2561,2562)
2.Faculty (It is the faculty that student has taken like engineering)
3.branch  (It is the branch  that student has taken like software engineering)
4.Admission type (how the student enter the college)
5.Graduated_high_school (it is the high school where student got graduated)
6.province_of_school
7.GPA_high_school(It is the GPA of student in high school)
8.GPA_college(It is the GPA of the student during college)

I am trying to predict the GPA of the student at the college by dividing the GPA into 4 quartiles with respect to percentile (25,50,75), The problem I faced is that the Graduated_high_school columns have around 1732 unique value with some school contain only one row which makes the prediction around 30-35 % accuracy
Any idea on how to fix it?


texts: Interpretation of 'recall' as a conditional probability P( X=>+ | X=+ )

texts:
In the context of performance measures for classification, I have a question about recall and precision.

Looking at the definition of recall:-

$recall = \frac{T_p}{T_p+F_n}$

When I look at this, it sounds like a $conditional$ probability to me -- probability that a test instance will be classified as positive, given that it is indeed positive:-

$ recall = Pr(X\hspace{1mm}is\hspace{1mm}predicted\hspace{1mm}as\hspace{1mm}positive | X=positive ) = \frac{T_p}{T_p+F_n}$

Here I am taking liberty to think (as it goes from the definition of $F_n$ ) that $F_n$ in the denominator is actually count of the test instances which are positive but got mis-classified as negative.

In the same light, if I think of precision now, this is how I am thinking of it as a probabilitty that a new test instance actually being positive given that it is predicted as a positive:-

$ precision = \frac{T_p}{T_p+F_p} = Pr(X=positive | X\hspace{1mm}is\hspace{1mm}predicted\hspace{1mm}as\hspace{1mm}positive)$

Is this interpretation of precision and recall correct?


texts: How can I group texts with similar content together?

texts:
I need to find a solution to group a corpus of texts according to document similarity. Premising I have no experience in ML - only a few readings - I'd like to know if calculating the tf-idf on each text is the right approach. I've read something about calculating and compare the cosine similarity on tf-idf, but I don't know how to further process the results.

Saying "almost equal" means "the order of the words matters", it' not just the same words in different order.

I was thinking about using MLlib from Apache Spark to do the job (or at least use a Scala library).

Can anybody head me to the right direction, or even better add a link to a tutorial to this page?


texts: Connecting to IEX with Pandas Datareader

texts:
My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some historical stock data.
After searching around and trying several methods I came up with this code here:

from datetime import datetime
import pandas as pd
pd.core.common.is_list_like = pd.api.types.is_list_like
import pandas_datareader as pdr
import os

#How to get historical Stock data

symbols = "FB" #Stock Kürzel
start = datetime(2018, 9, 1)
end = datetime(2018, 10, 2)

DataframeStock = pdr.DataReader(symbols, data_source = "iex", start = start, end = end, api_key = "pk_1234")

print(DataframeStock)

As far as I can tell everything looks fine and like in the tutorials (for example here: https://pandas-datareader.readthedocs.io/en/latest/remote_data.html#remote-data-iex) (I registered on https://iexcloud.io/docs/api/ and Iam using my own public key).
Unfortunately I get this error message:

  SSLError: HTTPSConnectionPool(host='cloud.iexapis.com', port=443): Max
  retries exceeded with url:
  /stable/stock/market/batch?symbols=FB&types=chart&range=2y&token=pk_8dfc5c0ae37940e78a2a0037bed89d5b
  (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines',
  'tls_process_server_certificate', 'certificate verify failed')])")))

It sound like I tried to often, but there shouldnt be to much of a restriction and overall I tried it less then 50 times and today stopped at 2 tries.

Thanks alot in advance.


texts: Weights and bias' relative to preprocessed X

texts:
I am currently using sklearn scale to preprocess my X data before being put into a perceptron - mean/stddev so as to prevent the data converging to infinity or 0. My perceptron returns the weights + bias after the network has been trained:

X = preprocessing.scale(X)

After processing the X and Y data through my perceptron I am returned with weights. From these weights I can calculate the line of best fit:

ls = cp.linspace(cp.min(X), cp.max(X))
best_fit = w[1]+w[0]*ls

where w[1] is a bias. This best_fit line is accurate but it is relative to the preprocessed X rather than the original X which I would like to plot. What is the technique to make these weights relative to the original X values if it is possible?


texts: Correctly evaluate model with oversampling and cross-validation

texts:
I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue.

As I see, the correct way of doing this is to sample ONLY the train set, in order to have a test performance that is more similar to reality. Moreover, I want to use CV for hyperparameters tuning. So, the tasks in order are

Divide dataset into train-test
Perform the 5 fold-CV, as...
Sample the "training" portion of the CV
Sample the "validating" portion of the CV
Train the model on the "training"
Validate it on the "validating"
Repeat 3-6 5 times
Evaluate performances on test

My doubt is: how can I compare the CV performances with the test, since the formare are based on sampled data and the latter does not?

An idea is to skip 4 and sample only "training" portion, but in this case how can I compare the "training" with the "validating"?

EDIT: added target ratio.


texts: Understanding Weighted learning in Ensemble Classifiers

texts:
I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was misclassified or not by the previous model in sequential boosting.
Although I intuitively understand that by weighting examples, we are letting the model pay more attention to examples that were previously misclassified, I do not understand "how" the weights are taken into account by a machine learning algorithm. Are the weights taken as a separate feature? I want to know "how" a simple machine learning model, lets say a decision tree, understands that a particular example carries more weight and therefore needs to be given more attention/ classified correctly.
Can someone please explain to me in simple words/equations how weighted training samples are handled by machine learning algorithms? How do they force themselves to classify samples with higher weights correctly? Should there be modifications made to existing ML algos to account for weights?


texts: Reinforcement Learning control with known dynamic equation

texts:
I know there is model-based reinforcement learning. But all the approaches assume an MDP.

If I want to do a feedback control of a system (i. e. control an inverted pendulum) it's quite easy to find the nonlinear differential equation. Can I somehow feed this knowledge into RL-algorithms or are there ways to transform a dynamic system to an MDP?


texts: Fixed target values but more ID for clafssification

texts:
I have a data set where there the target column as values between 1-15. Now I need to predict that based on the features. the feature values are have unqiue IDs. but the target column is the same value range.
one row could looks like

1,0.5,2.3,1.23,5 where first entry is the id and last entry is the target. In that why I have total 500 uniq IDs which has different feature values but same target range.

Now I would like to ask should I take the ID in the traning process ? or take features ? Or what would be the best approch for this classification ?

so one


texts: suppose I want to track the ratings of seasons of a TV show. so there are three columns of data-season number, year, and ratings. I want the year and ratings on the axes and season number as labels on the points. how can this be achieved considering visualization tools only take in values for X and Y axes?


texts: Is it possible to extract specific words from a sentence in Hindi/Marathi?

texts:
I have seen the different options to extract words from sentences in English but when I wanted to know if its possible to do the same thing in Hindi or Marathi
eg: टोमॅटो बेचना है
where the word at position 1 is TOMATO and it needs to be extracted i.e. extraction of product name from a sentence.
Any help will be appreciated.


texts: Using datasets to predict the results for other devices

texts:
I have a datasets that contain results from a series of physical tests. It has about a dozen features and the outcome of each test is distinguished by 3 different classes.

The dataset includes features like: [X, Y, Z, temperature, input voltage]
 and as far as I know, these are all independent variables.

I then have a result column for the test [failure, success, other]

I have multiple of these datasets from testing of different devices.

I am trying to figure out if I can use the data from one device to predict the results for any of the other devices. If I can, I would like some way to measure how closely they are correlated, and which variables most strongly contribute to the prediction.


texts: Write way to add samples to torch TabularDataset

texts:
I have a TabularDataset and i would like to add some examples to the dataset.

dataset = TabularDataset(path=path, format="csv", fields=[('label', l_f),('data', d_f)])
# I tried this
dataset.examples = new_examples # new_examples is a list of examples
len(dataset) == len(new_examples) # This is True

I can see that the length of the dataset has changed and in examples i have the examples from new_exampels, but i don't think this is the wright way to do it because i think that it messes up the other methods in TabularDataset.

What is the wright way to add samples or to create a new dataset from a list of examples?


texts: How to convert binary classifier to multiclass classifier?

texts:
I am a biggener student in Machine learning, and I want to ask if is it possible to convert a binary classifier label (y) by applying some condition on column1 to get a third situation.

I.e. Instead of having just 2 cases "Easy" and "Difficult" as output, I need to apply a condition on an additional feature in order to get as output "Easy", "Normal", "Difficult".

and I need also some keyword that can I use on Google as a request to look for solution.


texts: Feature transformation possible at selected features or only at all?

texts: I want to cluster. I have different features for that. Some features have a very small value range (from 0 to 0.8) and some have a very large value range (from 0 to 5 million). I want to use the logarithm. Do I have to apply this feature transformation to all features, including the small values (shift the values before by +1) or can I just choose the features I want to do the transformation on? Is there any literature or work on this?


texts: R, optimal return level from output chart, gpd.fitrange (ismev)

texts: For purpose of project I must find optimal return level using gpd.fitrange from ismev library. Can anyone tell me how to interprete the chart it outputs? What is the optimal return level here?


texts: Custom keras dataset generator not accepted by fit_generator

texts:
I've got a problem trying to use generators in Keras. I've got TensorFlow 2.1 with Python 3.7. I'm using the Keras version bundled with TensorFlow

I have defined a class for my generator derived from tf.keras.models.Sequential:

class RandomFramesFromPathsToVideos(tf.keras.models.Sequential):
    def __init__(self, x_set, y_set, number_of_videos_per_batch, ort_session, frames_per_video=25, type_of_frame='cropped_frame'):
        self.x, self.y = x_set, y_set
        self.batch_size = number_of_videos_per_batch * frames_per_video
        self.number_of_videos_per_batch = number_of_videos_per_batch
        self.frames_per_video = frames_per_video
        self.type_of_frame = type_of_frame
        self.ort_session = ort_session

    def __len__(self):
        return int(np.ceil(len(self.x) / float(self.number_of_videos_per_batch)))

    def __getitem__(self, idx):

        bla, bla, bla...

        return (np.array(devol_x, dtype=np.float32), np.array(devol_y, dtype=np.float32))

and it works since I'm perfectly able to train my model item by item using a crafted loop like this:

ds_train = RandomFramesFromPathsToVideos(x_set = set_of_paths, y_set = set_of_labels,
    number_of_videos_per_batch=4,
    ort_session = ort_session)

for i in range(0:len(ds_train)):
    (x,y)=ds_train.__getitem__(i)
    model.fit(x,y)

but when I try to use the generator as it should be used:

model.fit_generator(ds_train)

(which is now equivalent in TF2.1 to model.fit(ds_train), I know), I get the error:

ValueError: Failed to find data adapter that can handle input: <class 'functions.RandomFramesFromPathsToVideos'>, <class 'NoneType'>

I've tracked the error up to this point in internal code of TensorFlow:

def select_data_adapter(x, y):
  """Selects a data adapter than can handle a given x and y."""
  adapter_cls = [cls for cls in ALL_ADAPTER_CLS if cls.can_handle(x, y)]
  if not adapter_cls:
    # TODO(scottzhu): This should be a less implementation-specific error.
    raise ValueError(
        "Failed to find data adapter that can handle "
        "input: {}, {}".format(
            _type_name(x), _type_name(y)))
  elif len(adapter_cls) > 1:
    raise RuntimeError(
        "Data adapters should be mutually exclusive for "
        "handling inputs. Found multiple adapters {} to handle "
        "input: {}, {}".format(
            adapter_cls, _type_name(x), _type_name(y)))
  return adapter_cls[0]

The contents of ALL_ADAPTER_CLS is:

result = {list: 7} [<class 'tensorflow.python.keras.engine.data_adapter.ListsOfScalarsDataAdapter'>, <class 'tensorflow.python.keras.engine.data_adapter.TensorLikeDataAdapter'>, <class 'tensorflow.python.keras.engine.data_adapter.GenericArrayLikeDataAdapter'>, <class 'tensor
 0 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.ListsOfScalarsDataAdapter'>
 1 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.TensorLikeDataAdapter'>
 2 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.GenericArrayLikeDataAdapter'>
 3 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.DatasetAdapter'>
 4 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.GeneratorDataAdapter'>
 5 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.KerasSequenceAdapter'>
 6 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.CompositeTensorDataAdapter'>
 __len__ = {int} 7

I don't understand why my generator doesn't work. Has anyone any idea about what I'm doing wrong?.
 My dataset is a huge bunch of videos and is impossible to fit in memory by any means. Also, I'm doing non standard transformations like locating and cropping elements that appear in the frames so I can't use an standard generator that may be already implemented in Keras.

Thanks a lot


texts:
Using this TF tutorial, Classify structured data with feature columns, I was interested in this Embedding columns feature.
breed1_embedding = feature_column.embedding_column(breed1, dimension=8)

In this case, the Breed1 column which has 166 unique values can be collapsed into 8 dimensions. Here, the Breed1 column doesnt have a numeric value attached to it. However, say we have hypthothetical example like below with column Breed1_value added in. How can we still use this embedding columns technique ?


texts:
Data normalization: It ensures that each input (each pixel value, in this case) comes from a standard distribution.
This standardization makes our model train and reach a minimum error, faster!
my question is about standardization, I don't understand the relation between input data being between 0 and 1 to gradient decent being faster or even being correct(predicts well), why is that, I'm very ok with a deep math proof.


texts:
I have a data set in which the score column has to be between 0 to 100 and the subject column has to be one of ['Math','Science','English']. However, my data set has different values for some rows.

How should I handle those rows?

  subject score ...
1 Math    90    ...
2 Science 85    ...
3 English 105   ...
4 Comp    95    ...
5 Math    80    ...
6 Science 70    ...


texts: How to create a training set and classify them as positive or negative

texts:
I'm trying to create a Sentiment Analysis algorithm for a custom data (government dept specifc data) and not like any other social media data etc. The data exists but I need to categorise the data as positive or negative.
My requirement is to classify the test data as positive or negative using a sentiment analysis algorithm.
The big challenge for me is to take the similar set of government data which I can get. But I need to prepare the training data set. For this I need to classify the training data set as positive or negative.
On what basis do I need to categorise the data set as positive or negative. The reason I'm asking this question is that preparing a good training data set is very important to improve the accuracy of my sentiment analysis algorithm .
I have come up with my sentiment analysis algorithm but really need to prepare a robust set of training data(classified as positive or negative).

Experts - I'm a newbie to the machine learning area and I need advice from the researchers and experts out there.


texts: Can I train a CNN to detect the number of objects without localizing them first?

texts:
So I was trying to search but couldn't find any answers. I was wondering if it possible to train a model to detect the number of items of interest in a photo without having bounding boxes or dots to locate the objects in the training set.

For example say I wanted to count something simple like street poles in a photo, would it be possible with just the photos in the training set and the number of poles as the target only. So no bounding boxes or points labeled for the training data targets.


texts: Choice of method for classification and/or duration prediction

texts:
Let's consider an e-commerce problem.

I have data about users that almost place an order online : they give some information about themselves (name, age, etc.), but won't immediately validate the order.

Sometimes they'll come back to finalize it, but other times, they'll never be back (better offer/product at another place). I do have the data to know if a user came back or not, and if he came back, how long it took him to come back.

Thus, I am interested by 2 points :

Will the user come back ?
If yes, how long did it take him to do so ?

For the first problem, it is obviously a classification problem. And  I am considering a regression problem to answer the second question as it is a duration problem.

However, I was wondering if I should train these models separately, or if a specific method to take into account the link between these two was more appropriate ?


texts: Why is there a difference in performance across the feature descriptors for the same imaging modality?

texts: I've been using GIST, HOG and SURF descriptors for extracting features from different collections of Chest-X-rays and measuring performance using accuracy and area under the curve. These collections are obtained using different machinery, from different medical institutions, and, with different pixel resolutions. I could repeatedly see that one descriptor performs better than the other and the performance is not the same across these collections though all are frontal chest X-rays. What attributes to these differences in performance across the collections though they are from the same imaging modality?


texts: which application domains are not well suited to deep learning?

texts:
Deep learning seems to be the new cool thing in AI/machine learning and it works well in many domains, but I want to know- what are the specific application areas where deep learning is not the best approach and what is the reason for that?

Has some evaluation been made?

Is there a particular class of problems?

If so- what solutions are superior in solving that task?


texts:
does initialization play a role in finding a solution during training of neural networks in
SGD convergence


texts: What's the easiest way to quickly plot several features simultaneously for new data?

texts:
I'm starting a very beginner tutorial for DS/ML. One of the first things to do is to poke at the data they gave me. After doing the typical head() and describe() in pandas, I want to look more carefully at the features that are numerical, to see if there's any correlation between them and the labels (for example, a simple linear relationship).

I've used matplotlib and stuff before, but I want to do this the "correct" way. The tutorial I'm looking at suggests using seaborn, but to be honest it seems difficult to apply to what I'm suggesting (they're looking at more complicated relationships/etc).

I could use matplotlib, but I've never found a smooth, seamless way of making a grid of several plots simultaneously. Doing the "subplot" thing was really clunky. Is there a better way?

What I'd like to do is, let's say I have 4 numerical features, and I want to plot each of them (or their average, etc) vs the label, and display it, so I can quickly see patterns. What's the best way to do this?


texts:
I am trying to compare two images and detect the difference between them whether in shape or color.

-The change in shape: the number of parts has changed ( increased or decreased).
-The change in color: some of the parts were in pink and changed into white and vice versa.

I have tried 3 algorithms:

Compare by Compare_ssim.
Difference detection by PIL (ImageChops.difference).
Images subtraction.

The first algorithm:

(score, diff) = compare_ssim(img1, img2, full=True)
diff = (diff * 255).astype("uint8")

The second algorithm:

from PIL import Image ,ImageChops
img1=Image.open("canny1.jpg")
img2=Image.open("canny2.jpg")
diff=ImageChops.difference(img1,img2)
if diff.getbbox():
     diff.show()

The third algorithm:

 image3= cv2.subtract(image1,image2)

The problem is these algorithms are so sensitive. If the images have different noises and different angle of capture, they consider that the two images are totally different. Any ideas to fix that?


texts:
why is an optimizer used in something that is not a neural network?
in the neural style transfer architecture the image is updated instead of the neural network weights.
in stylegan encoder (find image in latent space), the latent space is updated instead of the weights of a neural network.
Why does it work in these cases, since it is not the weights of a neural network that is updated?


texts:
I've read a number of papers where the authors talk about "Unsupervised Hierarchical Agglomerative Clustering". They seem to imply that the algorithm determines the number of clusters based on a hyper-parameter:

We define the hetereogeneity metric within a cluster to be the average
of all-pair jaccard distances, and at each step merge two clusters if
the heterogeneity of the resultant cluster is below a specified
threshold

When I search for python implementations of Agglomerative Clustering I keep coming up with sklearn, which requires the number of clusters to be specified aprior. In most examples this is computed by plotting a dendogram and then determining by what appears to be eyeballing the chart how many clusters - for example https://towardsdatascience.com/machine-learning-algorithms-part-12-hierarchical-agglomerative-clustering-example-in-python-1e18e0075019 I'd argue it's impossible from the chart alone to determine if 3 or 5 is the optimal (based on largest vertical distance). I believe this is Wards method but I'm not sure it's the same as ""merging clusters where the heterogeneity is below a threshold" and
Is this possible in sklearn, or is there another python implementation which does this? I feel at the very least there should be a way to process the dendogram programmatically rather than plotting it?


texts: Interpretable models apart from Logistic Regression

texts:
I am wondering about other interpretable models apart from logistic regression.
I am looking for models that can interpret the effect on the target variable by unit change in any feature variable.
I was thinking tree-based models may help but I'm not sure.


texts: Is my Lasagne/Theano neural network running too slow?

texts:
So I'm a newcomer to the world of neural networks, and I've been getting a little familiar with the field and have started playing with my own networks.

I'm using Lasagne, and I'm finding that the training of my NN is taking an infeasible amount of time.  Maybe the problem I'm looking at is simply biting more than my computer can chew; I just don't know, because I'm not familiar enough with how these things go.

I'm trying to train a 1D convolutional NN, which has two 1D convolution layers, and two max-pooling layers, and two fully connected layers. All in all, the NN has around 95,000 parameters (weights + biases).  I'm running this on an Amazon EC2 GPU instance.  What I'm finding is that during training, it's taking 0.3 sec/sample.

By contrast, when I run the Lasagne MNIST CNN example, it's taking about 0.0003 sec/sample, 1000 times faster!  Now to be fair, my input samples are 18000x3 dimensional, and the MNIST samples are 28x28 dimensional; but the difference there is a factor of 68.  And the MNIST CNN actually has more parameters than mine, with 160,000 or so parameters.

Does this indicate anything wrong with what I'm doing?


texts: Multiply weights after using dropout in training - PyTorch

texts:
I have a Pytorch regression model as follows:

    model = nn.Sequential(nn.Linear(n_in, n_h_1),
                      nn.ReLU(),
                      nn.Dropout(p=0.1),
                      nn.Linear(n_h_1, n_h_2),
                      nn.ReLU(),
                      nn.Linear(n_h_2, n_out))

As you can see, I have used a Dropout regularization layer with dropout probability of 0.1.
When the training is completed, we should disable the dropout. This can be achieved using model.eval(). Reference to  this StackOverflow answer and other resources, we should multiply the output of hidden layer with  (1-p) during inferencing of model.

My question is: do we have to do this manually using pytorch or the library handles this itself  after applying model.eval()?
If we have to do it ourselves, how can we achieve thisfor the given model?


texts: Transfer learning with Keras for medical image classification

texts:
Good afternoon; I'm trying to do Transfer Learning from pre-trained model on imagenet to solve a classification task with Lung CT slices.
These slices are stored in dicom file, so their value are expressed on 12 bit in Hounsfield scale. My questions are: preprocess_input of keras / tf2 api need of a matrix of values in [0,255], so how can I preprocess my images to fed them to a pre-trained convnet? Can I to normalize range of image in [0,1] or it's mandatory to use preprocess_input of models? If preprocess_input is mandatory, how can I map the 12 bit value in range [0,255]?


texts: How to create Self learning data product

texts:
I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below.

Professional service: Statistician involvement to build regression model or any other kind of predictive model that fits specifically to client data and can be used.

Issue: So on the long run there will be issues around scalability as one analyst cannot build model simultaneously for hundreds of clients who want to come on board and use this service. Hiring 1 analyst for 100 clients is not an option for me as i do not run this operation.

Product: Plug and play with least human intervention. i.e. Some kind of self learning/artificial algorithm that works for any and every client. Additionally, it adjusts to new trends that appear over time which might not have been captured in the existing model. Personally, i will be biased towards this kind of solution.

Issue: This is the ideal solution, however i am not sure how to build this and how to approach in this direction.

To give context, a client in Europe might have same data structure in their sql database in terms of name, variable type in comparison to sql database of a client who is based in Singapore. However, Singapore data might have different trend, seasonality and pattern in comparison to data in Europe. Is there any way that a single Machine Learning or Artificial Intelligence or Deep Learning or Neural Net algorithm can be built and used for all regions and clients without having to update that model manually for any new client?

My background: i have previously worked in a professional service landscape and have statistics/predictive modelling background.


texts: Is an Arma model equivalent to a 1-layer Recurrent Neural Network without activation function?

texts:
Given a time series $f(t)$ to forecast, let us consider an Arma model of the form:
$$
f(t) = c + \sum_{i=1}^p a_i f(t-i) + e(t) + \sum_{j=1}^q b_j e(t-j)
$$

where $e(t)$ are the forecast errors.

On the train set, if $f(t)$ is the ground truth, then we define its estimate obtained with this model as $\widetilde{f}(t) = f(t) + e(t)$.

Let $m = min(p,q)$, we can rewrite the first equation as:
$$
\widetilde{f}(t) = c + \sum_{i=1}^m (a_i + b_i) f(t-i) + \sum_{i=m+1}^p a_i f(t-i) - \sum_{j=1}^q b_j \widetilde{f}(t-j)
$$
Then after reparametrization can be rewritten as:
$$
\widetilde{f}(t) = c + \sum_{i=1}^k c_i f(t-i) - \sum_{j=1}^q b_j \widetilde{f}(t-j)
$$
Which is the equation of a 1-layer recurrent neural network (RNN) without activation function.

So, are Arma models a subset of RNNs or is there a flaw in this reasoning ?


texts: How can I compare imputation techniques on a dataset with sci-kit learn?

texts:
I have a dataset data that has missing values. I am trying two ways of imputing these values, but I would like to compare them.
In the first method I am using a combination of techniques. Some are filled with the mean, others with conditional mapping, and a couple with models fit to other features. I then call this imp_data.
In the second method I am putting in the original data and using sklearn's IterativeImputer in a pipeline.
Both methods will be used to fit a RandomForestClassifier of the same structure. If I use the same random_state in the splits of imp_data and data, will this give me a good comparison of imputation techniques? e.g. will
train_test_split(imp_data[inputs], imp_data[target], random_state=1)

and
train_test_split(data[inputs], data[target], random_state=1)

produce directly comparable results?


texts: What are the criteria for updating bias values in back propagation?

texts:
During back propagation, the algorithm can modify the weight values or bias values to reduce the loss.

How does the algorithm decide whether it has to modify the weight values or bias values to reduce the loss?

Does it modify the weight values in one pass and bias values in another pass?

Thanks!


texts: Assigning manually a score based on some statistics?

texts:
I am trying to assign manually scores to identify bad accounts (not in Stacks community!) based on some specific conditions, for example if they have account names containing only numbers and a limited number of review, a past history poor, poor score, banned, and so on.
To do it, as I mentioned, I am doing the following:
-if number of reviews is < 2 then assign -1;
or
-if past history is poor then assign -1;
or
-if account score is less than 3 then assign -1;

and so on.
Since I have variables such as number of reviews, past history, account score, for all the users in my dataset, I am just wondering it I should assign a score in a different way, maybe using a more statistical approach based on average.
My goal is to determine an algorithm which can predict if an account is bad or not thought time based on the above conditions.
I would appreciate it if you could let me know what you think.
Thanks


texts: How do you, analytically, show you are not using too many features?

texts:
One of the managers at my company asked if there is a I could include a metric demonstrates that the my model is not using too many Features/Variables. Is there a metric or best practice that does this? Do you have a good way of communicating this to the business?
Specifically, she asked if I could include AIC about my models. After I asked some questions though, this is not quite what management is looking for because, unless I'm mistaken, AIC is only useful for comparing models with the same features.
In any case, I was able to use train-test validation for the model I was presenting, so AIC would have been overkill. To deal with this problem in practice I create a baseline model which uses all the variables, and then pair down as many variables as I can without sacrificing too much performance (on the train and test sets).


texts: Which method to use to remove correlation between independent variables comprising of both categorical and numerical variables?

texts:
The independent variables in the dataset contains categorical variables such as

Gender ( 2 levels)
Mode of Shipment ( 3 levels)
Product Importance ( 4 levels)

and Numerical Variables such as

Customer care calls
Discount Offered
Package weight

How do I find the correlations between these variables?

Converting categorical variables in to dummy variables and then using pearson correlation? What if the dummy variable categories also shows correlations too? such as correlation between Mode of shipment categories, Flight, ship, road? Do I need to remove the highly correlated dummy variable category with the other mode of shipment category?
or
doing separate correlations between numerical variables using pearson correlation, and for categorical variables using chi sq statistics?

How to go about it?
Thank you! It's a long question, but really need this clarity. Would appreciate any additional links too. Thanks again!


texts: Syntax error in function but everything seems correct

texts:
I am custom defining a function and following is my function but I get error as below
def  saveSlice(img, fname, path):
     img  = np.uint8(img * 255)
     fout = os.path.join(path, f'{fname}.png')
     cv2.imwrite(fout, img)
     print(f'[+] Slice saved: {fout}', end='\r')

I get error message as below
File "ipython-input-79-e8d731623c8b", line 3
fout = os.path.join(path, f'{fname}.png')
                                                                           ^

SyntaxError: invalid syntax
What am I doing wrong?


texts: Handling imbalanced data by deleting over represented rows vs. adding under represented rows

texts:
I am currently working with a very imbalanced data set
(frauded credit card data from kaggle, which has 492 rows of frauded cards and over 280,000 rows of non-frauded cards).

As much as I know, there are 2 ways to handle imbalanced data without adjusting penalty:

over-sampling the under-represented data - by creating a lot of copies of the frauded data
under-sampling the over-represented data - by not using a lot of the non-frauded data

Right now I go with under-sampling (since my computer is not that strong and I do not want to waste hours for answers).

Technically what I do, is create a 50-50 dataframe, which has all of the under-represented rows and randomly selected over-represented rows.

Problem is, every time I run the notebook, the algorithms use different random okay data and there are different results. When tweaking xgboost, there are different optimal parameters every time I run the notebook.

What should I do? Should I save as csv one of the balanced df I created and call it a day? Should I creat a lot of copies from the frauded cards? What is the best way to tackle the situation?


texts: Pandas how to fill missing values in one column if the values in another column are equal

texts:
I have a dataframe where I need to fill in the missing values in one column (paid_date) by using the values from rows with the same value in a different column (id). There is guaranteed to be no more than 1 non-null value in the paid_date column per id value and the non-null value will always come before the null values. For example:

index     id       paid_date
6         25220    2017-01-05 00:00:00
9         30847    None
11        30847    None
14        29369    2017-06-21 00:00:00
17        31232    2017-08-31 00:00:00
20        26196    2017-02-20 00:00:00
21        26196    None
24        28303    2017-05-09 00:00:00
25        28303    None

How can I replace the None values in the paid_date column if there is a row with a paid_date with a matching id?

index     id       paid_date
6         25220    2017-01-05 00:00:00
9         30847    None
11        30847    None
14        29369    2017-06-21 00:00:00
17        31232    2017-08-31 00:00:00
20        26196    2017-02-20 00:00:00
21        26196    2017-02-20 00:00:00
24        28303    2017-05-09 00:00:00
25        28303    2017-05-09 00:00:00

I tried using fillna with a dictionary that mapped ids to paid_dates and I tried using pd.Series.map but neither worked.

paid_dates = df[pd.notnull(df['paid_date'])]
pds = pd.Series(data=paid_dates['paid_date'].values, index=paid_dates['id'])
pds_dict = pds.to_dict()

# doesn't work
df['paid_date'].fillna(value=pds_dict)

# also doesn't work
df['paid_date'].map(pds_dict)


texts: Carlification of the MSE loss sum symbol

texts:
So I have a question regarding the MSE loss on the application of a Neural Network.
Loss function: $\text{MSE} = \frac{1}{2} \sum_{i=1}^{n} (Y_i - \hat{Y_i}) ^ 2$
I am wondering for what the $\sum_{i=1}^{n}$ stands.

Do I sum over the loss of all training examples for each output node in my Neural Network?
Or do I use a single training example and sum over all Neural network output nodes?
Or do I both and sum over all training examples and over all output nodes?

I want to use the MSE loss later than for updating my weights in the Neural Network. What would I do for that?


texts: Is summing a cosine similarity matrix a good way to determine overall similarity?

texts: I'm trying to similar research abstracts, so I'm using word embeddings to convert words into 1x768 vectors, so overall turning abstracts into embeddings with shape (#ofwords, 768). Cosine similarity between two abstracts returns a matrix (#ofwords1, #ofwords2), which I then sum up to get an overall score. What I'm wondering is if this summing up of all the values in a cosine similarity matrix is really a good way to determine overall similarity between two different texts? Is there a better, or less computationally expensive way to do this?


texts: Fitting a pandas dataframe to a Poisson Distribution

texts:
I have a simple dataframe df2 that consist of indices and one column of values. I want to fit this dataframe to a poisson distribution. Below is the code I am using:

import numpy as np
from scipy.optimize import curve_fit
data=df2.values
bins=df2.index
def poisson(k, lamb):
    return (lamb^k/ np.math.factorial(k)) * np.exp(-lamb)

params, cov =  curve_fit(poisson, np.array(bins.tolist()), data.flatten())

I get the following error:

  TypeError: only size-1 arrays can be converted to Python scalars


texts: Can the Entropy Given On a Decision Tree Be Explained As the Probability Of A Binary Dependent Variable

texts:
I'm not certain how to interpret the entropy output--though I have used the Gini criterion before and interpreted it as the probability of reaching 100% on any given leaf the tree splits on.Though after reviewing things on the internet--I think I've been interpreting results really wrong for the decision tree altogether.
If I have a regression model, I would interpret a coefficient as saying the coefficient of coffee increases the probability that a person will be ready for work in the morning by 42%, if 0.42 were the coefficient on my summary out put.
But if it's on my Decision Tree and Entropy = 0.42--is that still associated with the probability of a binary dependant variable, with a sample = 500 and value [200, 300]
All of this is made up because I can't give you my actual data


texts: Is it possible to use Generative Adversarial Networks (GANs) for text classification?

texts: I am working on the classification of fake and real news. I did use a CNN Model for this problem and got satisfactory results. But, I was just wondering if it's at all possible to use any type of GAN for this sort of a problem.


texts: Why does PCA assume Gaussian Distribution?

texts:
From Jon Shlens's A Tutorial on Principal Component Analysis - version 1, page 7, section 4.5, II:

  The formalism of sufficient statistics captures the notion that the
  mean and the variance entirely describe a probability distribution.
  The only zero-mean probability distribution that is fully described by
  the variance is the Gaussian distribution. In order for this
  assumption to hold, the probability distribution of $x_i$ must be
  Gaussian.

($x_i$ denotes a random variable - the value of the $i^\text{th}$ original feature.i.e. the quote seems to claim that for the assumption to hold, each of the original features must be normally distributed.)

Why the Gaussian assumption and why might PCA fail if the data are not Gaussian distributed?

Edit: to give more info, in the tutorial, page 12, the author gave an example of non-Gaussian distributed data that causes PCA to fail.


texts:
I'm not entirely sure why policy gradients have to be on-policy and have to update using trajectories sampled from the current behaviour. In REINFORCE, the loss function is determined by the log probability of an action times the reward (or discounted reward).

For a state $s$, if I take action $a$ and arrive in state $s'$ I will always see reward $r$. So if I have keep all these values, run an $s$ from the past through my current actor then why can't I update my actor with the new log probability of $a$ and the known reward $r$. I don't need to actually have played and seen a new result.

Can someone correct my understanding?

Thanks


texts: Interested in Mathematical Statistics... where to start from?

texts:
I have been working in the last years with statistics and have gone pretty deep in programming with R. I have however always felt that I wasn't completely grasping what I was doing, still understanding all passages and procedures conceptually.

I wanted to get a bit deeper into the math behind it all. I've been looking online for texts and tips, but all texts start with a very high level. Any suggestions on where to start?

To be more precise, I'm not looking for an exaustive list of statistical models and how they work, I kind of get those. I was looking for something like "Basics of statistical modelling"


texts: Self-designed objective for linear regression learning

texts:
A multiple linear regression is to use several predictor variables to predict the outcome of a response variable, like the following relationship:

$y_{i}=\beta_{1}x_{i1}+...+\beta_{p}x_{ip}+\epsilon_{i},   i=1,...,n$

I understand the typical objective to learn the $\beta$ paramters is least-squares, which means to minimize the sum of the sqaure of $\epsilon_{i}$. Now I want other kinds of objective, for example to maximize the Shannon entropy of the sequences of $\epsilon$ (or other self-specified objective). I googled towards this direction but no luck. I am wondering if there is any problem (and tool to solve it if possible) I can look into to do that?

Thank you for your help.


texts: Application of Control Theory in Data Science

texts:
I have recently completed an MSc in Control Systems from a top university. It seems to me that control theory must have an application within data science. I would like to apply my degree within this domain, but I want to be sure that it is relevant to the role of a data scientist.

The topics which I have particular interest and experience in are State Space Control, Systems Identification, Model Predictive Control and Optimal Control. I imagine that effective management of any large dataset must involve modelling of the system in terms of transfer functions/state space models (based on large sets of historical input/output data). These models could then be used to predict the evolution of a market/variable over time, and therefore optimise a given cost function such as profit, risk etc.

If this kind of role exists within data science/ other areas, can you please give me more information/ ideas of job roles/ industries to research.


texts: How to do phoneme segmentation using dynamic time warping?

texts:
Background Information:

Dynamic Time Warping (DTW):

In time series analysis, dynamic time warping (DTW) is one of the
algorithms for measuring similarity between two temporal sequences,
which may vary in speed. (Source: Wikipedia)

Phoneme Segmentation:

Phoneme segmentation is the ability to break words down into individual sounds. For example, a child may break the word “sand” into its component sounds – /sss/, /aaa/, /nnn/, and /d/. (Source)

The Question(s):

(a) How can we do phoneme segmentation using DTW?

(b) Which type of data do we need to implement this idea? (by asking about the type of the data, we mean which features should be available for each sample in the training dataset).

My try:
Assume that we have some audio files in each the phoneme segments are completely calculated and available. For instance, we know that from $t_1$ to $t_2$, the speaker is just saying /a/, and the other parts also have this kind of label. If we have a new sample which the system has not seen yet, a simple approach would be to calculate the difference between the new sample and each of the training samples. This approach would be like a KNN (K nearest neighbors) algorithm. We can just cast a vote to see which phoneme wins the game.
Another case is when the data is not labeled. In this case, I think we may be able to do some kind of clustering (e.g., K-means) to extract some cluster means, and use them. We could just calculate the distance between the new sample and the means of the clusters (which would be much faster than the previous calculations we had for the other case).
The problem is that these approaches seem too simple and inefficient to me. Is there a better (or smarter) way to tackle this problem of segmenting the phonemes using DTW? Should the samples have any other kind of features? (By other, I mean other than the time segments for each phoneme being specified).


texts: How to improve a CNN without changing the architecture?

texts:
I'm currently using an autoencoder CNN that's built upon the VGG-16 architecture that was designed by someone else. I want to replicate their results using their dataset first but I'm finding that:

-Validation losses diverge from training losses fairly early on (I get to around 10 epochs and it already looks like it's overfitting)
-At its best, the validation losses aren't even close to being as low as training losses
-In general, the accuracy is still worse than reported in their paper.

I'm new to machine learning and want to know if there are hyperparameters I should try to change or what I can do to maybe tinker with it without changing its architecture?


texts: sklearn random forest and fitting with continuous features

texts: Does anyone know how the python sklearn random forest implementation handles continuous variables in the fitting process? I'm curious to know if it does any sort of binning (and if so, how it does the binning), or if a continuous variable is just treated as a categorical variable? I'm hoping it's not the latter...thanks! Also, I'd be open to using some R implementation if anyone knows about that.


texts: Effect of adding gaussian noise to the input layer in a CNN

texts:
I often come across Keras code that adds GaussianNoise to the input, however its not clear to me what advantages does it offer to the learning.

input_img = layers.Input(t_x.shape[1:], name = 'RGB_Input')
pp_in_layer = layers.GaussianNoise(GAUSSIAN_NOISE)(input_img)


texts: MNIST - Vanilla Neural Network - Why Cost Function is Increasing?

texts:
I've been combing through this code for a week now trying to figure out why my cost function is increasing as in the following image. Reducing the learning rate does help but very little. Can anyone spot why the cost function isn't working as expected?

I realise a CNN would be preferable, but I still want to understand why this simple network is failing.
Please help:)

import numpy as np
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

import matplotlib.pyplot as plt

mnist = input_data.read_data_sets("MNIST_DATA/",one_hot=True)

def createPlaceholders():
    xph = tf.placeholder(tf.float32, (784, None))
    yph = tf.placeholder(tf.float32, (10, None))
    return xph, yph

def init_param(layers_dim):
    weights = {}
    L = len(layers_dim)

    for l in range(1,L):
        weights['W' + str(l)] = tf.get_variable('W' + str(l), shape=(layers_dim[l],layers_dim[l-1]), initializer= tf.contrib.layers.xavier_initializer())
        weights['b' + str(l)] = tf.get_variable('b' + str(l), shape=(layers_dim[l],1), initializer= tf.zeros_initializer())

    return weights

def forward_prop(X,L,weights):
    parameters = {}
    parameters['A0'] = tf.cast(X,tf.float32)

    for l in range(1,L-1):
        parameters['Z' + str(l)] = tf.add(tf.matmul(weights['W' + str(l)], parameters['A' + str(l-1)]), weights['b' + str(l)])
        parameters['A' + str(l)] = tf.nn.relu(parameters['Z' + str(l)])

    parameters['Z' + str(L-1)] = tf.add(tf.matmul(weights['W' + str(L-1)], parameters['A' + str(L-2)]), weights['b' + str(L-1)])
    return parameters['Z' + str(L-1)]

def compute_cost(ZL,Y):
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = tf.cast(Y,tf.float32), logits = ZL))
    return cost

def randomMiniBatches(X,Y,minibatch_size):
    m = X.shape[1]
    shuffle = np.random.permutation(m)
    temp_X = X[:,shuffle]
    temp_Y = Y[:,shuffle]

    num_complete_minibatches = int(np.floor(m/minibatch_size))

    mini_batches = []

    for batch in range(num_complete_minibatches):
        mini_batches.append((temp_X[:,batch*minibatch_size: (batch+1)*minibatch_size], temp_Y[:,batch*minibatch_size: (batch+1)*minibatch_size]))

    mini_batches.append((temp_X[:,num_complete_minibatches*minibatch_size:], temp_Y[:,num_complete_minibatches*minibatch_size:]))

    return mini_batches

def model(X, Y, layers_dim, learning_rate = 0.001, num_epochs = 20, minibatch_size = 64):
    tf.reset_default_graph()
    costs = []

    xph, yph = createPlaceholders()
    weights = init_param(layers_dim)
    ZL = forward_prop(xph, len(layers_dim), weights)
    cost = compute_cost(ZL,yph)
    optimiser = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    init = tf.global_variables_initializer()

    with tf.Session() as sess:
        sess.run(init)

        for epoch in range(num_epochs):
            minibatches = randomMiniBatches(X,Y,minibatch_size)
            epoch_cost = 0

            for b, mini in enumerate(minibatches,1):
                mini_x, mini_y = mini
                _,c = sess.run([optimiser,cost],feed_dict={xph:mini_x,yph:mini_y})
                epoch_cost += c
            print('epoch: ',epoch+1,'/ ',num_epochs)

            epoch_cost /= len(minibatches)
            costs.append(epoch_cost)

    plt.plot(costs)
    print(costs)

X_train = mnist.train.images.T
n_x = X_train.shape[0]
Y_train = mnist.train.labels.T
n_y = Y_train.shape[0]
layers_dim = [n_x,10,n_y]

model(X_train, Y_train, layers_dim)


texts: Fasttext error while loading wiki pre-trained data

texts:
I am loading the model using gensim package this way:

from gensim.models import FastText
model = FastText.load_fasttext_format('wiki-news-300d-1M-subword.bin')

as stated here.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 57: unexpected end of data

The .bin file is downloaded from this source.

How to load the model correctly?


texts: I have time series gravity variation data with inherent uncertainty in the data itself e.g. ($x+\sigma_x$) where $\sigma_x$ represent the error in data. Now I am doing time series modeling (ARIMA) for the same data. Whatever theory I have read, they consider that original data has no error but this is not true in my data. When I do modelling, definitely the error in data itself is going to affect the model parameter. I want to know how to incorporate inherent error $\sigma_x$ in my modeling and check its impact on model.


texts: Increase number of iterations in a logistic regression

texts:
I have achieved 68% accuracy using glm with family = 'binomial' while doing logistic regression in R. I don't have any idea on how to specify the number of iterations through my code. Any suggestions on it? Will application of stochastic gradient descent work for epoch. If so, how do I apply it in my code? I want to increase the accuracy of my model. Following is my code:

rm(list=ls())
library(dplyr)
data1 <- read.csv("~/hj.csv", head`enter preformatted text here`er=T)
train<- data1[1:116,]
VALUE<-as.numeric(rownames(train))
testset<- data1[1:116,]
mylogit <- glm(VALUE ~ POINT1 + POINT2 + POINT3 + POINT4 , data = data1, family ="binomial")
testset$predicted.value = predict(mylogit, newdata = testset, type="response")
for (i in 1: nrow(testset)){
  if(testset$predicted.value[i] <= 0.50)
    testset$outcome[i] <- 0
  else testset$outcome[i] <- 1
}
print(testset)
tab = table(testset$VALUE, testset$outcome) %>% as.matrix.data.frame()
accuracy = sum(diag(tab))/sum(tab)
print(accuracy)
print(tab)
table(testset$VALUE, testset$outcome)


texts:
I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see in the following figure:

However, I have read this amazing blog about binning data this whereby the author claims that adaptive binning is better than fixed-width binning. I understand the idea behind this because the some of the bins that we define in the fixed-width approach may have too little data distribution in comparison to other bins, which won't be a fair game to play whereby the adaptive approach, motivated by the idea of quantiles is better. Are there any more arguments, or more in depth analysis, of this hypothesis ?


texts: I have trained an ANN model for a regression problem which takes 10 parameters as input and gives 1 output. After training, I saved the model as json and weights as a .h5 file using keras. Now I want to make predictions on the new data. I have loaded the model and my question here is how do I scale this single row of input values, before feeding it to the model? Some of the machine learning algorithms did not require scaling(Standardization/Normalization) So I could load those models and used for making predictions. How do I do in an ANN model, since we scale the data for training?


texts: Why the results from Matlab curve fitting function differ when using custom equation?

texts:
I'm using Matlab's curve fitting app, When I select Power fitting it returns values which perfectly describe the data but when I use the custom equation and enter ax^b as the equation it returns very bad coefficients. Shouldn't the results be the same?

I need to fit a data to a $ax^{-6} + bx^{-4} + cx^{-2}$ function so I need to use custom equation.

EDIT: Added the screenshots.


texts: How to create ROC - AUC curves for multi class text classification problem in Python

texts:
I am working on a multiclass text classification problem and trying to plot ROC Curve but no success so far. Tried many solutions available but didn't work. Kindly please someone help me out with the following piece of code to plot the ROC curve. There are actually five different classes for which I am performing text classification.
categories = ['Philonthropists', 'Politcians', 'Showbiz', 'Sportsmen', 'Writers']
train = dt.load_files(r'C:\Users\...\Learning\Train', categories=categories, encoding='ISO-8859-1')
test = dt.load_files(r'C:\Users\...\Learning\Test', categories=categories, encoding='ISO-8859-1')
count_vector = CountVectorizer()
x_trian_tf = count_vector.fit_transform(train.data)
tfidf_transformer = TfidfTransformer()
x_train_tfidf = tfidf_transformer.fit_transform(x_trian_tf)
learn = MultinomialNB().fit(x_train_tfidf, train.target)
x_test_tf = count_vector.transform(test.data)
x_test_tfidf = tfidf_transformer.transform(x_test_tf)
prediction = learn.predict(x_test_tfidf)
print("Accuracy is of Multinomial Naive Bayes Classifier", accuracy_score(test.target, prediction) * 100)


texts: Why $L2$ loss is strictly convex if number of samples $N$ is larger than input dimension $d$?

texts:
I am using $L2$ loss in my linear regression problem and I have to prove that my $L2$ loss is strictly convex if number of samples $N$ is larger than input dimension $d$.

I think, if I can prove that my hessian matrix, $ X^T X $ for $L2$ loss is positive definite then that will do. Here, $X$ is the feature matrix with  $N$ samples as rows and  $d$ columns.

Also, I understand $v^T X^T X v$ $\ge 0$,

as,  $v^T X^T X v$ = $(Xv)^T. Xv$ = $|| X.v ||^2_2 $ (Which is the euclidean norm of $X.v$) i.e hessian matrix is positive semi-definite.

But I don't understand how to prove $ X^T X $ is positive definite or $L2$ loss is strictly convex.

-please ignore the constants in $L2$ loss.


texts: Convolutional Neural Network with feature inputs also directly connected to fullly connected layers

texts:
Imagine you are trying to classify movie reviews you have both the actual review and also some information about the movie. I pass the actual review through a few convolutional layers and some max pool layers and then pass that information with the other raw features into a few fully connected layers. I haven't seen this approach being used very often, the network was having some difficulty generalizing so I applied some dropout. I was wondering if there are other techniques that people would recommend. I am not sure what to try next. Below is the pytorch code:

class Net(nn.Module):

def __init__ (self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv1d(in_channels = n_input_channels, out_channels = 40, kernel_size = 9, padding= 4)
    self.conv2 = nn.Conv1d(in_channels = 40, out_channels = 80, kernel_size = 3) #.double
    self.pool1 = nn.MaxPool1d(3)
    self.fc1 = nn.Linear(2484, 200)
    self.fc2 = nn.Linear(200, 200)
    self.fc3 = nn.Linear(200, 200)
    self.fc4 = nn.Linear(200, 99)

def forward(self, x):
    x2 = x[:, :, :4].contiguous().view(x.size(0), -1)
    x1 = x[:, :, 4:]

    x1 = F.relu(self.conv1(x1))
    x1 = self.pool1(x1)
    x1 = F.relu(self.conv2(x1))
    x1 = x1.view(x1.size(0), -1

    x = torch.cat((x1,x2), 1)

    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = F.softplus(self.fc4(x))
    return x

def init_weights(m):
    if type(m) == nn.Linear:
        nn.init.xavier_uniform_(m.weight)
        m.bias.data.fill_(0.01)

  net = Net()
  net.apply(init_weights)


texts: Correlation feature selection followed by regression

texts:
I have quarterly results data for a company with around 100 variables. Total 60 quarters results are available (total records 60).
sample data: (only few columns & 10 rows)

I would like know following,

which ML algorithms / methods to be used to get the information about most important variable(s) affecting the price movement

how much each variable is contributing towards price movement (+ve or -ve)

predict, if a variable value is changed then how it'll affect the price movement

Thanks!


texts: What is the difference between dcast and recast in R?

texts:
I am working with a dataframe in R that is formatted like this sample:

Countries <- c('USA','USA','Australia','Australia')
Type <- c('a','b','a','b')
X2014 <- c(10, -20, 30, -40)
X2015 <- c(20, -40, 50, -10)
X2016 <- c(15, -10, 10, -100)
X2017 <- c(5, -5, 5, -10)

df_sample <- data.frame(Countries, Type, X2014, X2015, X2016, X2017)

The dataframe looks like this:

  Countries Type X2014 X2015 X2016 X2017
1       USA    a    10    20    15     5
2       USA    b   -20   -40   -10    -5
3 Australia    a    30    50    10     5
4 Australia    b   -40   -10  -100   -10

I want to be able to create columns of year values for each type by each country, yielding something that looks like this:

        Countries   Year     a     b
1       USA         X2014    10   -20
2       USA         X2015    20   -40
3       USA         X2016    15   -10
4       USA         X2017     5    -5
...

With recast I get this:

recast(df_sample, Countries ~ Type)

  Countries a b
1 Australia 4 4
2       USA 4 4

With dcast I get this:

dcast(df_sample, Countries ~ Type)

  Countries a   b
1 Australia 5 -10
2       USA 5  -5

The dataset I'm working with has 44 years of data, so I'd like to be able to indicate all columns of yearly data without having to enter each column id manually into a cast formula.

What is the difference between dcast and recast (i.e. what situations might they be best suited to), and is it possible to shape my data with them?


texts:
I am a new user to Matlab kmeans and have the following question:
I would like to use the following call from a Python application which is listed in this URL.
[idx, C] = kmeans(data, k)

Question 1: Do I need to allocate idx and C before this call to kmeans() or should I call the function without pre-allocation?
idx = []
C = [][]

[idx, C] = kmeans(data, k)

Question 2: After the call, can I manipulate idx array as well as C array independently?
In closing my source of confusion is [idx, C] implies one multi-dimensional array but dimensions of idx is "nx1" where n is the number of data points while dimension of C is two dimensional array [kX2]?