texts: Convert .cbf to lossless image format with defined "dynamic range"

texts:
We collected electron diffraction data and converted the data into .cbf (Crystallographic Binary File). Those are about 10'000 files. Now I would like to convert the .cbf into an image format like .tif but with a specified dynamic range.

I tried it with fabIO which is a python module. But I could not figure out how to successfully convert .cbf into .tif. .edf worked but it's not the  desired format. Also I don't know how to pick the dynamic range.

Maybe it is possible to use Adxv in the command line? I've been using it to export .tif but was only able to do so one by one.

What follows is my python script to convert .cbf into .edf:

import os
from glob import glob
import fabio

path = '/Users/myuser/Documents/Crystals'

result = [y for x in os.walk(path) for y in glob(os.path.join(x[0], '*.cbf'))]

i = 0
text = ''
while i < len(result):

    if i == 4000:
        image = fabio.open(result[i])
        image.convert("edf").save("4000.edf")

    i += 1

Do you know how to use fabIO in order to export .tif with specific dynamic range? Or do you maybe know how to use Adxv in the command line in order to process many images? Or maybe you have another solution?


texts: Is it better to use F1 score or AUC metric for imbalanced data classification?

texts: I have a text classification problem, where the "positive" examples are the minority. What metric is better to use for binary classification for this case - F1-score or AUC?


texts: Classification/Regression of Dataset taken at different time

texts:
I have a dataset with some gait parameters (step length, stride length etc)  of 100 people taken 3 times at different time(in every 6 months).
Now I have to train my model  on this dataset and predict if the person has a disease or not for any new person data that is given.
How can I take all this 3 data of parameters in my model for training considering time factor.
I checked time series forecasting but it looks like for that the dataset should be dependent on continuous time instances.


texts: How to use a multiple linear regression model built from normalized data

texts:
I built a linear multivariable regression model from normalized data (for the interval [0; 1]). Initially, the data was not normalized, I normalized the data by myself (independent and dependent variables).
I want to use this model to make predictions from newly received data (I get the values of independent variables and I want to predict the value of the dependent variable). The problem is that the data comes in a raw, unnormalized form.

How can I normalize newly arriving data if only one "observation" is received?
What if I want to get the real values of the dependent variable using my model, and not the normalized ones?


texts: How to standarize feature vector with data in different scales?

texts:
Let's suppose I have a dataset with numerical attributes of different types.

Let's suppose I want to employ a Neural Network for supervised classification with that dataset. For that, I need to extract feature vectors from that data.

Those feature vectors must be suitable for NNs. (should be normalized/standarized vectors...)

  As an example, our dataset consist of data from football games.

DATASET:

-------------------------------------------------------------------
| local_elo| vis_elo| local_pts | vis_pts | loc_goals | vis_goals |
-------------------------------------------------------------------
|   2820   |  3250  |     45    |    54   |    13     |     17    |
-------------------------------------------------------------------
|   4230   |  5125  |     87    |    81   |    67     |     65    |
-------------------------------------------------------------------

The feature vectors this two data points:

x_1 = [2820, 3250, 45, 54, 13, 17]
x_2 = [4230, 5125, 87, 81, 67, 65]

but they are not suitable for feed them into a Neural Network.

How could this dataset be preprocessed in order to extract feature vectors suitables for feeding them into a Neural Network?


texts: How does graph classification work with graph neural networks

texts:
I am reading the paper The Graph Neural Network Model by Scarselli et al. I understand how node classification works. I am having trouble understanding how graph classification works however. In particular, in the section titled The Learning algorithm, the authors mention that

  Learning in GNNs consists of estimating the parameter such that w
  approximates the data in the learning data set

  where qi is the number of supervised nodes in Gi. For graph focused
  tasks, one special node is used for the target (qi = 1 holds), whereas for
  node-focused tasks, in principle, the supervision can be performed on
  every node.

The node focused task approach makes sense to me; you would essentially compare the ground truth to each output of the "local output function" for each node, and backprop accordingly. However, based on the above description, I do not understand what you would do to classify the graph as a whole, given its label. What do they mean by "one special node is used for the target (qi = 1 holds)"? Why are they talking about a "special node"? Why is there no mention of the graph's label? Isn't that what we want to predict?

EDIT:

After reading through the entire paper, and specifically looking at the Mutagenesis example, I got a better understanding of how graph classification works (as described in this paper at least). However, my understanding is still not complete. I will explain what I understand, and raise a followup question below.

As the text above suggests, a particular node in the graph is chosen (I believe this can be done at random), and it will be the only node in the graph that is "supervised." All other nodes will be unsupervised (so we will not make any predictions on those nodes). We choose the local output function in such a way as to have it output a number between -1 and 1 (although I'm unsure as to whether or not you could pick a function that instead outputs a number between 0 and 1. I believe you can, and it's just a matter of what activation function you would like to choose i.e sigmoid vs tanh in this example). If the output is < 0, we predict the graph has label -1, and 1 otherwise.

Now we just do what we did with node prediction, except we only backpropagate on this single node that we chose.

This, however, raised a followup question for me. If you are training on multiple graphs (for graph classification), each of which has a different connectivity (which is usually the case, and is the case in the Mutagenesis example), how do you backpropagate? Each graph (in this case, molecule) represents a different neural network ...


texts: Regression error increase after shuffing data

texts: I'm trying to do multivariate regression using a 3-dimension data set. I noticed a strange problem that my fitting error increase dramatically after I pre-shuffled the data matrix comparing using original data. Is that possible? I've checked my code and confirmed that it's the shuffled process that increased the error. Why is that? Shouldn't shuffle process only change the order of the data?


texts: Overfitting with text classification using Transformers

texts: I am trying to make a binary text classification model by using the encoder part of the transformer and then using its output to feed into an LSTM network. However, I am not able to achieve good accuracy on both the training set (92%) and the validation set (72%). Is my approach correct? Please tell me a better way to design the model and improve accuracy.


texts: Understanding the Gini/AUC metric as out-of-development performance metric

texts:
Assume we develop a model for a binary classification task that reaches a certain Gini/AUROC estimate on the validation ( or training ) sample, among others. This is an overall good metric, often used for evaluating the ability of the model to separate the samples into, say, goods vs bads.

Further, assume this model is adequate and  will be used for further collection of new samples with a certain cutoff value. What should be expected Gini/AUC estimates on the newly collected sample?

From what I'm noticing, on the training sample there were clear cases where the model was able to distinguish and separate it with large probabilities. On the other hand, with applied cuttoff, say, <50%, the new sample with collect only those cases where no such clear separation is possible (because if it would, the case might not get collected). With such approach, for me it seems logical that the overall separation in the new sample will be lower, resulting in lower out-of-development-period Gini/AUC.

Is this the expected behaviour in normal production environments? Am I understanding things correctly?

Note: I understand that there are other simple metrics, such as sensitivity/specificity, hoslem.test and others, allowing for measuring and visualising True/False Positives. However, I have found that Gini/AUC is often a key metric when discussing and comparing classification models.


texts: How to compute conditional probability in code?

texts:
In this paper http://www.aclweb.org/anthology/D13-1176, in "2. Framework" it says

  We begin by describing the modelling framework underlying RCTMs. An RCTM estimates the probability $P(f \ | \ e)$ of a target sentence $f = f_1, ..., f_m$ being a translation of a source sentence $ e = e_1, ..., e_k$. Let us denote by $f_{i\ : \ j}$ the substring of words $ f_i, ..., f_j $. Using the following identity,

$$
P(f \ | \ e) = \prod_{i=1}^m P(f_i \ | \ f_{1:i−1}, e)
$$

  an RCTM estimates $P(f|e)$ by directly computing for each target position $i$ the conditional probability $P (f_i|f_{1:i-1}, e)$ of the target word $f_i$ occurring in the translation at position $i$, given the preceding target words $f_{1:i-1}$ and the source sentence $e$.

I understand the identity that they used, but how is it implemented in terms of code? How can a module calculate such a probability?


texts: What is the difference between a decision tree and something called "subgroup discovery algorithms"?

texts:
I'm reading a paper which states that subgroup discovery is:

Subgroup discovery is a data mining technique whose goal is to detect interesting subgroups into a
population with respect to a property of interest

The paper goes on to make the distinctions between a decision tree and subgroup discovery, but does not explain the rationale/reasoning.

With a google search on subgroup discovery algorithms I find the following:

The goal of the subgroup discovery algorithm SD, outlined in Figure 1, is to search for rules that
maximize $q_{g} =
\frac{TP}{FP + g}$, where TP are true positives, FP are false positives, and g is a generalization
parameter. High quality rules cover many target class examples and a low number of non-target e
xamples

and decision tree according to Wikipedia:

A tree is built by splitting the source set, constituting the root node of the tree, into subsets—
which constitute the successor children. The splitting is based on a set of splitting rules based on
classification features.[2] This process is repeated on each derived subset in a recursive manner
called recursive partitioning.

This + my prior knowledge of decision trees is not able to make any distinctions between the two. Is a decision tree a part of "Subgroup discovery algorithms" or not?


texts: Running out of memory while processing csv file data

texts:
I'm getting an error while processing 0.2 million of text data. I'm using CNN text classification in tensorflow.
Output raw data shape is 204177x22000. while passing to numpy.array(out_raw), here it is consuming 100% memory(Using 8GB RAM).
Tried with data in batch but didn't work.
If i need to increase my RAM size then kindly help me out with formula.

What are the methods to take care of this problem statement?


texts: Laplacian smoothing on Class Probability (Naive bayes)

texts:
I am implementing a Naive Bayes classifier in Python from scratch. The instructions I have asks that I incorporate Laplacian Smoothing with K=1 to computing the probability that a message belongs to a given class. I have two classes, ham and spam (E-mail spam filtering problem). So one example without smoothing would be if I had X spam messages and Y spam messages:

P(X) = X / (X+Y) --> Probability that a class is spam in my dataset
P(Y) = Y / (X+Y) --> Probability that a class is ham in my dataset

I don't know how to incorporate Laplacian Smoothing to this simple formula.


texts: Kernel on graphs and SVM : a weird interaction.

texts:
I have a collection of 80k graphs, half of them are labeled 1 (they represent the cities of France) and the other half is labeled 0 (representing cities in the USA). My goal is to create a classifier that can predict whether a graph (representing a city), using a kernel on graphs.

Since nodes in graphs representing cities are generally of degree << number of nodes, I decided to use the all connected 3,4,5 graphlet kernel presented in this article : Article about all connected graphlet kernel.

What this kernel does is simple : it represents each graph G by a vector F(G) which is the distribution of the all connected 3, 4 and 5 graphlets in the original graph. There are 29 different graphlets of size 3, 4 and 5 so every graph is now represented by a vector of length 29. The article then defines the kernel between two graphs G and G' by the inner product between the two vectors :

K(G,G') = <F(G),F(G')>

This means that this kernel does not behave like others kernels we are used to work with : we specifically know the feature map and the Hilbert space associated to the kernel. The Hilbert space is R^29 and the feature map is F.

Then in order to create the classifier I use SVM.

But here's the problem : if I use the feature map F as it is, then compute the gram matrix and apply SVM, all predictions on the test data are 0. That's pretty sad.

What I have been noticing is that 3 of the 29 graphlets are much more present than the others in the graphs. So for a typical city, 95% of the distribution is concentrated on theses 3 graphs, which means that when you compute the kernel (the inner product), the gram matrix have really similar values : the standard deviation of the matrix is 0.0003. Naturally, I though : I want to transform F so that the differences between graphs are clearer.

So I computed the mean distribution M of 3,4,5 graphlets in my database, I derived the inverse mean distribution M' (I just inverted every element of M), and instead of using F, i used F* which takes a graph G, computes F(G), and then F*(G) by multiplying point by point F(G) and M'. That way, the differences between graphs when computing the kernel are more easy to see. Note that the kernel is still valid since it's still an inner product in R^29.

I have no idea how mathematically accurate is this intuition, but when I applied SVM with this other feature map F*, I got a 82% accuracy, and the AUC was 0.88. The ROC curves is beautiful too. So I must have done something right ?

My actual question is why does SVM likes the feature map F* more than F ? I mean I didn't add any information using the new feature map, I just modified the old one using my intuition.


texts:
I've been playing around with YOLOv3 and obtaining some good results on the ~20 custom classes I trained. However, one or two classes look like they can use some additional training data (not a lot, say about 10% more data), which I can provide.

What is the most efficient way to train my model now? Do I need to start training from scratch? Can I just throw in my additional data (with the appropriate changes to the config files etc.) and run the training based on the weight matrix I already acquired, but for a small number of iterations? (1000?) Or is this more like a transfer learning problem now?

Thanks for all tips!


texts: Is it possible to use grayscale images to existing model?

texts:
From tensorflow's object recognition (R-CNN)

I'm re-training the existing model with new categories: the types of clothes (jeans, pants, blouse, and so on). Since we don't need colors to determine the type of clothes that user is wearing, I want to re-train it with gray-scale images. Is it possible to use gray-scale images to train existing model (which are trained with color images)?

I'm concerned because they trained their model with color images.

Does the model just consider the grayscale image as color image? And does it still work? :)

p.s I'm generating XML and csv files to put data for training and testing.


texts: How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?

texts: I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both the train score and test score as 99%. So I am pretty sure the model is giving biased result. One of the reason might be because the record of communication loss events are very less compared to the the record with no communication loss Some people advised me that it might not be possible to build a prediction model based on the situation. But I would like to have more suggestion or advice if there is anything I can do about it.


texts: K-means is a well known algorithm for clustering, but there is also an online variation of such algorithm (online K-means). What are the pros and cons of these approaches, and when should each be preferred?


texts: Image classification if rotated version same

texts:
I asked this question on stackoverflow but was advised to come here.

I have some images to classify. I see that Convolutional neural network may be best for this, e.g. here. However, for my images, their rotated versions (to any degree) is effectively same image. What should I do so that this aspect is also taken into account.

One method I thought is to create many rotated versions (say, every 30 degree rotation) of each training image and add them to train set with same label. Is there any other way? Also, will it help if I get each image in a circular area?

Two example images are shown below:


texts: Is there a RandomForest implementation that handles categorical data without encoding in python?

texts: I am working on a binary classification project with both continuous and categorical features. I know that the R implementation of RandomForest can handle categorical data passed in as factor type data. Python's scikit-learn implementation requires encoding of categorical data however (e.g. one-hot). I'm curious about the difference in the results I receive using the two implementations, and I'm wondering if anyone knows of a python implementation of RandomForest that can handle categorical data without encoding.


texts: does random forest require more trees to be built if # rows increase in train data

texts:
I am building a randoForest model on some data. To be able to tune the model faster, I was working with 10% sample. I have tuned it now and now I want to build the model on the whole (100% data).

I am wondering, if I will have to increase the number of trees now that I am building the model on more training data points? Please let me know.


texts: I'm involved in a smart viticulture project. I need some datasets for machine learning models. Datasets about grape disease or ripeness related to climate variables or grape.


texts: When introducing new data against the model an error is produced, this is due to some new words within sentences, how to adapt R code for it to work?

texts:
In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results for the model produces different outcome in the tables, which is good. However, when executing..

results2 = classify_models(container2, models)

..when feeding in new data (matrixdata2) against the model it produces an error message:

Error in predict.svm(model, container@classification_matrix, prob = TRUE,  :
  test data does not match model !

In checking the datasets, I understand the train set's sentences used to create the model contains specific words, but the new data fed against the model include new words not recognised in the train set. I am wondering how to overcome this problem. Few options I am thinking of are  the following:-

Limit sentences in the new data to include only words that are in the train set,
Or point the ML to consult the dictionaries to look up additional words
Or for the machine learning process to identify ones that match and ones that do not match simply say I don't know response from ML.

What is the best option to overcome the problem? What would I need to translate this idea into adapting the R code below?

Update: 18/04

I created a few sentences that contained only words that appeared in the creation of the model. I fed in this few sentences with its labelled sentiment (new data) in against the model, but it still produced the same error message above. I do not understand why this is the case as these words are recognised in the trainset. However, when I used one of the same sentences in the new data to feed in against the model, it worked, so from what I can tell is if the sentence does not exactly match whats in the trainset, then it produces the error. I am unsure how to overcome this, please help me resolve this problem.

#Load Libraries
library(RTextTools) #RTextTools available for 3.4.1
library(e1071)
library(gmodels)

setwd(directory/path)
text= read.csv("matrixdata1.csv", header = FALSE)

# build dtm
matrix= create_matrix(text[,1:2])
mat = as.matrix(matrix)

# build the data to specify response variable, training set, testing set.
container = create_container(mat, as.numeric(as.factor(text[,3])),
                             trainSize=1:672, testSize=673:840,virgin=FALSE)

models = train_models(container, algorithms=c("MAXENT" , "SVM", "RF", "BAGGING", "TREE"), set_heldout = 168)

#container1
results1 = classify_models(container, models)

text2 = read.csv("matrixdata2.csv", header = FALSE)
matrix2= create_matrix(text2[,1:2])
mat2 = as.matrix(matrix2)

container2 = create_container(mat2, labels=NULL, trainSize=1:500,testSize=NULL, virgin=TRUE)

#Results from feeding in new data against the model
#When running this code below, it produces the error message outlined above in the description of the problem.
results2 = classify_models(container2, models)

Datasets used in the R code


texts: Does convergence of loss function is always guarnteed?

texts:
Which of the following is true, given the optimal learning rate?
(i) For convex loss functions (i.e. with a bowl shape), batch gradient descent is guaranteed to eventually converge to the global optimum while stochastic gradient descent is not.
(ii) For convex loss functions (i.e. with a bowl shape), stochastic gradient descent is guaranteed to eventually converge to the global optimum while batch gradient
descent is not.
(iii) For convex loss functions (i.e. with a bowl shape), both stochastic gradient descent and batch gradient descent will eventually converge to the global optimum.
(iv) For convex loss functions (i.e. with a bowl shape), neither stochastic gradient descent nor batch gradient descent are guaranteed to converge to the global optimum
Which option is correct and why?


texts: Simple 3 layers Neural Network cannot be trained

texts:
#3 layers neural network
import numpy as np
from __future__ import division
def nonlin(x,deriv=False): #activation function
    if(deriv==True):
        return np.exp(x)/(1+np.exp(x))**2
    return 1/(1+np.exp(-x))

X = np.array([
                [0,0,1],
                [0,1,1],
                [1,1,1],
                [1,0,0],
                [0,1,0],
                [1,1,0]])
Y = np.array([[0,1,0,1,1,0]]).T
np.random.seed(1)
l0 = X
syn0 = 2*np.random.random((3,30))-1
syn1 = 2*np.random.random((30,1))-1

for i in xrange(60000):
    l0 = X
    l1 = nonlin(np.dot(l0,syn0))
    l2 = nonlin(np.dot(l1,syn1))
    l2_error = Y-l2
    l2_delta = l2_error*nonlin(l2,deriv=True)

    l1_error = l2_delta.dot(syn1.T)
    l1_delta = l1_error*nonlin(l1,deriv=True)
    syn0 += l0.T.dot(l1_delta)
    syn1 += l1.T.dot(l2_delta)
print l2

I have been messing with the neural network implementation at https://iamtrask.github.io/2015/07/12/basic-python-network/

This is the output of the code:

[[1.85572928e-04]
 [9.99755942e-01]
 [5.21248255e-09]
 [9.99767481e-01]
 [9.99963580e-01]
 [2.07334909e-04]]

I expect something like Y

[[0]
 [1]
 [0]
 [1]
 [1]
 [0]]

What could have possibly gone wrong here?


texts: Mxnet deepdog (hot dog not hot dog) example - how does the network know it is classifying a hotdog

texts:
Looking at the mxnet documentation.
It takes the pretrained squeenext1_1 weights, and sets imagenet_hotdog_index variable to 713.
    net = models.squeezenet1_1(pretrained=True, prefix='deep_dog_', ctx=contexts)
# hot dog happens to be a class in imagenet.
# we can reuse the weight for that class for better performance
# here's the index for that class for later use
imagenet_hotdog_index = 713

Then they set a 2 class output layer on top of it
    deep_dog_net = models.squeezenet1_1(prefix='deep_dog_', classes=2)
deep_dog_net.collect_params().initialize(ctx=contexts)
deep_dog_net.features = net.features
print(deep_dog_net)

Where I get confused is in the classify_hotog function - it is applying softmax to the output layer, and then returning the highest result index. This would make perfect sense if we had somehow told the network to compare against index 713. But it is called for prediction before the index variable is being reused? How does the network know the class to compare against is index 713/hot dog? We've basically taken squeezenet and reduced it down to 2 class output. But how does the network know what class to compare it to? Why would it give a high probability to the second argument/class when showing a hotdog - seems to me it shouldn't know what class it is comparing it to?
out = mx.nd.SoftmaxActivation(net(image.as_in_context(contexts[0])))
print('Probabilities are: '+str(out[0].asnumpy()))
result = np.argmax(out.asnumpy())

I would have expected it to maybe use the full squeezenet output layer, and set something like
if np.argmax(out.asnumpy()) == 713:
     "Hot Dog!'
Else:
     "Not hot dog!"

Never understood this and would appreciate if anyone could help me get this detail.


texts: Comments vs. Files (& how to import w/o packages) for Data Science

texts:
I am trying to settle on an efficient & pythonic workflow for developing scripts in data science work that are scalable and don't turn to spaghetti once things get complicated and parameters are changed. Until now, I've been using header comments, but it seems like splitting things into multiple files keeps things the cleanest.
However, I haven't been able to figure out how to import functions/variables from within the script directory because it is not a package. I do not think I want to make every little script I write into a package, so what I can I do to get around this? The best I've found is from this thread:
from . config import * -> ImportError: attempted relative import with no known parent package
Alternatively, does anyone have experience with this issue or resources on structure? I've been loosely using Cookie Cutter Data Science, but I'm reaching the point where every item in the "features" folder (for example) needs its own file for data assimilation/processing/QC visualization.


texts:
I have data of intraday electricity consumptions (by half hours - 48 a day) over a year of 4000 households. Task is to establish baseline consumption of each of these households - possibly also differentiated on seasonality.

One way how to do this would be just taking the mean of the consumption signals. What would be more sophisticated method for this?

I would be very grateful for pointing out to methods I could look into.


texts: How to apply TFIDF in structured dataset in Python?

texts:
I know that TFIDF is an NLP method for feature extraction.
and I know that there are libraries that calculate TFIDF directly from the text.
This is not what I want though
In my case, my text dataset has been converted into Bag of words
The original dataset that I "DO NOT" have access to, looks like this
RepID     RepText
------------------
1         Doctor sys patient has diabetes and needs rest for ...
2         Patients history: broken arm, and ...
3         A dose of Metformin 2 times a day ...
4         Xray needed for the chest...
5         Covid-19 expectation and patient should have a rest ...

But my dataset looks like this
RepID   Word         BOW
-------------------------
1       Doctor       3
1       diabetes     4
1       patient      1
.       .            .
.       .            .
2       patient      2
2       arm          7
.       .            .
.       .            .
5684    cough        9
5684    Xray         3
5684    Covid        5
.       .            .
.       .            .

What I want is to find TFIDF for each word in my dataset.
I was thinking of converting my dataset into a unstructured format
so it would look like this
RepID     RepText
------------------
1         Doctor Doctor Doctor diabetes diabetes diabetes diabetes ...
2         Patients patients arm arm arm arm arm arm arm ...
.
.
5684      cough cough cough cough cough cough cough cough cough Xray Xray

so each word repeated the same number of BOW
but I do not think this is the best way to do as I convert a structured dataset into an unstructured one..
How to find the TFIDF from the structured dataset? is there a library or algorithm for that?
Note :
Dataset stored in MS SQL Server, and I am using Python code.


texts: Should I consider feature scaling for all gradient descent based algorithms?

texts:
In the Coursera course machine learning in the section on Multivariate Linear Regression, Andrew Ng provides the following tips on gradient descent:

Use Feature Scaling to converge quicker

Get feature into an approx -1 < x < 1 range
Mean normalization

Andrew Ng also provides some other tips:

Plot cost vs iterations

to ensure cost decreases on every iteration (try smaller alpha)
to identify if convergence is too slow (try larger alpha)
to identify approximately the number of iterations to converge

Are these tips applicable to all problems involving gradient descent using different machine/deep learning algorithms or just to Multivariate Linear Regression?


texts: I am getting different mean_absolute_error when i retrain my model everything same

texts:
I have set my numpy random seed to 0.
I am training on colab and using keras.
I didn't change anything.
I just re-ran my cell and the val_absolute_error changed.

Code:

np.random.seed(0)
regressor = Sequential()
regressor.add(Dense(10, input_dim=1, activation='tanh'))
regressor.add(Dense(20, input_dim=1, activation='relu'))
regressor.add(Dense(15, input_dim=1, activation='tanh'))
regressor.add(Dense(1))
regressor.compile(optimizer='adam', loss='mean_absolute_error', metrics=['mae'])
model = regressor.fit(X_train, y_train, epochs=450, batch_size=10, validation_data=(X_val, y_val), verbose=1)

print(regressor.evaluate(X_test, y_test)) --> This is the error on the test set

Please note that I made a mistake in the screenshot, it is the error in test set not val. set


texts: Can we compare a word2vec vector with a doc2vec vector?

texts:
I have set of categories and I want to compare a document vector with word vector of categories to find best matching category.

Is it possible to compare a word vector with document vector? If yes, is there any literature which gives proof of concept for this?


texts: Regarding the application of RNN on new data

texts:
Suppose the two univariate time series $X_{1,T}=(x_1, x_2, ..., x_T)$ and $Y_{1,T}=(y_1, y_2, ..., y_T)$. The next step would be to train an RNN or LSTM with input $X_{1,T}$  and output $Y_{1,T}$, in order to model the function $f(x_t) = y_t$. The model produced will be denoted $M$.
The questions regards the application of $M$ on new data $x_{t},t>T$.

If $x_{T+1},x_{T+2},x_{T+3}$ are unknown, but $x_{T+4}$ is known, can $M$ be used to predict $y_{T+4}$? Or is it necessary first for the model to predict $y_{T+1},y_{T+2},y_{T+3}$ beforehand?
I know that for standard machine learning algorithms e.g. decision trees or svm, applying the model for prediction does not change the model in any way. Is this also generally true for RNN/LSTM? For instance, if $M$ is applied twice in row on $x_{T+1}$, will I get two different results for $y_{t+1}$, as the model assumes that I am providing $x_{T+1}$ and $x_{T+2}$?


texts: How is maximizing L(lambda1, lamda2, lamda3) equivalent to minimizing perplexity?

texts:
In language modeling, L(lambda1, lambda2, lambda3) is defined as:

Sum(count of trigram(u,v,w) x q(w|u,v))

where u, v, w are words in the corpus
and
perplexity is defined as:

2^-l

where

l = (1/M)Sum(log(q(w|u,v)).

where M is the total no. of words in the corpus.

Also,

q(w|u,v) = lambda1*q(w|u,v) + lambda2*q(w|v) + lamdba3*q(w)

The q-values on the right hand side are maximum likelihood estimates.
Some of the materials for Natural Language Processing state that maximizing L is same as minimizing perplexity. I don't see how that is true or can be mathematically proved.


texts: Difference between tf-idf and tf with Random Forests

texts:
I am working on a text classification problem using Random Forest as classifiers, and a bag-of-words approach.
I am using the basic implementation of Random Forests (the one present in scikit), that creates a binary condition on a single variable at each split. Given this, is there a difference between using simple tf (term frequency) features. where each word has an associated weight that represents the number of occurrences in the document, or tf-idf (term frequency * inverse document frequency), where the term frequency is also multiplied by a value that represents the ratio between the total number of documents and the number of documents containing the word)?

In my opinion, there should not be any difference between these two approaches, because the only difference is a scaling factor on each feature, but since the split is done at the level of single features this should not make a difference.

Am I right in my reasoning?


texts: L2 regularization increase the loss rate of the deep learning model

texts:
When I add L2 regularization to my deep learning model the training and validation loss rate is  increased.

Why ????


texts: Approach fpr extracting/cropping features images using deeplearning and no annotations

texts:
Let's say I want to have a bunch of images of hats from videos. How would I priniciple build something that would learn to recognize, and crop or bound box hats? I heard you need a dataset with bounding boxes manually drawn for training, but it seems there would be a way for a nn to identify that on their own?

I'm trying to understand the possibility of scraping video for different items. I.e. give it images of 1000 hats, and then it will crop out images of hats from a bunch of video files.

I am thinking this could be an interesting thing to work on, but would need some advicein terms of how to arrpoach it.

Also, the next logical thing is then to put hats on people in movies somehow, but that would be phase 2.

Thanks


texts:
I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes:

2019-02-20
20.02.2019 ("German format")
02/2019 ("February 2019")
"tomorrow" (datetime.timedelta(days=1))
"yesterday" (datetime.timedelta(days=-1))

Is there a Python package / library which offers this already or do i have to write all of those regexes/logic myself?

I'm interested in Information Extraction from German and English texts. Mainly German, though.

Constraints

I don't have the complete dataset by now, but I have some idea about it:

10 years of interesting dates which could be in the dataset
I guess the interesting date types are: (1) 28.02.2019, (2) relative ones like "3 days ago" (3) 28/02/2019, (4) 02/28/2019 (5) 2019-02-28 (6) 2019/02/28 (7) 2019/28/02 (8) 28.2.2019 (9) 28.2 (10) ... -- all of which could have spaces in various places
I have millions of documents. Every document has around 20 sentences, I guess.
Most of the data is in German


texts: How to find vertical clusters in 1-D data

texts:
I have residuals of a multivariate time series data obtained from sensors on a server.spikes in the plots of residuals indicate abnormal server state. I want to cluster the data into vertical clusters and get the indices of the points in each cluster so that I can go back and look at the actual data and get the mean values of various parameters of a cluster.

I tried 1d Gaussian ,Kmeans etc but they all seem to cluster the data horizontally.

I want each spike to be a separate cluster and get the indices of the values in the cluster. Can any one suggest a technique to overcome this issue.
Thanks.


texts: Counting how many rows contain chinese in mixed value dataset using python

texts:
I'm wanting to count how often Chinese is used, I've tried langdetect but the issue is the values are a mix of numbers, text and even URLs which langdetect doesn't seem to support

import pandas as pd
from langdetect import detect

df = pd.read_csv("annotation_sample.csv", nrows=10)
df = df.drop(['id', 'profile_id', 'created', 'document_id'], axis=1)
df = df[pd.notnull(df['txt'])]
df

df.txt.apply(lambda x: detect(x))

Above works (although I've not figured out how to count a specific language) although it only works as its limited to 10 rows, when using the entire 10,000 it throws the 'No features in text' error

Pic of the data:

What I need help with is;
How do I avoid running this on rows that contain just numbers, urls etc? Is there a way to skip error'd ones and continue?

Secondly, how do I then count how many have come back as a specific language?


texts: Any issue with "overlapping" sliding windows in time-series data analysis?

texts: I am developing some classification/regression models form accelerometry time-series data. So far, I have created datapoints by extracting features from non-overlapping sliding windows of the time-series data. I would like to try using overlapping windows as well. However, I was wondering whether it is conceptually sound or there might be some caveats to keep in mind as the data is reused in the overlapping windows.


texts: How to choose a kernel function and a feature mapping function?

texts:
Although, after extensive of reading, I know the concepts of support vector machines pretty well by now, I have trouble translating the concept of the kernel function $K$ and the feature mapping function $\phi$ to a simple example such as the following.

My example data $x \in \mathbb{R}^2$: $(1,0), (4,0)$ are from one class, $(2,0), (3,0)$ are from another.

So here are my two questions:

Would $\phi((x_1,x_2))=(x_1,x_2,(x_1-2.5)^2)$ be a wise choice for the mapping function $\phi:\mathbb{R}^2 \to \mathbb{R}^3$ ? If not, what $\phi$ would be a wiser choice?
What would be the corresponding choice for the kernel function $K$?


texts:
kNN is a distance-based method, so it requires the input to be in numerical form.
I was wondering if it is possible to use kNN imputer for non-ordinal categorical variables (like color).
Since the input has to be in numerical form, we have to encode the color feature before applying the kNN imputer.
Using ordinal encoding doesn't seem like a good idea. If we assign numbers 1-10 to colors, then the distance-based measurement will assume that the distance between color 1 and 3 is the same as the distance between color 2 and 4. In reality no such relationship exists.
We could use one-hot encoding. Let's say we end up with 10 one-hot encoded color columns, including a column which indicates the missing values. Our task would then be to apply the imputation to the rows which have a 1 in the one-hot-encoded missing-color column. And we would want the imputation to decide which of the other 9 one-hot color columns would have a 1 instead of a 0. But this is generally not at all how kNN imputation works.
Can you advise if using kNN imputation in this case is possible? How can I apply it?


texts: How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?

texts:
Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of simply quick-sorting one dimensional data of size n goes as n*log(n).

So, for example, if we regress y against X where X is a d-dimensional variable, does it go as O(n^2*d)? How does it scale if I want to find the solution via exact Gauss-Markov solution vs numerical least squares with Newton method? Or simply getting the solution vs using significance tests?

I guess I more want a good source of answers (like a paper that summarizes the scaling of various statistical techniques) than a good answer here. Like, say, a list that includes the scaling of multiple regression, logistic regression, PCA, cox proportional hazard regression, K-means clustering, etc.


texts: How metric learning works for content based item retrieval

texts:
I was doing some computer vision experiments and recently I have started learning about metric learning and the image retrieval problem. I was experimenting with the inshop image retrieval dataset to build a content based image recommendation system.

The dataset is structured as shown above, here is the complete file.
I thought of using Multi-Similarity Loss Function with Batch Hard Triplet Mining for training. What I could understand from the way any variant of triplet loss works; is that it tries to bring the items from same class closer to each other and those from different classes away from one another. I [referred this post][4] for the same.

I wondered then, how will it learn global concepts or will it learn
only local semantic information? I thought it should be the latter
because we're explicitly framing the problem in this manner. But the
results which I got confirmed that it also captures global level
semantic information.

For eg. If there's a blue floral dress, a green floral dress and a red floral dress in the training data, it will try to bind the products of blue, green, red floral dresses closer whereas pushing the inter-class members as far as they can. But when I try doing a similarity search in the test set; if I have a blue floral dress & if I have 10 dresses in my inventory (let's say 8 solid print dresses and 2 floral dresses (green and red)), then the algorithm gives a higher ranking to the two floral dresses than and puts them above the ones with the solid print.
What's confounding here is that I didn't give any tag related information or try to weight one class/product set more over the other when training the model with the triplet loss function but it still captured the similarity across different product sets pretty well. So, how did it achieve that? How come a byproduct so good is achieved that captures the semantic info so well; when our main focus was to separate the images of different classes as far apart as we could (As per the definition of the triplet loss function)?
I would be highly obliged if anyone could provide me some insight into the working of metric learning using contrastive/triplet loss functions and if you think there's any better way to actualy do it I would be delighted to hear about that from you as well
Thanks!


texts: Can setting of different thresholds help in model performance in case of handling class imbalances?

texts: In a binary classification problem where there is class imbalance, after applying undersampling/oversampling or SMOTE techinques, is it still convention to use a 0.5 threshold if we are making the two classes completely balanced? Or should we still change the threshold based on what we are trying to optimize?


texts:
I installed bert-as-service (bert-as-service github repo) and tried encoding some sentences in Japanese on the multi_cased_L-12_H-768_A-12 model.  It seems to work as I am getting vectors of length 768 per word but np.shape() shows this for each sentence:

np.shape(vec_j[0]): (25, 768)
np.shape(vec_j[1]): (25, 768)
np.shape(vec_j[2]): (25, 768)
np.shape(vec_j[3]): (25, 768)
np.shape(vec_j[4]): (25, 768)
type: <class 'numpy.ndarray'>

My sentences are short so there is quite a bit of padding with 0's.  Still, I am unsure why this model seems to have a maximum sequence length of 25 rather than the 512 mentioned here:  Bert documentation section on tokenization

  "Truncate to the maximum sequence length. (You can use up to 512, but
  you probably want to use shorter if possible for memory and speed
  reasons.)"


texts: Why is T test reweighting on a word X word co-occurrence matrix so effective?

texts:
I am going through Stanford NLP class: http://web.stanford.edu/class/cs224u/

A task in the homework is to implement T-test reweighting on a word X word co-occurrence matrix:
https://nbviewer.jupyter.org/github/cgpotts/cs224u/blob/2019-spring/hw1_wordsim.ipynb#t-test-reweighting-[2-points]

$$\textbf{ttest}(X, i, j) =
\frac{
    P(X, i, j) - \big(P(X, i, *)P(X, *, j)\big)
}{
\sqrt{(P(X, i, *)P(X, *, j))}
}$$

I have 2 questions:

What is the intuition behind this formula? It looks a little like PMI but I can't understand what it's doing. The T-test explanation out there seems to be unrelated to this task.
It works amazingly well (when evaluated by this test): raw matrix yield a correlation score of 0.014, PMIed matrix 0.123 and t scored matrix: 0.408979. This number seems almost too good to be true for such a simple model. Can anyone bring some intuition/experience about why that is?


texts: SOM initial values for learning rate and neighborhood sigma

texts:
I am using SOM (Self-Organizing Maps) of Kohonen, or more specifically, the MiniSom, found here to cluster and visualize my data.

As you can see in the above site, the example given is:
from minisom import MiniSom
som = MiniSom(6, 6, 4, sigma=0.3, learning_rate=0.5) # initialization of 6x6 SOM
print "Training..."
som.train_random(data, 100) # trains the SOM with 100 iterations
print "...ready!"

the three parameters of sigma and learning rate and number of iterations are hardcoded, and apparently it is working for the given examples. However, my data is bigger and different in size, so when I use these hardcoded parameters, the results is awful. Or even worse, when I change my corpus and implement it on another corpus size, the results get worse... however when I try to find the "best initial values" by trial and error, I get something that is meaningful!
Question:
How can I determine the initial values of these parameters? Is there any other SOM implementation that facilitates this?
Your comments and answers are welcome. thanks.
annex:
There is an article on this, but apparently no right answer...


texts: Why do we use gradients instead of residuals in Gradient Boosting?

texts:
I have found mentions of two advantages in using gradients instead of actual residuals:

1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base learners to make them compatible with the loss function.

2) It can be computationally infeasible (in the case of MAE, we would have to calculate the median in every split).

However, I don't understand these points, namely:

1) If we just calculate the residual and have the base learner fit on those values, how exactly would that be any more difficult than calculating the gradients and then fitting on those values?

2) Following on the above question, why exactly do we need to calculate the median in the MAE example instead of just getting the residuals?

Perhaps I don't understand the exact mechanism behind optimizing an individual loss function. It would be really great if someone could demonstrate exactly how MAE is done and in which step would using residuals have been infeasible.


texts:
I was trying the Keras CNN Stater Code on Ubuntu 16.04, from the below link:
https://www.hackerearth.com/challenge/competitive/deep-learning-3/machine-learning/predict-the-energy-used-612632a9/#c144537

I get “MemoryError:” for

X_train = np.array(train_img, np.float32) / 255.

Any idea, what should I be doing?


texts: Error using corpus widget in Orange v 3.25.0 text mining and no import document option?

texts:
I am experimenting with Orange data mining tool. When I use the 'Corpus' widget from the text mining section it gives me the error: [![Corpus widget error][1]][1]
I have tried many things, but am still unable to resolve this issue.
[1]: https://i.stack.imgur.com/X4tKu.png
Besides that, in text mining, it does not show import document option in orange v 3.25. I just want to know the options to import text files into the Orange.


texts:
I have not understood why the maximum number of binary questions needed to reduce
uncertainty is essentially log(T).
If i have a ball in a box and 10 possible classes, is it enough to ask log(10) binary questions to know which is the class of the ball ? Sure ???


texts:
A fictional Broadway show has 3 shows every Saturday.  Tickets are valid for a particular show and enumerated seat.   The process of encoding the showtime and seat number is a unique.

Is the process of encoding the notion of a time and seat an example  serialization? enumeration?  If not, what?


texts: Augmenting the Cost Function with a Gaussian Prior

texts:
I'm doing an online course and one of the assignment says:

  To avoid parameters from exploding or becoming highly correlated, it is helpful to augment
  our cost function with a Gaussian prior: this tends to push parameter weights closer to zero, without
  constraining their direction, and often leads to classifiers with better generalization ability.

Exactly what does "augment our cost function with a Gaussian prior" mean, and how could I go about doing that?

I tried searching on google but didn't find anything helpful.


texts: Distractor Generation for Multiple Choice Questions

texts:
I'm currently working on generating distractor for multiple choice questions. Training set consists of question, answer and 3 distractor and I need to predict 3 distractor for test set. I have gone through many research papers regarding this but the problem in my case is unique. Here the problem is the questions and answers are for a comprehension(usually a big passage of text story) but the comprehension based on which is not given nor any supporting text is given for the question. Moreover, the answers and distractor are not a single word but sentences. The research paper I went mostly worked with some kind of support text. Even the SciQ dataset had some supporting text but the problem im working is different
This research paper was the one which I thought closely went by what I wanted and I'm planning to implement this. Below is an excerpt from the paper which the authors say worked better than NN models.

We solve DG as the following ranking problem: Problem.
Given a candidate distractor set D and a MCQ dataset M = {(qi , ai , {di1, ..., dik})} N i=1, where qi is the question stem, ai is the key, Di = {di1...dik} ⊆ D are the distractors associated with qi and ai , find a point-wise ranking function r: (qi , ai , d) → [0, 1] for d ∈ D, such that distractors in Di are ranked higher than those in D − Di.

My questions are a) From what I understood, The above lines says we first create a big list containing all the distractors in the dataset and then we create a pointwise ranking function with respect to all distractors for every question? So if we have n questions and d distractors. We will have a (nxd) matrix where pointwise function values range between o and 1. Also, a question's own distractors should be ranked higher than the rest. Right?

To learn the ranking function, we investigate two types of models: feature-based models and NNbased models.
Feature-based Models: Given a tuple (q, a, d), a feature-based model first transforms it to a feature vector φ(q, a, d) ∈ R d with the function φ. We design the following features for DG, resulting in a 26-dimension feature vector:

Emb Sim. Embedding similarity between q and d and the similarity
between a and d.
POS Sim. Jaccard similarity between a and d’s POS tags.
ED. The edit distance between a and d.
Token Sim. Jaccard similarities between q and d’s tokens, a and d’s tokens, and q and a’s tokens.
Length. a and d’s character and token lengths and the difference
of lengths.
Suffix. The absolute and relative length of a and d’s longest
common suffix.
Freq. Average word frequency in a and d.
Single. Singular/plural consistency of a and d. This
Wiki Sim.

My questions: Will these feature generation idea applies to both word distractors and sentence distractors? ( As per the paper, they claim it will).
Apart from all of these, I have other simple questions such as should I remove stopwords here?
I'm new to NLP. So any suggestions about which SOTA implementation would work here would be very helpful. Thanks in advance.


texts: arima analysis after clustering time series

texts:
I have a database of multivariate time series that I want to cluster in order to find natural grouping of features. I am thinking of taking each cluster points and perform an ARIMA analysis on its points. Within those clusters, what should be the best way to infer those groupings in the original mutltidimentional feature space?
a sample of my time series is :

TimeStamp       | Sensor0 | Sensor1| Sensor2
2015-02-05 11:30|<Min     | On     | off
2015-02-05 11:31|<Min     | on     | off
2015-02-05 11:32| Action2 | 10     | 0.0001
2015-02-07 11:33| Action2 | 10     | 0.00012
2015-02-07 11:34| Action2 | 10     | 0.00012
2015-02-07 11:35| Action2 | 20     | 0.00015

I have roughly 10 separate groupings and I don't think performing different ARIMA calculations would be the best efficient method. Any thoughts or ideas are also welcome!


texts: How does one decide when to use boosting over bagging algorithm?

texts: What kind of problem, circumstances and data makes it more suitable to apply boosting instead of bagging methods?


texts:
I know AlexNet does object classification in images [categories] and R-CNN does object localization [category and bounding box].
How does R-CNN and AlexNet compare?

Are they used for the same purpose or R-CNN does more? Does R-CNN use Alexnet as a sub-module?


texts:
I would like to know the meaning of an autocorrelation graph of a sine wave. When the time lag is 0, then the autocorrelation should give the highest value of 1 since a copy of the signal is completely correlated to itself. By this logic, after a time equal to the period of the signal, the correlation should again be maximum since the shifted signal is again the signal itself. However, when I plot the correlation in python, I get a function which keeps increasing which goes against my intuition that the correlation function should be periodic. Can anyone please explain why the autocorrelation shows this trend in its graphical form?

import matplotlib.pyplot as plt
import numpy as np
time = np.arange(0, 10, 0.1);
y = np.sin(time)
result = np.correlate(y, y, mode='full')
plt.plot(result[:int(result.size/2 )])
plt.show()


texts:
I have a large-ish dataset (100k samples, ~100 features), that I am trying to cluster, to an unknown number of clusters.
I thought of using PCA first, to reduce dimensionality, since I understand that distances (euclidean, cosine) in high-dimensionality might not be actually accurate.

However, I found very little on using PCA before AP, unlike K-Means, which made me wonder - is there any reason to not use PCA and then affinity propagation?


texts: HR employee attrition modeling - making a balanced sample question

texts:
I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- contains demographic data of people who voluntarily left the company.
My aim is to identify within dataset 1 who is at risk of leaving the company.
Question: what would be a good approach to build a training data set?
I am thinking about some kind of train_test_split
My thinking is right now to split stayers (dataset 1) into 8 groups of roughly 180 records large groups
then combining each of these groups individually with the complete dataset 2 (leavers) build a logistic regression
With each of these combinations I do a logistic regression on the remaining stayers data and do a prediction on attrition (yes/no) and then compare all the resulting models
What do you think? Any glaring pitfalls or risks in my approach?


texts:
I have three different modalities, each of which has a shape of (n-users, m-timestep, p-features). I want to do feature level fusion, by adding an attention layer that would compute the scores of every feature PER modality, and then fuse these weights into one feature vector to add to an LSTM for binary classification.
Here is the Paper that applies attention for feature fusion. But I do not fully understand from an implemention perspective how is this possible to implement on keras?
Would I add a standalone self attention layer and return the weights for every group of features in a modality individually, or is there a different way that this is implemented?
If you could guide me to resources to better understand its implementation, I would appreciated. The network I need to apply is as follows:
 modality1: shape[10, 24, 20] --- compute attention scores \
                                                        merge attention
                                                      weights feature vector
                                                          [10, 24, ?]       -> LSTM --> Output 1/0
modality2: shape[10, 24, 12] ---  compute attention scores/


texts: Hypothesis Testing - Median Test for linear regression model given standard error and co-efficient

texts:
I need help approaching this particular problem on hypothesis testing.
A researcher believes that a diet high in raw cornstarch would increase the median blood glucose levels by 20. The diet can either be high in raw cornstarch (have a value of 1) or low in cornstarch (have a value of 0); let's call this feature x. To test the researcher’s hypothesis, say I build a linear regression model with x as a covariate and the coefficient associated with raw cornstarch is 2.76 with a standard error of 0.78. How can I compute if there is evidence to reject the researcher’s claim?
And how can I defend this analysis with someone who has no knowledge of linear models?


texts:
So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should not normalise it. However, isn't this always the case? For example, if we have age and salary, isn't salary always going to be greater than age? Wouldn't normalising these values lose possible significant information?

Also, should you always normalise the entire dataset, or can one simply normalise a number of specific columns?


texts: Looking for suggestions on performing Sementic Analysis of ASR text

texts:
Currently I am working on a project where I have ASR on which I am performing semantic analysis to extract meaning out of it. The ASR text contains huge amount of vague conversational text which needs to be eliminated using some algorithm.
Trial 1: I tried to use NER to find some important entities in text and then I eliminated the sentences which did not have any important entities. It worked for some samples but failed for majority because it couldn't retain meaningful sentences.
Trial 2: Tried summarizing it using Seq2seq model supervised learning, but it was an entire failure. It couldn't even generate proper sentences.
Trial 3: Tried to perform Sementic Analysis of the text, couldn't find a way to perform it successfully.

I need suggestions for something to try and also any help or examples for performing Sementic Analysis for extracting meaning out of it.

Hope anyone would have performed similar research.


texts:
I want to build neural network, where my input will be a word(not a sentence). My set of words has many words with different lenght (number of chars). My idea to do it is transform chars to numbers using predefine dictionary, next use it as input. Number of neurons in input layer set as a number of chars in a longest word in my set. In vectors witch are shorter I'll fill them by zero.

My set of words looks that:

1. head
2. hello
3. butterfly
4. hotel
5. fly
6. spy

So for this set my input to netowrk should has 9 neurons. When I transform 'head', I will get vector with 4 numbers and 5 zeros.

It's a good idea? Or maybe you have another better idea how input of this network could look.

EDIT:

My task is described here: ML model to transform words:
And I want to build GAN as an answer suggest.
So this words will be my 'real words' and I want to generate fake words.


texts: How much data do you need to build a classifier?

texts:
I would like to ask you what a good size of dataset would be for building a classifier.
I know that there are datasets of 1000 obs and datasets of 1m obs. But I also read papers where classifiers were built on datasets of 300 obs.
I think the size may affect the accuracy/precision of a classifier, but I am not sure 100% of that.


texts: Why does a randomly initialised convolution kernel correspond to an edge detector?

texts:
In this nice tutorial about CNNs, the authors build a single-layer CNN. The initial convolution weights are set randomly, according to a uniform distribution.

By the end of this scetion, the authors note that the randomly initialised kernel behaves very similar to an edge detector and give the following input and output as example.

Why does the randomly initialised kernel behave like an edge detector?


texts: Is it possible to implement a vectorized version of a Maxout activation function?

texts:
I want to implement an efficient and vectorized Maxout activation function using python numpy.
Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al).
For example, if k = 2:
def maxout(x, W1, b1, W2, b2):
    return np.maximum(np.dot(W1.T,x) + b1, np.dot(W2.T, x) + b2)

Where x is a N*D matrix.
Suppose k is an arbitrary value(say 5). Is it possible to avoid for loops when calculating each wx + b? I couldn't come up with any vectorized solutions.


texts: When developing machine learning models, is the size of each class in the test set important?

texts:
I am thinking about the prospective application of a trained classifier in a real-world context. We know that when we do over/under-sampling to balance our dataset, we never touch the testing set as we want to keep our dataset's real behaviour. But the part that I do not understand is the role of the test set's distribution in a classifier's performance.
Let's say I have a model that can label an email as spam or non-spam. If I launch this model in my email-service, in a specific time window, all the emails that my classifier receives might be non-spam. But the trained model has a 50-50% distribution for each category. My question is, does this difference in the distribution--during the prospective application-- change the performance of the model? e.g. if my web-service receives 5 spams and 5 non-spams in that time window, should I receive a more accurate classification? Based on my understanding, the answer should be a No. Still, I see everywhere that people are talking about the importance of the test distribution and its role in the performance and accuracy of predictive models.
Thank you.


texts:
Can someone answer this question:

It is from an exercise in the book:
  Mining of massive datasets:

Chapter 3: Finding Similar Itemsets

What is the largest number of k-shingles a document of n bytes can have?

  Exercise 3.2.3 : What is the largest number of k-shingles a document of n bytes can have? You may assume that the size of the alphabet is large enough that the number of possible strings of length k is at least as n.

the answer is (N-k)

I wanted an explanation


texts:
I have used the following code to test the behaviour of CTC loss.
from torch.nn import CTCLoss
import torch.nn.functional as fn
import torch
def get_char_maps (vocab):
    char_to_index={}
    index_to_char={}
    cnt=0
    for c in vocab:
        char_to_index[c]=cnt
        index_to_char[cnt]=c
        cnt+=1
    vocab_size=cnt
    return (char_to_index, index_to_char, vocab_size)

loss_function = CTCLoss()
char_to_index, index_to_char, vocab_size = get_char_maps(['~', 'a', 'b', 'c', 'd', 'e', 'f', 'g', ' '])

empty_char = '~'
label = 'abc'
seq_len = 20
predict = 'abc'
pred_out = predict[0]
for i in range(1, len(predict)):
    if predict[i] == predict[i - 1]:
        pred_out += empty_char + predict[i]
    else:
        pred_out += predict[i]
print(pred_out)
predict = pred_out
label_len = len(label)
left_side = (seq_len - len(predict))//2
right_side = seq_len - left_side - len(predict)
seq_predict = [empty_char]*left_side + list(predict) + [empty_char]*right_side

out = []
for char in seq_predict:
    temp = [0]*vocab_size
    temp[char_to_index[char]] = 1
    out.append([temp])
out = torch.FloatTensor(out)

scores = fn.log_softmax(out, dim=2)
out_size = torch.tensor([seq_len]*1, dtype=torch.int)
y_size = torch.tensor([len(label)], dtype=torch.int)
y = [char_to_index[c] for c in label]
y_var = torch.tensor(y, dtype=torch.int)
l = loss_function(scores, y_var, out_size, y_size)
print(l.item())

With this code, when the target is abc and the prediction is also abc, it produces a loss of 7.770. However with the same target when the prediction is g, it produces a loss of 7.654. Which is lower than the perfect prediction. This behaviour seems counterintuitive.
Is this the expected behaviour from CTC loss? Or is this test code has an error?


texts:
Im Working With Mongodb And Wanted to do a query using Aggregate fucntion. Query Is Each city has several zip codes. Find the city in each state with the most number of zip codes and rank those cities along with the states using the city populations.
The documents are in the following format
{
  "_id": "10280",
  "city": "NEW YORK",
  "state": "NY",
  "pop": 5574,
  "loc": [
    -74.016323,
    40.710537
  ]
}

I was able to count no of Zipcodes for each state using below command
db.zips.aggregate([{"$group":{_id:"$state",count:{$sum:1}}}])
Zips is name of collection
What should be my approach to solve the query?


texts: how to transform a folder of images into csv file?

texts:
I have a data set for segmentation project. The dataset have 2 folders containing train folder and annotated_train_data folder, both folders have images. input and label both are images. I want to make a csv file of this dataset to feed into a neural network. what is the easiest and efficient way to make csv file?

./images
./images/train
./images/train/label1.jpg
./images/train/label2.jpg
./images/train/label3.jpg
./images/annotated_train_data
./images/annotated_train_data/label1.jpg
./images/annotated_train_data/label2.jpg
./images/annotated_train_data/label3.jpg


texts: Time Series Classification for loan data

texts:
I have multiple columns for loan installment repayment. As there is a field for month of repayment, I want to predict if the customer is going to pay next month's installment or not.
As I have multiple variables and target variable as installment paid (Y/N), despite repayment being dependent on time variable, i.e., installments paid in past months, I'm looking to solve this problem with time series classification.
Any references will be appreciated.


texts: For the line $w^Tx = 0$, how do we know the direction of the vector $w$

texts: On page 17/28 of Lecture 3: Linear Classification, why the slope of $w$ must be positive?


texts: Where can I find freely available multi-label datasets online?

texts:
I'm trying to find multi-label classfication datasets, which are available for free online.

By "multi-label" I mean that each instance can be labeled with anywhere from a single to $k$ labels, where $k$ is the total number of different labels in the dataset. Typically all information about the labels would be represented in a binary matrix $\mathbf{M}$, where $\mathbf{M}_{ij}=1$ if instance $i$ has label $j$, and $0$ otherwise.

I've found the following two datasets so far:

iMaterialist Challenge (Fashion) at FGVC5 from Kaggle.com
DeliciousMIL: A Data Set for Multi-Label Multi-Instance Learning with Instance Labels Data Set

I've also looked at the Mulan multi-label datasets page, but they are pretty opaquely (and sometimes erroneously) described.

Where can I find more multi-label datasets (preferably with 20-200 different labels in total)?


texts: Dimensionality reduction and prediction when all columns have approximately same variance

texts:
I have a dataset of 25 columns where the goal is to predict the value of the 25th column based on the previous 24 columns.

The dataset is quite big that's why I initially thought to proceed with PCA before doing any prediction. The problem is that PCA did not produce any good results in the sense that it outputs a 4% variance explained on each PC.
I suspect that this can be attributed to the fact that the variance of all columns is approximately the same (say from 90-91% for each column).

I am wondering what can be done in such a case to reduce dimensionality and what data science algorithms are most suited to address this problem.

I have already tried OLS, Random Forests, SVR and Gradient Boosting regression but their scores seem quite disappointing at the moment, letting aside the fact that the computational time is quite large.


texts: Which Classification Metrics Are Appropriate For Each Class Distribution Scenario?

texts:
Currently, I have a balanced dataset (that I artificially over-sampled to make it balanced). My classes are binary (0 or 1). I'm wondering if "accuracy" is the "best" metric to use in the situation when a dataset has roughly equal class balance?

More broadly though, is there a good "rule-of-thumb" or "best practice" regarding which scoring metric (accuracy, precision, recall, AUC, ...) to use based on the dataset's class label "distribution".

Common scenarios I can think of are:

Binary Classification

high imbalance of 0 class
high imbalance of 1 class
roughly equal class quantities
last, imagine a 75/25 ratio for either

Multiclass Classification

Majority of samples are dominated by one class (very common in my experience)
Classes have roughly equally the same amount
Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)

I'm aware of techniques like under-sampling and over-sampling to deal with imbalance by altering the dataset's number of samples, but let's say you didn't in this scenario. Which metrics are best for the above scenarios?


texts: Is Chi Square goodness parametric or non-parametric?

texts: Does Chi Square goodness of fit require normality assumption? Is it a parameteric or non-parametric test? What it its relation with t-test (parametric test) and u-test (non-parametric test)?


texts:
I have the dataframe which has two colums(Reviews and Label):

  Reviews                                     Label
0 [cameron, diaz, woman, marri, judg, play]   1
1 [turgid, dialogu, feebl, character, harvey] 0
2 [misfortun, watch, rubbish, sky, cinema]    1

I want to apply the TfidfVectorizer on the DF.
I have written the following code.

from sklearn.feature_extraction.text import TfidfVectorizer
df_x=train_df["Reviews"]
df_y=train_df["Label"]
cv = TfidfVectorizer()
df_xcv = cv.fit_transform(df_x)
a=df_xcv.toarray()
cv.get_feature_names()

which is giving an error:

  AttributeError: 'list' object has no attribute 'lower'

Why is this throwing an error?


texts: additional of features decrease the accuracy of the model

texts:
I am using sklearn's random forests module to predict a binary target variable based on 166 features. When I increase the number of dimensions to 175 the accuracy of the model decreases (from accuracy = 0.86 to 0.81 and from recall = 0.37 to 0.32) .

I would expect more data to only make the model more accurate, especially when the added features were with business value.

I built the model using sklearn in python. Why the new features did not get weight 0 and left the accuracy as it was ?


texts: neural network function approximation with constraints

texts:
I would like to approximate a function $f(\cdot)$ by means of a neural network given a finite set of observations $f(x_i)$ where $x_i\in\mathbb{R}^n$ and $i=1\dots,N$. However, I have some prior knowledge on how this function should behave, for example that it is monotonic in the first coordinate.

Are there methodologies accounting for this type of shape constraints when training a (D)NN?


texts: Why do we double the number in a quadratic cost function or MSE?

texts:
$$ C(w,b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $$

Where y is a 10-dimensional vector, a is the output, w is the weight and b is the bias and n is the number of inputs. If this is the MSE, shouldn't it be $\frac{1}{n}$ instead?

Link


texts:
I have to compose a feature which summarizes the blocks area of different sections of cities. (A block is defined as the space contained by streets).

I could compute the arithmetic average of areas, but the data is often very skewed in the sections. I could compute the arithmetic average of log(areas), which transform the data to a more normal distribution, but the problem is that the data is skewed differently section by section.
How would you create this feature?


texts:
I have a csv file with characters in Persian and I cannot view them in R correctly. Also, I cannot subset based on Persian characters values. Here is a sample code:

list1 <- c("x","y")
list2 <- c("ب","الف")

list1
list2 ##OK-readable

writedf <- as.data.frame(cbind(list1,list2),encoding="UTF-8")
write.csv(writedf,"test.csv")
testdf <- read.csv("test.csv",encoding="UTF-8")

testdf
## not readable

testdf[,testdf$list2=="ب"]
## data frame with 0 columns and 2 rows  ???


texts: Neural networks - Find most similar images

texts:
I am working with Python, scikit-learn and keras. I have 3000 thousands images of front-faced watches like the following ones:
Watch_1, Watch_2, Watch_3.

I want to write a program which receives as an input a photo of a real watch which maybe taken under less ideal conditions than the photos above (different background colour, darker lightning etc) and find the most similar watches among the 3000 ones to it. By similarity I mean that if I give as an input a photo of a round, brown watch with thin lace then I expect as an output watches of round shape, of dark colour and with thin lace.

What is the most efficient machine learning algorithm to do this?

For example, by following this link I have two different solutions in my mind:

1) Using a CNN as a feature extractor and compare the distances between the these features for every pair of images with reference to the input image.

2) Using two CNNs in a Siamese Neural Network to compare the images.

Are these two options the best ones for this task or would you suggest something else?

Do you know any pre-trained neural network (with pre-determined hyperparameters) for this task?

I have found some interesting posts on StackOverflow about this but they are pretty old: Post_1, Post_2, Post_3.


texts: Difference between PCA and regularisation

texts:
Currently, I am confusing about PCA and regularisation.
I wonder what is the difference between PCA and regularisation: particularly lasso (L1) regression?
Seems both of them can do the feature selection. I have to admit, I am not quiet familiar with the difference between dimensional reduction and feature selection.


texts:
After conducting a cluster analysis using K-means, I have new data coming online that I need to detect anomalies with.  Anomalies are assumed to not be within the clusters.

So, how is one to define "inside a cluster" in K-means?


texts: What is difference between Fully Connected layer and Bilinear layer in CNN?

texts: What is the difference between Fully Connected layers and Bilinear layers in deep learning?


texts: On Noise Contrastive Estimation, replace noise distribution with difficult examples

texts:
In noise contrastive estimation, we learn a binary classifier to classify noise from the true distribution.

The problem I'm trying to solve has discrete input variables yet it's intractable to sum over. I want to approximate the true distribution by using NCE.

The Question is: is it okay to draw difficult examples (noise samples)
 the model is likely to misclassify, instead of sampling naive noise examples.

From a manifold perspective I want to learn the structure of the manifold, I know for each data point a group of data points that are for sure out of the manifold.
So I directly replace the noise distribution by this (difficult examples distribution) is still valid? Am I still doing NCE?


texts:
First off, I don't really know much about machine learning.

In a virtual world, such as a video game like minecraft or an application like Google Street view, a user can navigate the world using the keyboard and mouse.

I'm looking to predict what interaction the user will perform. For example, the user might hold down w for some period of time (to move forward) and then press and hold a for a period of time to move left. Alternatively, the user may hold w and use the mouse to turn and look up/down.

I've looked into N-Grams, and I can predict interaction to a certain degree, but I am not considering how long a user is not performing an action. I think performing no action is an important indicator for correct interaction prediction as "no action" could itself be classified as an interaction. The duration of an interaction should also be important, but again, my not sure how to consider that in a model. My Ngram model simply considers the past 3 or 5 inputs.

I'd like to develop something which is very fast to predict interaction  (this is key) and learns rapidly from mistakes. Ideally, it would be "Online" in that it constantly "Learns".

I'd be interested in hearing which models you think would be well suited to my task (are ngrams appropriate?), any examples which closely match my task and I can therefore adopt to suit my needs, any datasets containing user interaction and what I should look at to predict the mouse interaction.


texts: Getting the leave-one-out error on least square regression to fit polynomials

texts:
I need to implement least square regression to fit polynomials of degree 1-27. I then need to get the leave-one-out error (kfold cross validation where k = n). After doing a lot of research it seems the best way to get the LOO error is to use sklearn cross_val_score(). My problem is I do not know how or if it is possible to use with regression models.

n = len(x)
p, res, _, _, _  = numpy.polyfit(x,y,1,full=True)

cv = cross_val_score(?, X, y, scoring=mse, cv=n)

I cannot figure out what the estimator would be or how to make it in cross_val_score. Being new to python and these topics make this twice as challenging.


texts: Neural Networks for Unsupervised Learning

texts:
Why cannot we use neural networks for unsupervised learning problem.
I do know that it can be used using the Kohenon’s Self Organizing Map (KSOM) but is this the only method that we can use or are there any other.


texts: Sentiment Analysis of comments to understand support on a topic

texts:
I have downloaded comments from a website which asked people whether they supported or opposed the implementation of a certain political policy related to immigration.

I would like to get any resources or ideas on how to extract aggregate support/opposition to this policy.

In particular, I need a method that correctly identifies that all of the following comments are "anti-immigration" in some senses such as:

We need to give Americans first priority in the job market.
We should not let Americans suffer more unemployment on account of immigrants.
Immigrants should not be allowed to take American jobs.

Similarly the method should be able to identify "pro-immigration" comments, such as:

Providing this service to immigrants will be good for the economy.
The American economy will suffer if immigrants aren't allowed to continue to work here.
I do not think that passing this law will be detrimental to American jobs.


texts: How do you test your results for overfitting in a k-means run? Some people have said use a training set. I have about 1500 records and about 20 fields.


texts: Deploying ML/Deep Learning on AWS Lambda for Long-Running Training, not just Inference

texts:
Serverless technology can be used to deploy ML models to production, since the deployment package sizes can be compressed if too large (or built from source with unneeded dependencies stripped).
But there is also the use case of deploying ML for training, not just inference. For example, if a company wanted to allow power users to retrain a model from the front-end.
Is this feasible for Lambda given the long training times?
Whereas latency wouldn't be issue (cold start delay is fine) the runtime could be fairly long (hours).


texts: "Deep Noether's Theorem": Building in Symmetry Constraints

texts:
If I have a learning problem that should have an inherent symmetry, is there a way to subject my learning problem to a symmetry constraint to enhance learning?

For example, if I am doing image recognition, I might want 2D rotational symmetry. Meaning that the rotated version of an image should get the same result as the original.

Or if I am learning to play tic-tac-toe, then rotating by 90deg should yield the same game play.

Has any research been done on this?


texts: A multivariate linear regression for explaining impacts of the predictors

texts:
I am trying to build a multivariate linear regression and the main goal is to understand how the various features impact the response by understanding the coefficients and their confidence intervals.

For this reason, I chose multivariate linear regression because the coefficients are intuitive to interpret and from the standard error and the degrees of freedom, I can get the 95% confidence interval of the coefficients. Therefore, I can tell what the impact of a unit increase of a predictor on the outcome is. I could use more complex models such as tree based models but, even if I can get the variable importance, it is not easy to quantify the coefficient of each variable.

My problem, though, is the data is time series data and hence it shows auto-correlation. I know using regression with ARIMA errors can handle the autocorrelation issue but I found it difficult to interpret the coefficients specially when d is non-zero in ARIMA(p, d, q). The attached image, for example, is model diagnosis for one of the models I have (I have to build 1000s of them).

How can I handle the autocorrelation issue and still get coefficients that are readily interpretable like in a multivariate linear regression? My residuals are not normally distributed and I am planning to use box-cox transform to see if it can solve that problem. But, I am not sure what I should use for the autocorrelation issue.


texts: Why does balancing the test dataset improve precision-recall curve?

texts:
I have a fairly imbalanced dataset for default-risk credit scoring (2:98). Both costs are fairly important i.e False negative means loss from default and false positive is a lost-revenue opportunity.

I tried training some models on the imbalanced set and tried over/under-sampling approaches as well as weight-balancing based on class. All of these approaches led to a decent ROCAUC. However, the precision-recall curves have looked horrible overall.

But if I were to balance the test set, the precision-recall curve looks much better. Why does this happen? And is there some adjustment I should be applying to improve precision-recall under the imbalanced data distribution?


texts: Why Tensorflow does NOT quit when CUDA_ERROR_OUT_OF_MEMORY

texts:
It was my understanding that when trying to create a tensorflow session, if the amount of available GPU memory is not enough for what the program asks for, the program should raise an error and exit. But it seems to be not this case, and this behavior is very annoying and hard to detect especially when using Jupyter Notebook.

Code

I was trying to run the following program (supposedly debug.py)

import tensorflow as tf
a1 = tf.constant([0.0,0.0])
b1 = tf.constant([2.0,2.0])

mse1, update1 = tf.metrics.mean_squared_error(a1,b1)
mse2 = tf.losses.mean_squared_error(a1,b1)
with tf.Session() as sess:
    sess.run(tf.local_variables_initializer())
    update1.eval()
    print(mse1.eval())
    print(mse2.eval())

This program simply calculate the mean_squared_error (MSE) between the two vectors a1 and b1, and the correct output (from the two print() lines) should be 4.0 and 4.0. If no other python programs are using my GPU, this is indeed the output.

Unexpected behavior

If another program is using the GPU (say, another jupyter notebook running something with tensorflow without limiting its GPU usage by gpu_options.per_process_gpu_memory_fraction), then the above code would output something different. The first print line (mse1) would output some random number, e.g. "0.0" (most common), "3e-45", "nan", etc, which is obviously wrong. The second print line (mse2) always print the correct answer: 4.0.

Running this code by python debug.py SOMETIMES generate an error message

  E tensorflow/stream_executor/cuda/cuda_driver.cc:924] failed to allocate 176.44M (185008128 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

But the program does NOT exit, and still prints the results, although incorrect.

In other cases, running this code by python debug.py  generate another error message

  E tensorflow/core/common_runtime/direct_session.cc:170] Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY; total memory reported: 8502771712

and the program quits without printing any output.

The most annoying thing is, running the same code in Jupyte Notebook ALWAYS generates no error or warning messages, though the output is incorrect. That's why I always thought I misunderstood about how to call tf.metrics.mean_squared_error without realizing there could be a memory error.

System

I verified the problem with my two computers running Tensorflow 1.3 and 1.4 respectively, both with CUDA 8, Ubuntu 16.04 and Nvidia GPUs (1080 and 770M).

My Questions

(0) Is there anything wrong with my code?

(1) If you also use Tensorflow with Jupyter Notebook, do you ever get the wrong output (printing anything other than 4.0, 4.0) when running the above code? If so, does Jupyter alerts you about any error?

(2) What is the nature of this problem of returning the wrong mse1? Is it a GPU memory leak?

(3) Why could there be such inconsistent behavior when the problem occurs? To be specific, why the debug.py sometimes quits, sometimes not? Why Jupyter Notebook never quits or raises an error?

(4) Why the mse2 never goes wrong when mse1 goes wrong? My understanding is that tf.metrics.mean_squared_error is a streaming operation and therefore need to store some variable in a different place.


texts: What is the hypothesis space used by this AND gate Perceptron?

texts:
Per this post

  The hypothesis space used by a machine learning system is the set of
  all hypotheses that might possibly be returned by it.

Per this post, the Perceptron algorithm makes prediction

$$
\begin{equation}
  \hat y =
    \begin{cases}
      1 & wx+b >= 0\\
      0 & wx+b<0
    \end{cases}
\end{equation}
$$

  we can conclude that the model to achieve an AND gate, using the
  Perceptron algorithm is

$x_1 + x_2 – 1.5$

In this case, what is the hypothesis space used by this AND gate Perceptron?


texts: Active learning with Fisher information matrix for regression with vector output

texts:
I am trying to enhance the learning capabilities of my regression task with the help of active learning. I  decided to use variance reduction using the Fisher information matrix as a query strategy as explained in Settles' popular survey "Active learning literature survey, 2010" (https://www.cs.cmu.edu/~10701/slides/Settles_notes.pdf). According to it the output variance of an input x can be approximated by

where F is the Fisher information matrix.
It appears like in this equation y is expected to be a scalar output. I am working with a neural network that has 3 output nodes and I am not sure if the equation can still be applied without modification. Of course I would get a covariance matrix instead of a scalar variance.
Unfortunately I could not find any literature regarding active learning using the Fisher information matrix for regression tasks with multiple outputs.
Can I still use this equation for estimating the output variance of my inputs? Is there any literature that goes into more detail regarding my problem?


texts: Recommendation Systems User Profile Streaming Data on GCP

texts:
I have a recommendation system that recommends articles to different users. I am planning to provide the recommendations in an off-line fashion. Where I already have a table in BigQuery which has the recommendations and an API call returns the recommendations for each page on the website.
Now I want to have another table called user_profile which stores the information about the user_id|shown|clicked| articles to the users. This should happen in real-time. I looked into https://cloud.google.com/bigquery/streaming-data-into-bigquery but it has limitations. I cannot write multiple rows into the table at once. Let's say I have 5000 unique users looking at an article once.
Any other suggestion for updating the user_profile table or am I missing something?


texts: Population stability Index vs Population Accuracy Index

texts: Can anyone explain to me the difference between Population Stability Index(PSI) and Population Accuracy Index(PAI)?


texts: How can i find the accurcy for time series?

texts:
I have to work for the first time with time series and I have some question about this interesting field of  machine learning.
What I have to do:
I should make a forecast for quantity for some special good in the future. I should predict, that this good is available e.g. in next week in our shop or not. My Methods are Prophet and ARIMA and fitting method by ARIMA is bfgs
My Questions:

1- How can is find the accuracy of my forecasting? Does something like R2-Score for regression but for time series in scikit-learn?

2- Shall I calculate the accuracy with RMSE? I mean that should I first calculate the MSE and after that sqrt(MSE)?

3- If the answer of 2 is YES, how should be the RMSE for a good forecasting? Big or Small?

4- How can I compare the accuracy of both models together?

5- Which method is suitable for small data (400-800) univariable?


texts: Can we use DecisionTreeClassifier of sklearn for continuous target variable?

texts:
I have a continuous target variable named "quality" which ranges from 0 to 10. Also I have 11 input variables in my dataset.

When I'm building my model using DecisionTreeClassifier() of sklearn then I'm getting a score of 60% but when I'm building my model using DecisionTreeRegressor() of sklearn then I'm getting accuracy of 3% only and also RMSE as 85%.

Also, when using Linear Regression my R-squared value is 0.376. Is it good?

Dataset Link : https://archive.ics.uci.edu/ml/datasets/Wine+Quality

Am I doing something wrong?

I need help. Thank you.


texts:
Can anyone please explain what a PDF and CDF are in simple words.

(Please don't define it from wiki.)


texts: Python3 Tensorflow: Difference Between truncated_normal and uniform_normal when creating weights

texts:
So, I am new to using Tensorflow, and I am trying to create a neutral network, and in all the guides I saw one of the below lines of code was used.

Weight1 = tf.Variable(tf.truncated_normal([-,-]))

or...

Weight1 = tf.Variable(tf.random_uniform([-,-]))

Both of them work in the examples, but I want to know is what is the difference between them and is one better for certain situations than the other?


texts: Can tuning individual precision and recall classification thresholds improve deep learning models?

texts:
I learned that Keras doesn't have a built-in way to set a threshold for precision and accuracy when building a classifier.

Courtesy of a solution here, I wanted to see what would happen when I fit a simple 3 layer Multilayer Perceptron with different classification thresholds for these metrics.

So I went ahead and fit a model with as such:

model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics = [precision_threshold(0.5), recall_threshold(0.5)])

and got out these results:

val_precision: 0.7048 - val_recall: 0.5795

Further, I plotted a normalized confusion matrix. And using this:

def precision(cm = confusion_matrix):
    return round(cm[1,1] / (cm[0,1] + cm[1,1]), 4)

def recall(cm = confusion_matrix):
    return round(cm[1,1] / (cm[1,0] + cm[1,1]), 4)

I calculated the resulting precision and recall:

Normalized confusion matrix
[[ 0.6026  0.3974]
 [ 0.2204  0.7796]]

Precision: 0.71
Recall:    0.78

This seems pretty reasonable so far.

Then I switched up the thresholds and looked at what outcomes resulted. The table below shows what I got when I changed the precision threshold to 0.1 and the recall threshold to 0.9:

|-----------------------------------------------------|
| paramter settings  for model fitting:               |
| [precision_threshold(0.1)    recall_threshold(0.9)] |
|                                                     |
| validation results                                  |
|- val_precision  0.4089     - val_recall 0.0457      |
|                                                     |
| Normalized confusion matrix                         |
| [[ 0.6148  0.3852]                                  |
| [ 0.2303  0.7697]]                                  |
|                                                     |
| Precision   0.70                                    |
| Recall      0.79                                    |
|-----------------------------------------------------|

Interestingly, if you look at the final Precision and Recall scores, I notice that they didn't change, even with the extreme parameters I set. Basically the same confusion matrix as well.

I repeated this for a large set of different settings and got generally the same final Precision and Recall scores. I could past them in here but it would take up a lot of space.

One more detail then I'll switch to my actual question.

My model is based on the business case where a large precision value is very desirable and recall is not relevant.

My question is this -- given that I desire large precision -- is tuning the specific precision and recall thresholds for the model valuable at all, if the final Precision and Recall scores are left fundamentally unchanged across all configurations of the thresholds?

I hope this makes sense. Thanks for reading this.


texts: Forward and backward pass in Conv2D transpose Layer

texts:
I’ve several questions regarding the transposed convolution 2d layer. I’ve not been able to find a proper resource explaining the forward and backward pass. What I know (but not for sure) is, that this layer can be emulated by adding stride - 1 zeros between the input pixels and kernel_size - 1 zeros to the input’s border. Doing so, causes the deltas for the original input (before fancy padding) do not match.

Could someone tell me in details how to handle this or a different approach to implement forward and backward pass for this layer type.


texts: Object Detection - How to evaluate saved model weights?

texts:
I've recently completed training an object detection model but only printed out the losses for each epoch. Since I have the saved model weights, how would I go about evaluating the accuracy/precision?
Here's how I've gone about training:
import torch
import matplotlib.pyplot as plt

from detecto import core

training = core.Dataset(train/"")

validation = core.Dataset(val/"")

model = core.Model(["Frame"], device=device)

losses = model.fit(training, validation, epochs=10, learning_rate=0.005)

model.save('model_weights.pth')

print(losses)

plt.plot(losses)
plt.xlabel('Learning step')
plt.ylabel('Loss')
plt.show()

Is it possible to do something with the sklearn classification report? I only have one class though


texts: sklearn SimpleImputer using mean from different column groups?

texts:
I'm looking at the SimpleImputer, in particular in here, and I would like to do the imputation on different columns. My data has 3 different sample groups, and I would like to do mean imputation by sample group rather than column-wise as done with SimpleImputer. My end goal is to then compute MSE through crossvalidation in the same way it is done here. Is this possible?
I could impute values using standard dataframe manipulation, but don't know how to then use it for crossvalidation.
Thanks in advance


texts: Deep Neural Network and Activation Function

texts:
I want to know why we need Activation function in DNN hidden layers.
I know a bit, like it will help us in,

Increasing model complexity and introduce non-linearity
Avoiding Gradient Vanishing  problem.

Still I'm not able to map its requirement.
An illustrative example will be definitely helpful.