texts:
I am using SGD matrix factorisation (python) using the movielens dataset to make recommendations. I have a website which allows users to give feedback which is positive or negative to whether an item is a good recommendation for a particular movie.

I was wondering if I could use this feedback in my matrix factorisation. I wasn't 100% sure how I would proceed.

So for example I would have a vector like

  m1 m2 m3
m1 0 0  0
m2 5 0 -4
m3 0 0  0

Where m2-m1 is a score of 5 so with an example case where noone rated it negative, 5 people think its good. With the other for m2-m3 being -4 so a poor recommendation.

Any help would be greatly appreciated.

Edit: response from answer

I am currently using a sparse matrix for user item ratings and am using bias.

I have been trying to add additional input sources and am using the following to create a attribute matrix containing a genre representation:

for genre in item_genres:
       genres[genre] = 1
       self.attribute_item_matrix[i] = sum(list(genres.values()))

 pred = self.global_mean + self.bias_user[u] + self.bias_item[i] +np.dot(self.P[u, :],(self.Q[:, i]+self.attribute_item_matrix[i]))

However, I don't think this is correct as it eventually fails and causes a nan error.

I have been following this paper Matrix Factorisation Techniques for recommender systems

Thanks


texts: Python: Unable to install pandas_profiling

texts:
I have tried installing profiling pandas profiling library with the following command from the terminal:

conda install pandas-profiling

Output:

ModuleNotFoundError: No module named 'pandas_profiling'

What am I doing wrong?


texts: Unexpectedly long computation times with nested cross validation

texts:
Hello StackExchange community,
I am trying to apply Nested Cross Validation on a pipeline to get a reliable estimate of the generalization error of my model. The pipeline includes two steps:

Scaling the data
Building a logistic regression model.

The issue I am experiencing is as follows:
When I run code below with scaling as part of the pipeline, the first outer loop runs quickly, i.e. prints the results in a maximum of 1-2 minutes. However, something gets stuck on the second outer loop. I have tried this various times, and the outer loop runs for hours and still does not print any results. I am unclear on why.
To evaluate the code and pipeline, when I try without scaling, the entire nested cross validation procedure runs within minutes on my computer.
I have pasted my code below, but please note I have replaced a reference to my actual dataset with a synthetic dataset, as the actual dataset is proprietary. I do not experience any runtime issues on the synthetic dataset, but I suspect that this is because the make_classification code already produces scaled data.
Any advice that anyone could provide would be much appreciated. I know NCV is much more computationally expensive, but I am confused why:

It runs relatively quickly without feature scaling and
It runs the first outer loop iteration with scaling relatively quickly, but the second (and presumably subsequent ones) for an eternity.

Thank you!
#Importing libraries
from numpy import mean
from numpy import std
import sklearn.metrics as metrics
from sklearn.datasets import make_classification
from sklearn.model_selection import KFold
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import make_classification
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.model_selection import StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X_combined, Y = make_classification(n_samples=1000, n_features=20, random_state=1, n_informative=10, n_redundant=10)

#Setting up the outer cross validation procedure
cv_outer = KFold(n_splits=2, shuffle=True, random_state=1)
outer_results_f1 = list()

for train_ix, test_ix in cv_outer.split(X_combined):
    #Split data
    X_train, X_test = X_combined[train_ix, :], X_combined[test_ix, :]
    y_train, y_test = Y[train_ix], Y[test_ix]

    #Setting up the inner cross validation procedure
    cv_inner = RepeatedStratifiedKFold(n_splits=2, n_repeats=1, random_state=1)

    #Initializing the model
    model = LogisticRegression(solver = 'liblinear', random_state=1, max_iter=3000)

    #Building a pipeline that includes scaling the data

    #NOTE - THIS IS WHERE I THINK THE PROBLEM IS OCCURRING
    #The entire procedure runs quickly if I take out the "scaling" part
    lr_pipe = Pipeline([('scaler',  StandardScaler()), ('LR', model)])

    param_set = [{'LR__penalty': ['l1','l2'],
                  'LR__C': [100000, 10000, 1000, 100, 10, 1]}]

    #Define and conduct the grid search
    search = GridSearchCV(lr_pipe, param_set, scoring='f1', cv=cv_inner, refit=True)

    #Get the best performing model & evaluate on the whole training set
    result = search.fit(X_train, y_train)
    best_model = result.best_estimator_

    #Evaluate model on the hold out dataset
    yhat = best_model.predict(X_test)

    #Calculating the F1 score
    f1 = f1_score(y_test, yhat)
    outer_results_f1.append(f1)
    print('>F1=%.3f, est=%.3f, cfg=%s' % (f1, result.best_score_, result.best_params_))


texts: How to use fine tuning of BERT when i have unlabelled dataset of text documents?

texts: I have gained a basic understanding of using BERT for various NLP/text mining tasks. When it comes to fine-tuning of BERT, I always see that fine-tuning is performed using some classification tasks. So, how should I refine the word/sentence embeddings vector given by the BERT model in the case when I have a set completely unlabelled set of documents? I'm aware that the BERT model is originally trained on unlabelled data, so there must be some way.


texts: Score for Ranked plots using scatter plot widget

texts:
I'm using orange 3 and when I Score plots using the scatter plot widget how do I view the numerical score associated with the ordered list of plots. Did that feature get removed with the upgrade from 2.7 to 3?

Thanks,


texts:
In machine learning, an overfitted model fits training set very well but cannot generalize to new instances. I evaluated my model using cross-validation and my accuracy drops when setting the maximum number of splits of my decision tree beyond a certain number. Can this be associated with overfitting?

UPDATE
Here is the result. I am doing Human Activity Recognition in MATLAB with 16 features.


texts: Something is disastrously wrong with my neural network and what it's produced

texts:
I just got a neural network to run and although it doesn't raise any exceptions, I'm left with a horrible mess after 80 to 100 epochs:

After 100 epochs with the adapted code:

After 100 epochs with the original code:

I am trying to generate a grid of synthetic images of cats from my own database of cat photos that I compiled using a crawler. I am using an adapted code originally intended for the MNSIT handwritten digits database (hence the shape of the grid).

The network doesn't appear to be training, generating or discriminating properly because the epochs aren't taking long at all and what is being produced is very poor.

To be clear, I've tried to adapt another author's code that I found online and I've added other snippets of code to try to get it to work. It's evident that my 'FrankenNet' has fallen to bits and my 'bolt it together and see what happens' approach has its limitations.

Maybe I haven't loaded in the data correctly or perhaps there are a few other issues such as converting the data to a numpy array?

I'd love some advice because I really want to generate something and I've spent a long time trying to work it out through trial and error with no results. I'd especially appreciate some specific suggestions about what lines I need to change, add or remove to get this beast up to scratch.

Firstly, I have removed keras.datasets because I am not working with their database.

Original code:

from keras.layers.advanced_activations import LeakyReLU
from keras.datasets import mnist

Adapted GAN:

from keras.layers.advanced_activations import LeakyReLU
from keras.optimizers import Adam

This is how the data is loaded from MNIST in the original GAN:

def load_minst_data():
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train = (x_train.astype(np.float32) - 127.5)/127.5
    x_train = x_train.reshape(60000, 784)
    return (x_train, y_train, x_test, y_test)

And this is what I have replaced it with to try to load and train my own JPGs:

from os import listdir
from PIL import Image as PImage

def loadImages(path):
    # return array of images
    imagesList = listdir(path)
    loadedImages = []
    for image in imagesList:
        img = PImage.open(path + image)
        loadedImages.append(img)
    return loadedImages

DATASET_NAME = 'catscats'
ROOT_DIR = '/Users/Darren/desktop'
DATASET_DIR = f'{ROOT_DIR}/{DATASET_NAME}'
 input_files = [os.path.join(dp, f) for dp, dn, fn in
 os.walk(os.path.expanduser(f'{DATASET_DIR}/processed')) for f in fn
               if f != '.DS_Store']
imgs = np.ndarray(shape=(len(input_files), 100, 100, 3),
                  dtype=np.int)
for i, input_file in enumerate(input_files):
    # print('processing file: {}'.format(input_file))
    image = imread(input_file)
    imgs[i] = image
    # your images in an array
    imgs = loadImages(path)

 def loadImages():
    (x_train, y_train), (x_test, y_test) = input_file
    x_train = (x_train.astype(np.float32) - 127.5) / 127.5
    x_train = x_train.reshape(60000, 784)
    return (x_train, y_train, x_test, y_test)


texts: Why LSTM models do not require labels for each step?

texts:
For time related problems like, for example, stock prediction:

Let's say we have 300 days of data, 10 features, and one target: the price.

Why, for the training, we only need the price of the 300th day?
I know this is the way LSTM models work, but wouldn't it be useful to take into account the price of the 299 other days for the model?


texts: How to optimize for time correlated hidden function - the magical candy machine

texts:
Let's assume that we have this magical candy machine which takes candies as input and delivers again candies as output.

For any given time t, it picks a random function which is strictly
increasing up to a point such as f(2) = 6 here, and then it strictly
decreases. It likes to be challenging but if you become greedy it
punishes you like most of the stuff in life.

  f(1) = 5

  f(2) = 6

  f(3) = 4

  f(4) = 2

  f(100) = 0

The tricky point is that this function is changing all the time, but
still highly correlated with time. So f() will be similar between
t(1) and t(2), but very different between t(1) and t(100).

I want to write a program to maximize my candies using this magical candy machine. I know the fundamentals of ML but I'm not sure which approach would fit best here. Any ideas?

Note: You can only play only once every minute.


texts: How to decide whether to use categorical embeddings in a neural network?

texts:
I have a binary classification task with a whole slew of binary categorical features, one multiclass categorical and a few continuous features. I initially treated the categorical features using one-hot encoding, but am encountering some problems with sparse data due to the one-hot encoding.

I've been reading up on categorical embeddings and was wondering if there was any benefit to use this for the binary categorical features. I couldn't find any clear cut reasoning why or why not to use them, but in this notebook, Deep Learning for tabular data using Pytorch, she has a condition to only use an embedding if the number of categories is greater than 2 (see cell 18):

#categorical embedding for columns having more than two values
embedded_cols = {n: len(col.cat.categories) for n,col in X.items() if len(col.cat.categories) > 2}
embedded_cols

Also, I noticed here Classify structured data with feature columns, they state that "Key Point: using an embedding column is best when a categorical column has many possible values."

Is there any rational for this decision? Any reason one could not use an embedding for a binary categorical feature, especially if I were to concatenate some of my binary categorical features together?


texts: How to transform features order to make their distribution normal?

texts:
After sns.pairplot(df) command, I got this picture:

The question is what transformation to use in order to make the distributions normal?


texts: How to increase sales and revenue of a Client?

texts:
I was asked this in an interview for a Data Scientist position:
Lets say Holland and Barret came to you and said they'd like to increase their sales and revenue. How will you go about it?
My answer wasn't hitting the mark or touching the points the interviewer was looking for. How to go about answering this?


texts: what are the differences between feature weighting and feature selection? And is feature importance like feature weighting?


texts: how to calculate the cosine similarity between two files?

texts:
I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a string and then calculate the similarity.
code
  val lines=Source.fromURL(Source.getClass().getResource("file:///usr/loca/spark/dataset/algorithm3/comedy")).mkString("\n")

      val lines2=Source.fromURL(Source.getClass().getResource("file:///usr/local/spark/dataset/algorithm3/funny")).mkString("\n")

   val result=textCosine(lines,lines2)
   println("The cosine similarity score: "+result)
  }

  def module(vec:Vector[Double]): Double ={
    math.sqrt(vec.map(math.pow(_,2)).sum)
  }

  def innerProduct(v1:Vector[Double],v2:Vector[Double]): Double ={
    val listBuffer=ListBuffer[Double]()
    for(i<- 0 until v1.length; j<- 0 until v2.length;if i==j){
      if(i==j){
        listBuffer.append( v1(i)*v2(j) )
      }
    }
    listBuffer.sum
  }

  def cosvec(v1:Vector[Double],v2:Vector[Double]):Double ={
    val cos=innerProduct(v1,v2) / (module(v1)* module(v2))
    if (cos <= 1) cos else 1.0
  }

  def textCosine(lines:String,lines2:String):Double={
         val set=mutable.Set[Char]()
    lines.foreach(set +=_)
    lines2.foreach(set +=_)
    println(set)
    val ints1: Vector[Double] = set.toList.sorted.map(ch => {
      lines.count(s => s == ch).toDouble
    }).toVector
    println("===ints1: "+ints1)
    val ints2: Vector[Double] = set.toList.sorted.map(ch => {
      lines2.count(s => s == ch).toDouble
    }).toVector
    println("===ints2: "+ints2)
    cosvec(ints1,ints2)
  }

}

but the output gives me error:
> Exception in thread "main" java.lang.NullPointerException
    at scala.io.Source$.fromURL(Source.scala:141)
    at com.algorithm.similarity$.main(similarity.scala:18)
    at com.algorithm.similarity.main(similarity.scala)

How to solve it?


texts: Model not learning in Cat-Dog Image classification using Keras

texts:
def CNN(cat,dog,test):

    cat=np.reshape(cat,(cat.shape[0],64,64,3))
    dog=np.reshape(dog,(dog.shape[0],64,64,3))
    xtrain=np.concatenate((cat,dog),0)
    ytrain=np.asarray([1 if i<len(cat) else 0 for i in range(len(cat)+len(dog))])
    xtest=np.reshape(test,(test.shape[0],64,64,3))

    indices=np.arange(xtrain.shape[0])
    np.random.shuffle(indices)
    xtrain=xtrain[indices]
    ytrain=ytrain[indices]

    model=Sequential()
    model.add(Conv2D(32,kernel_size=(3,3),activation="sigmoid",input_shape=(64,64,3)))
    model.add(MaxPool2D(pool_size=(2, 2)))
    #model.add(BatchNormalization())
    #model.add(Dropout(rate=0.25))
    model.add(Conv2D(32,kernel_size=(3,3),activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    #model.add(BatchNormalization())
    #model.add(Dropout(rate=0.25))
    model.add(Flatten())
    model.add(Dense(128,activation="relu"))
    model.add(Dense(1,activation="sigmoid"))
    sgd = optimizers.SGD(lr=0.1)
    model.compile(loss="binary_crossentropy",optimizer='adam',metrics=['accuracy'])
    print(model.summary())

    #print(ytrain)
    #print(xtrain[0].mean())

    model.fit(xtrain,ytrain, batch_size=32, epochs=10, shuffle=1)

    #ynew=model.

    outcomes=model.predict(xtest)
    ytest=[np.argmax(i) for i in outcomes] #change to round-off

    return (outcomes,ytest)

After training, I am getting accuracy of around 55-59 % but it is expected to go above 90% easily.

I think I have made a very silly error with the model or with the data formatting. Can you please help me point the mistake.


texts: Layers' function in a convolutional neural networks

texts:
From Wang et al (2015) "Visual Tracking with Fully Convolutional Networks":

  A top layer encodes more semantic features and serves as a category
  detector, while a lower layer carries more discriminative information
  and can better separate the target from distracters with similar
  features.

I don't quite understand this statement: does this mean that the algorithm first identifies the semantic group to which the object belongs and then tries to distinguish among them? Also, are there any other publications where the difference in the roles of layers been demonstrated?


texts: How to extract numerical data from a matplotlib scatter plot in python?

texts:
I have a scatter plot with about 19,000 data points. By visual inspection, I noticed some points for which I want to look at the corresponding numerical data from the data frame (basically a subset of the original data whose scatter plot we are looking at).

Is there a way to isolate the data corresponding to the point(s) [when the coordinates of the data points are not apparent from the figure] ?

Below is an example. How do I get data corresponding to the circled point on the plot?

Any response would be of great help.

Thanks.


texts: Why does classifier chain ask for at least 2 classes, when I have it

texts:
I'm using Classifier Chain with logistic regression and when i try to use fit, i get

  This solver needs samples of at least 2 classes in the data, but the
  data contains only one class: 1

but I'm pretty sure I have two classes in my data, this is my X_test

<5825x2000 sparse matrix of type '<class 'numpy.float64'>'
    with 219990 stored elements in Compressed Sparse Row format>

and I have this as my y_train:

array([[1, 0, 0, ..., 0, 1, 0],
       [1, 0, 0, ..., 1, 1, 0],
       [1, 0, 0, ..., 0, 1, 0],
       ...,
       [1, 0, 0, ..., 1, 1, 0],
       [1, 0, 0, ..., 0, 1, 0],
       [1, 0, 0, ..., 0, 1, 0]])

and this is the part I'm trying to run

cc_clf = ClassifierChain(LogisticRegression(penalty='l1', C=1, dual=False, solver='liblinear'))
cc_clf.fit(X_train, y_train)
cc_y_pred = cc_clf.predict(X_test)

Am i using the classifier chain incorrectly? When I pass in the same data into one vs rest, it works.

clf = OneVsRestClassifier(classifier1)
clf.fit(X_train, y_train)


texts: Predicting probability of order arriving on or before customer Requested Delivery Date(RDD)

texts:
I have a historical data of orders from different customers of my company and delivery details. I wanted to develop a model which predicts the probability of customers order getting delivered before or on the "Requested Delivery Date".

In my data-set, i've details about
customer
delivery location
product
delivery type
product classification (Can be exported or not) etc.

I was not able to start this model as I am not sure how to tackle this problem. It would be of great help, if you could suggest some reading material on the same or few link directing the same.


texts: What is difference between final episodes of training and test in DQN?

texts:
What is difference between running in final episode of training mode and running in test mode in DQN?

Is there any difference more than after training and tune the hyper-parameters, we test for one episode and without any exploration?
This means that test mode is similar to training mode in episode n+1 without exploring (while we train for n episode) ?Is it correct?

Why in some test code of DQN, they test for multiple episodes?


texts: Weighted Linear Combination of Classifiers

texts:
I am trying to build an ensemble of classifiers whereby I want my algorithm to learn a set of weights such that it can weight the outputs of different classifiers for a set of data points.

I am wondering, how would I go about learning these weights? I tried using automatic differentiation but the weights are not moving at all (no gradient information).

Does anyone know how I can fix this?


texts: What algorithmic approach for selecting similar, relevancy based documents

texts:
I have an application that tracks people making mentions of various topics. We've used a Bayes algorithm to do some simple classification (users give a thumbs up/thumbs down) to pick the people that they believe are the best fit for their project.

Our intention was to use this data to help us sort and order the influencers based on "fit" to the customer's needs.

However, we've got some trained data, and now all we can do is say are they similar to the "thumbs up" group, or the "thumbs down" group.

What algorithm should we have used for this instead?

Basically, the ideal is to have a score.. and the biggest, smallest based on the trained data is the one that gets shown first.

Thoughts?

Even better if its in Ruby.


texts:
I want to make a new dataset containg thousands of (different-sized) images. Now I need to assign multiple labels to each image.

Of course I already looked at github etc. and there are good labelling programs there. The problem is it seems like you can always only assign one label to an image.

Do you know a simple way of assigning multiple labels to images and saving them in a dedicated file?


texts: fluctuating values for validation set only

texts:
My model's structure is
                                  Output
                                    ^
                                    |
                             ----------------
                             | Dense Network |
                             ----------------
                                   /\
                                   ||
                                   ||
                                   ||
   |--------------------|          ||          | ----------------------|
   | RNN on features    | ========>||<======== |  Dense Network on non |
   | changing with time |        [concat]      |  time series data     |
   |--------------------|                      |-----------------------|

These are the Training and Validation set metric outputs of my model. Why are the values fluctuating so much for the validation set?. Any ideas?
[![4 Graphs. Loss, Accuracy, Precision, Recall][1]][1]
Update :
As suggested in comments I have tried increasing the validation set size Now the size ratio is
49.6%-50.4%
Also I have made the model very simple by Using fewer layers. The new graph obtained looks like this
[![4 Graphs. Loss, Accuracy, Precision, Recall on simpler model][2]][2]
Is this acceptable as 'okay-fluctuating'?


texts: The classifier aims to minimize the loss function (($F(x)$ - $\hat{F}(x)$)2), where $F(x)$ is unknown function and $\hat{F}(x)$ is the predicted function. If $F(x)$ is not known for unseen data, how do we compute this loss? Why is the training error used to estimate the error for unseen data?


texts: How to consider different size of input for "Graph Conv Network"

texts:
I'm a student who just start study deep learning.

I hope to practice with simple project using Graph Convolution Network.

The question is that "How can I handle with different size of input graph for GCN?"

Padding with zero is the only way to handle this problem? And zero padding like CNN is proper also for GCN?

Thanks


texts: Predicting next number in a sequence - data analysis

texts:
I am a machine learning newbie and I am working on a project where I'm given a sequence of integers all of which are in the range 0 to 70. My goal is to predict the next integer in the sequence given the previous 5 integers in the same sequence. There isn't much more information on the sequence of integers itself (for example, how was the sequence obtained, etc).

The following are the things I tried.

The first thing that came to mind was to use a LSTM regression model with 5 input time steps and one output (corresponding to the next integer in the sequence - in Keras this would be return_sequences=False). I passed 5 previous integers themselves as the input. This resulted in the model predicting pretty much the average (~30) all the time.
I tried the model in (1) above but with more input time steps (say 100), but still no improvement.
I then tried (1) and (2), but this time using the difference between consecutive integers as the input and trying to predict the difference to the next integer in the sequence. The results with this are still bad.
I then tried a LSTM classification model by one-hot encoding the input and output since I know that all integers in the sequence are in the range 0 to 70. Again, no improvement.
I then tried a seq2seq (encoder-decoder) LSTM model with 5 inputs in the encoder and 5 outputs in the decoder with the correct outputs also being fed into the decoder (teacher forcing). Still the results are bad.

At this point I started doubting whether I can train a model on the given data and whether the given data is just a bunch of random integers.

I looked for statistical tests to determine whether the data is random or not and found out about pandas autocorrelation plot. This is what the plot looks like when plotted on the difference between consecutive integers (looks similar when plotted using the actual integers themselves).

As I understand, since the values are very close to zero, it means that the data is random. Is that right ?

I also used statsmodels "plot_acf" and the following is the plot I got for the difference between consecutive integers.

I see that there is some negative correlation when the lag is 1. Why doesn't this show up in the plot using pandas' autocorrelation_plot()?

I tried building a AR (auto regression) model as well, but still the results are bad.

The histogram of the integers in the sequence also seems to suggest that the integers are random (all values have about the same count except some higher integers).

Am I wasting my time trying to build a machine learning model to predict the next integer in the sequence ?


texts: What is the right approach to bucket users for algorithms with different coverage for A/B testing

texts:
I've couple of recommendation algorithms that I want to A/B test. Algorithm A has 90% user coverage and algorithm B has 95% user coverage. That means if the algorithms are asked to provide recommendations for 1000 users, algorithm A can give it for 900 of the users and algorithm B can give it for 950 other users. Say for example out of these 1000 users 87% has recommendations from both algorithm, 3% has recommendations from only algorithm A and 8% has recommendations from only algorithm B. What would be the ideal approach to bucket the users

Bucket only those 87% users as that can generate even bucketing, but the higher coverage of algorithm B won't reflect in A/B test.
Bucket all 95% users, but that could cause uneven bucketing.


texts: When does random forest feature importance fail?

texts:
I'm curious about the assumptions of random forest feature importance.

In this paper, the author says that

  "We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories."

I don't understand what it means by "predictor variables vary in their scale of measurement." Is this referring to the need to standardize data before fitting a random forest model?

Does RF based feature importance require standardization of features?
Does it require fitting a good RF model (ex: pruning), as if I'm making an actual prediction with RF?
What do I do if I have categorical variables and continuous variables at the same time? Is permutation method the only option?


texts:
I am using forward and backward hook in my pytorch densenet121 model.
I set requires_grad to False at the time of training.
for param in model.features.parameters():
    param.requires_grad = False

and I want to get gradient of last conv layer in Neural Network , for this i define hook like this:
model.features.denseblock4.denselayer16.conv2.register_backward_hook(h1)

but this is giving me empty list. But is i set requires_grad = True for the last layer then i get the gradient.
for param in model.features.denseblock4.denselayer16.conv2.parameters():
    param.requires_grad = True

I use gradient for gradCAM .
is this write way to doing or is there  any other way to get gradient or does i need to set requires_grad = True for whole network ?


texts:
I have a supervised learning (boolean classification) problem that involves strings.  Are there any resources where I can learn about the state of the art in techniques for this?

I'm familiar with many supervised learning techniques where we assume that each instance has a fixed number of attributes: e.g., the instance is represented by a feature vector.  But now I have something different; I have a string, i.e., a sequence of characters.  I don't know a priori what suitable features might be.  Are there any general techniques for doing machine learning in this context?

I'm especially interested in techniques that work given an existing training set (I don't have the ability to label additional data; I don't want an active learning algorithm), and that support some kind of regularization or can handle noisy labels.

Research I've done and approaches that don't meet the above requirements:

I'm familiar with regexps, finite automata, grammar induction, LearnLib, Angluin's algorithm, and similar concepts.  However, most of that work is in the active learning context, where the algorithm generates a new instance (a query) and asks the human to label that instance.  In my setting, I must work with passive learning: I have a training set $(x_i,y_i)$ where each $x_i$ is a string and $y_i$ is its class (positive or negative), but I can't obtain any more examples.
I'm also familiar with automata minimization, so I know how to find the minimal DFA that can generate all positive instances in the training set, but that's not really a good solution.  It doesn't have any notion of regularization or Occam's razor and is not robust to a small amount of error/noise in the labels.  There might be a very small DFA that predicts the proper label with 99% accuracy, but automata minimization won't find it; it is overly focused on finding an automaton that generates all positive instance, even if this yields a much larger automaton.
I'm also familiar with the notion of separating automata, i.e., the smallest DFA that has 100% accuracy on the training set (accepts all positive instances and rejects all negative instances).  However, this has the same problems as automata minimization.  Also, this seems likely to overfit and essentially end up memorizing the training set.  Finally, the research literature I've seen stops at noting that finding the smallest separating automaton is NP-hard, and doesn't concern itself with practical algorithms for finding a separating automaton that is as small as possible.
I'm familiar with recurrent neural networks, but my impression is that they tend to require very large training sets, so probably won't be useful in my setting.  Are there variants of this technique that work with training sets that aren't enormous (say, hundreds of examples in the training set)?
I'm familiar with some techniques from the natural language processing (NLP) literature, but they seem very specialized to parsing human languages.  Are there any general techniques that apply to other structured strings?  In my situation, the strings are not human-readable text written in some human language, but rather have some other content.


texts:
We have a classification algorithm to categorize Java exceptions in Production.
This algorithm is based on hierarchical human defined rules so when a bunch of text forming an exception comes up, it determines what kind of exception is (development, availability, configuration, etc.) and the responsible component (the most inner component responsible of the exception). In Java an exception can have several causing exceptions, and the whole must be analyzed.

For example, given the following example exception:

com.myapp.CustomException: Error printing ...
... (stack)
Caused by: com.foo.webservice.RemoteException: Unable to communicate ...
... (stack)
Caused by: com.acme.PrintException: PrintServer002: Timeout ....
... (stack)

First of all, our algorithm splits the whole stack in three isolated exceptions. Afterwards it starts analyzing these exceptions starting from the most inner one. In this case, it determines that this exception (the second caused by) is of type Availability and that the responsible component is a "print server". This is because there is a rule that matches containing the word Timeout associated to the Availability type. There is also a rule that matches com.acme.PrintException and determines that the responsible component is a print server. As all the information needed is determined using only the most inner exception, the upper exceptions are ignored, but this is not always the case.

As you can see this kind of approximation is very complex (and chaotic) as a human have to create new rules as new exceptions appear. Besides, the new rules have to be compatible with the current ones because a new rule for classifying a new exception must not change the classification of any of the already classified exceptions.

We are thinking about using Machine Learning to automate this process. Obviously, I am not asking for a solution here as I know the complexity but I'd really appreciate some advice to achieve our goal.


texts:
I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.)

For example

Height  | Weight
180     | 70
175     | N/A
N/A     | N/A

I want to fill missing values and normalise the columns.

The data is heights and weights so I thought a good fill value would be 0 or -1. This is based on the book Deep Learning in Python:

  In general, with neural networks, it's safe to input missing values as 0, with the condition that 0 isn't already a meaningful value.

EDIT I assumed 0 wasn't meaningful in a dataset with values from 150-200

I was also recommended to normalise the data by subtracting the mean and dividing by the std for each column.

Both of those are fine on their own - I understand how  and why to do them. What I don't get is how to combine them. I can either ...

Fill missing values then normalise, but then a) my zeros will no longer be zeros (will my network still learn they are a special value?), and b) the zeros will affect the mean/std to a degree determine by how many values are missing. I suppose I'm concerned this would give a weird distribution
Normalise then fill missing values. But after I've normalised my data, 0 is now the mean of my column and so isn't a fill value of the same kind. I'd rather let the network know the values are unfilled than assume they all take the mean value

I'm using Keras, Numpy and Pandas with Dense layers for a multiclass classification problem.


texts: What am I supposed to see on tensorboard images tab?

texts: I'm training an object detection model with Tensorflow and monitor the training task with tensorboard. I was expecting in the Images tab of tensorboard that displayed images would show a bounding box (at a specific point of training). What I see though is only images with an orange line drawn above the picture (the same orange that I expect for the bounding box). Am I missing something? Am I right when I say that a bounding box should appear or not? Picture of what I see is joined. Any help greatly appreciated.


texts: Different models for different time durations of a day

texts:
I have hourly temperature and power consumption data of several days of a month. The pattern is almost similar across days like this:

Using this data I want to predict the usage of a coming day. I have features : 1) hour of the day 2) temperature; and the response variable, power. Looking at the data, I believe I should fit three separate models and not a single model

First model for data from Midnight to 10 AM as usage remains almost constant during this time, and temperature does not vary too much
Second model for data from 11AM till 6 PM. This portion follows a sharp increase and then almost constant usage
Third model for data from 7 PM till midnight. This portion shows constant decrease in power

To follow this intuition, I used three models accordingly and later combined predictions from these to output sequence of 24 numbers for coming day. Formula for each of these models is:
lm(power ~ time_hour + temperature, data = xxx), but each model is trained with the data corresponding to specific time duration of the day.

Instead of dividing the data manually and using three separate models, is there any other existing technique which will take care of our intuition and does not need manual division of data or creation of separate models.

During my search, I found that I can use GAM (Generalized Additive Models) and I came up with following formula

library(splines)
lm(power ~ ns(time_hour, knots =(9, 18)) + temperture, data = xxx)

Using above formula I think that I am placing knots at 9 AM and at 6 PM. Right?
I don't know how should I enforce knots in temperature feature exactly at these specific times, so that knots of temperature and time_hour will sync.

The above plots were plotted using following data:

dframe <- structure(list(time_hour = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24), temperature = c(22.5,
24, 23.5, 20.5, 22.5, 22.5, 19.5, 23.5, 23, 20.5, 26.5, 28.5,
30, 32, 33.5, 33, 30.5, 30, 29.5, 29, 28, 27, 28, 28.5), power = c(97.04319,
95.7225, 88.59191, 88.34882, 90.17179, 88.82062, 87.73833, 89.36342,
85.31775, 91.1292, 116.79035, 149.58614, 172.32438, 171.27931,
159.53858, 162.03544, 170.78468, 164.0275, 155.86717, 135.77197,
133.01235, 116.29253, 100.87483, 97.84942)), .Names = c("time_hour",
"temperature", "power"), row.names = c(NA, -24L), class = "data.frame")

and minimal code used is:

par(mfrow= c(1,2))
plot(dframe$time_hour ,dframe$power,type="l",xlab = "Hour of day", ylab = "power" )
plot(dframe$time_hour ,dframe$temperature,type="l",xlab = "Hour of day", ylab = "Temperature" )


texts: When to use Standard Scaler and when Normalizer?

texts:
I understand what Standard Scalar does and what Normalizer does, per the scikit documentation: Normalizer, Standard Scaler.

I know when Standard Scaler is applied. But in which scenario is Normalizer applied? Are there scenarios where one is preferred over the other?


texts: How to find parts of text that answer a "why"

texts:
Given some text that describes something,
how to tag / identify parts of the text
that explain specific aspects - what, who, why?
For example, given the following input text...
Evaporation is an essential part of the water cycle.
The sun (solar energy) drives evaporation of water
from oceans, lakes, moisture in the soil, and other
sources of water.

In hydrology, evaporation and transpiration
(which involves evaporation within plant stomata)
are collectively termed evapotranspiration.

Evaporation of water occurs when the surface of
the liquid is exposed, allowing molecules to escape
and form water vapor; this vapor can then rise up
and form clouds. With sufficient energy,
the liquid will turn into vapor.

How would one go about obtaining output like this:
What = "Evaporation", "Hydrology", "Evapotranspiration", "Vapor"
 Who = ""
 Why = "surface of the liquid is exposed, allowing molecules to escape", "With sufficient energy"

Could a parts-of-speech tagging framework help here?
Which parts-of-speech should be mapped as answering a why.

PS: Is there a term for this already? PoS-Tagging and Sentiment-Analysis are close but not quite what this is, right?


texts: How do I calculate the maximum likelihood (machine learning statistics) of this table of data?

texts:
I’m taking a machine learning course and I’m stuck. I've already tried to ask around in the forums of the class but I don't understand how to implement the formulas they give me. Can you please help me understand how to solve this problem from the ground up?

  “Calculate the likelihood of this data.”

  X  | Y

  2.5 | +1

  0.3 | -1

  2.8 | +1

  0.5 | +1

  the current estimates of the weights are $w_0 = 0$ and $w_1 = 1$. ($w_0$: the intercept, $w_1$: the weight for $x$).

Now Here's what I've already done in case that matters:

When I went on to the forums to try to understand I was told this:

  You can use the sigmoid function to find the likelihood of any given
  one, and then once you have the probabilities of any given one
  occurring you should be able to find the probability of the whole
  thing

Which is nice; now I know it has something to do with the sigmoid function: $$\frac{1} {1 + e^{-x}}$$

sigmoid(2.5) = 0.9241418

I hope I'm on the right track, I really have no way of knowing. So then I calculated the sigmoid for all of them and multiplied them together hoping that would give me the probability of the whole thing because that's how you combine probabilities, right? (The probability of a dice role hitting 2 three times is $1/6 \times 1/6 \times 1/6$).

But that apparently isn't the right answer, and I didn't take into account the $y$ column or the weights because I don't know how. How sad for me.

Can you please explain it to me, I feel like I'm a blind man looking for the right colored paper.


texts: How to interpret dummy variable in ML prediction?

texts:
I am working on a binary classification problem where I have a mix of continuous and categorical variables.

Categorical variables were created by me using get_dummies function in pandas.

Now my questions are,

1) I see that there is a parameter called drop_first which usually is given the value True. Why do we have to do this?
Let's say for the purpose of example, we have 2 values in gender column namely Male,Female. If I use drop_first=True, it returns only one column. like gender_male with binary 1 and 0 as values
For example, If my feature importance returns gender_male as an important feature, Am I right to infer that it is only Male gender that influences the outcome (because male is denoted as 1 and female is 0) and female (0's) don't impact the model outcome? or 0's in general doesn't play any role in ML model predictions?

2) Let's say my gender has 3 values for example Male,Female,Transgender. In this case if I use drop_first=True, it would only returns two columns

gender_male with 1 and 0  - Here 0 represents Transgender right?

gender_female with 1 and 0 - Here 0 represents Transgender right?

3) What's the disadvantage of not using drop_first=True? Is it only about the increase in number of columns

Can you help me with the above queries?


texts: Varying results when calculating scatter matrices for LDA

texts:
I'm following a Linear Discriminant Analysis tutorial from here for dimensionality reduction. After working through the tutorial (did the PCA part, too), I shortened the code using sklearn modules where applicable and verified it on the Iris data set (same code, same result), a synthetic data set (with make_classification) and the sklearn-digits dataset.

However, then I tried the exact same code on a complete different (unfortunately non-public) data set that contains spectra recordings of two classes. The LDA crashes at the eigenvector verification part, where the $\lambda \mathbf{v}$ is supposed to be almost equal to $S_W^{-1} S_B \mathbf{v}$ (with $\lambda$ being the eigenvalue and $\mathbf{v}$ the corresponding eigenvector; $S_W$ and $S_B$ are the in/between-class scatter matrices). The first vector to be wrong seems at random positions, meaning each run it's a different vector that's causing this error.

I suspect it's related to rounding during calculations, since I get complex eigenvectors. For the PCA I just discarded the complex part (I think I read it somewhere in this forum), but this approach does not seem to work with LDA. Has anybody encountered similar problems or knows what's wrong?

Following is my code for the analysis, which is more or less the same as in the tutorial. I'm using the manual approach, since I'm interested in how many linear discriminants are needed to describe my data. (I'm not sure how to do this with sklearn's LDA.)

def LDAnalysis_manual(X, y):
    n_features = X.shape[1]
    n_classes = len(np.unique(y))

    print("Mean vectors...")
    mean_vectors = []
    for cl in range(n_classes):
        mean_vectors.append(np.mean(X[y == cl], axis=0))
        # print("Mean vector class {}: {}".format(cl, mean_vectors[cl - 1]))

    print("In-class scatter matrix...")
    S_W = np.zeros((n_features, n_features))
    for cl, mv in zip(range(1, n_classes), mean_vectors):
        class_sc_mat = np.zeros((n_features, n_features))  # each class' scatter matrix
        for row in X[y == cl]:
            row, mv = row.reshape(n_features, 1), mv.reshape(n_features, 1)  # column vectors
            class_sc_mat += (row - mv).dot((row - mv).T)
        S_W += class_sc_mat  # sum class scatter matrices

    overall_mean = np.mean(X, axis=0)
    print("Between-class scatter matrix...")
    S_B = np.zeros((n_features, n_features))

    for i, mean_vec in enumerate(mean_vectors):
        n = X[y == i + 1].shape[0]
        mean_vec = mean_vec.reshape(n_features, 1)  # make column vector
        overall_mean = overall_mean.reshape(n_features, 1)
        S_B += n * (mean_vec - overall_mean).dot((mean_vec - overall_mean).T)

    eig_vals, eig_vecs = np.linalg.eig(np.linalg.inv(S_W).dot(S_B))

    print("Eigenvector test")
    for i in range(len(eig_vals)):
        print("\r{:3}".format(i), end=" ")
        sys.stdout.flush()

        eigv = eig_vecs[:, i].reshape(n_features, 1)
        np.testing.assert_array_almost_equal(np.linalg.inv(S_W).dot(S_B).dot(eigv).real,
                                             (eig_vals[i] * eigv).real,
                                             decimal=6, err_msg='', verbose=True)
    __log.debug("\nAll values ok.")

    eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:, i]) for i in
                 range(len(eig_vals))]  # make list of value & vector tuples
    eig_pairs = sorted(eig_pairs, key=lambda k: k[0], reverse=True)  # Sort tuple-list from high to low

    __log.info("\nEigenvalues (decending):")
    for i in eig_pairs:
        __log.info(i[0])

    tot = sum(eig_vals)
    var_exp = [(i / tot) for i in sorted(eig_vals, reverse=True)]
    cum_var_exp = np.cumsum(var_exp)
    cum_var_exp = cum_var_exp.real
    plot(len(var_exp), var_exp, cum_var_exp)

    idx_98 = next(idx for idx, val in enumerate(cum_var_exp) if val > .98)
    return idx_98 + 1


texts: Methods for analyzing food diary (time series CSV) to correlate foods to symptoms

texts:
Background

The doctor recommended that I keep a food diary to try and figure out what foods are correlated to the symptoms I experience. However, they didn't have a recommendation for a method to analyze the collected data (apparently normal people don't do that!) ... so I figured, I can do statistics using python, right?

While I'm very familiar with python, unfortunately my statistics background is weak. As a starting point, I would like to know which foods are more correlated with which symptoms. Knowing what statistical method(s) to use to analyze this type of data would be a great starting point, python code would be even better.

The data

I have transcribed my food diary into CSV similar to this form:

time,event
2018-07-12 16:30:00,food1
2018-07-12 16:35:00,food2
2018-07-12 17:00:00,symptom1
2018-07-12 17:05:00,symptom2
2018-07-12 18:10:00,food1
... etc, etc

It was trivial to load the data into jupyter+pandas+matplotlib and make scatterplots showing the data (which is cool, but not super helpful yet). I've been reading about various ways to one could analyze time series data, but a lot of that doesn't quite fit what I'm trying to do.

My specific question

Given data of the form shown above, how does one determine that particular named items are correlated with other named items? For example, "86% of the time food1 occurs 90 minutes before symptom1"

A subproblem: My google searches for time series data analysis seems to yield answers that all involve numbers, whereas this data is just a list of things and isn't very numerical (other than that it's a time series). If I load the data into pandas and call data.corr(), it gives me an empty dataframe. How does one typically overcome that?


texts: How to tune parameters for Time Series Analysis, when forecasting is only dominated by one feature and error is not getting reduced?

texts:
I am trying to predict time series based on 150 features. When I plot correlation of these features, I am getting 20 features with more or less importance but every model I use, it is completely dominated by only one feature which is competently in sync with predicted output but not actual output . Please refer to the image below.

The green line is prediction which is completely in sync with one of the feature.And for every valley in actual output, I am getting 2 valleys in predicted output. No model is able to generalize this. Is it the case of bad data for model?


texts: Estimating Titan X graphics card impact on performance

texts:
I'm currently training CNNs using Tensorflow (Python) on my GTX 970 (specs here). I recently took a look at the new pascal based Titan Xs and I'm wondering what an estimated performance/speed gain would be if I upgraded?

The memory increase is an obvious benefit for larger models, but I'm mostly wondering about speed.

Does the doubling of core count and bump in memory speed offer that much of a performance gain? Anyone have any experience with the new (or old) Titan X cards?


texts: Can sampling like SMOTE/UP/DOWN applied on Validation set?

texts:
I am trying to predict classification problem. For that I have used Ranger, Xgboost and naive bayes. My Response class is imbalance . 92:8 ratio. My positive Response is only 8% of whole data.

Because of class imbalance I am getting more FP / FN.

I tried different sampling on training set and predicted with original set gives me more FP.

Can I do something like ?:
- split data into train and test
- apply sampling method individually on train and test
- and use sampled test to predict.

Thanks.


texts: Does Non Negativity Constrains increases the estimation error

texts:
I have been working with Tensor and matrix Non negative constrained algorithms. I have never seen a non negative constrained algorithm (ex. Non Negative Tucker Decomposition NTD) with error that is comparable to the non constrained algorithm (ex.High Order SVD HOSVD). Publications normally don't compare estimation error of unconstrained algorithms with constrained algorithms instead the comparison is always between different algorithms of same type.

By Testing, I found that the result of constrained algorithms with noise is around $10^{-7}$ compared to unconstrained ones is around $10^{-15}$.

Any reference that I can start from or any explanation would be appreciated.


texts: Why would you use word embeddings to find similar words?

texts:
One of the applications of word embeddings (such as GloVe) is finding words of similar meaning. I just had a look at some embeddings produced by glove on large datasets and I found that the nearest neighbors of a given word are often fairly irrelevant. Eg. ‘dad’ is the closest neighbor of ‘mom’, ‘dealership’ is the seventh closest neighbor of ‘car’.

In light of this, if you wanted to find words of similar semantics why would you prefer using embeddings instead of just downloading a database of synonyms from an online dictionary that is compiled by humans?


texts: When creating a classification model, should predictors with little correlation to the response variable be included in the model?

texts: I am building a predictive model designed to predict attrition within my organization.  I am trying to decide whether to add certain predictors to my model.  I used a Kruskal-Wallis rank sum test to check the correlation between a few of my predictor  variables and my response variable and found that the predictors are independent from my response.  Should I still include these variables in my model? I am leaning towards no due to the lack of correlation with my response variable but don't want to discard a potential "split" in my decision tree.  My data set contains 8,225 observations and 173 columns which I have been using as predictors.


texts: Keras or TensorFlow Examples for Working with Large Text Datasets (~10M Sentences)

texts:
I'm looking for any tutorials or examples for using either Keras or TensorFlow on large text datasets (at least a few million sentences). I've looked at some of the examples using the fit_generator() method in Keras, but these are mostly tailored to image classification and involves really minimal preprocessing. I've also looked into using the newer HDF5Matrix class in Keras, and have continued to run out of memory with batch sizes larger than 20.

Are there any other useful data reading/streaming tools in TensorFlow that could be used to solve this problem? Are there examples for how to do standard NLP preprocessing in such a pipeline in a memory-efficient way before feeding to a model? Any references would be appreciated. Thanks for reading.


texts: Does it make sense to "reorder" a categorical feature to make it monotonic?

texts:
Sorry for the vagueness of the title; I'll explain what I mean. I'm doing the Kaggle Titanic beginner tutorial. The label you're interested in is the "survived" rate (0 or 1), which you want to predict in the test data.

One of the first things I did was plot the average survival rate for each of the features, for each one, taking the average survival rate for that value of the feature. For example, for the "Sex" feature, you can look at the average survival rate for males and females, which gives you good info you can use.

For features like "Fare" and "Age", which are more continuous, it wouldn't really make sense to take the average in the same way, because there aren't necessarily many people with the same exact age. So, I did "banding", where I do a histogram-like thing of saying that everyone from 0-10 is one band, 10-20 another band, etc, and those bands are now categorical features that can be averaged over well (this is a good move, right? at least initially?).

This is what I get:

That's the background. Now, here's my question. I want to start with a simple model like linear regression. For some features, like Sex, or Pclass, it's clear that a linear model could fit it very well. For some however, like AgeBand, there's not really a clear, monotonic trend for increasing AgeBand...but there definitely is a difference in the survival rate between different bands!

So what you could do is, sort those "non monotonic" features by the survival rate, so it is monotonic, and then use them for linear regression. Is that a good idea or no?

For example, here's the AgeBand feature sorted by the survival rate:


texts:
Would someone know how to conduct some type of visualisation within a cluster in R (i.e. plot histogram/frequency of a certain variable within a cluster)?
For example, suppose my partitioning around medoids output was summarised as follows:
pam_results$the_summary

Would anyone know a set a commands which would help plot a histogram for something like 'MR' from cluster 1?
Would appreciate all thoughts.
Best


texts:
I've seen both of these techniques be used for image search. One difference I can think of is that autoencoders don't rely on labeled data. I'm not sure, but it seems logical therefore that they can possibly generate more discriminatory dimensions for the final vector-representation, given that you're no longer bound by the classifications from the labels.

For my particular problem, I have labeled data, which is why I'm stuck between the 2. This is for non-CNN autoencoders.

What are the advantages/disadvantages of using Autoencoders over CNNs for image search?


texts: Search for implementation of Faster RCNN

texts: What are the best written and best structured Faster RCNN implementations that you know? Please provide references.


texts: Feature selection for circular data in time-series

texts:
I'm predicting ozone concentration based on meteorological and seasonal variables. In the feature engineering stage I converted the MONTH, DAY_OF_WEEK, DAY_OF_YEAR to its sin and cosine components because I read in some articles that this is how circular values should be treated.

I have 3 questions regarding this:

1) When I try to do feature selection, or feature importance, why is the cosine having more importance than sine for the same variable? Is there a scientific explanation?

2) I'm trying to study which variables have higher importance for predicting ozone, but how can I determine that since the seasonal variables are split into sine and cosine?

3) In the feature selection, can I drop the sine of the same variable and keep the cosine if the cosine has more importance/weight while the sine isn't?

For example, here is the feature importance by XGBoost Regressor:


texts:
When you connect a data source, Tableau automatically infers the type of each column of your data.  Whether it's Number (decimal), Number (whole), String, Boolean, etc.  A few questions about this:

1) Has Tableau ever misclassified one (or more) columns of your data with their automatic labeling?  If so, would you please give details?

2) Do you feel that these data type options can be improved?  For example, Tableau doesn't seem to be distinguishing between unordered categorical data and ordinal data.

Thanks!


texts: When is a neural network better "traditional" models like decisions trees and lassos?

texts:
There's a whole theory of statistical inference based off calculus studying consistency, efficiency, robustness, BLUE, unbiasedness of linear models (Gaussian,Exponential, Chi-square, F-distribution, etc)... that make up regression models.

When is a neural network better than these traditional models based off calculus that are used regression-analysis? Is there a whole mathematical theory including robustness, Rao-blackwell stuff, consistency, sufficency dedicated to them that someone like Wackerly/HOFF writes about in a book?

Are neural networks not explainable by calculus as logistic regression is explainable as the maximum likelihood estimator?

What's the mathematics behind neural networks or is that more CompSci theory?

Are neural networks unbiased estimators that are consistent, efficient, etc that are on average the right answer? Can they estimate the true parameters of the population?


texts: Best Way to tackle to time series classification problem?

texts:
I have a dataset where the input is a dataset for ICU patients where each ICU stay has 40 features (20 vitals, 20 lab values) and multiple time steps (the stays' length is between 6 and 19-time steps). The output is the 20 lab values (represented as binary numbers where 1 means the lab value is in range and 0 is out of range). The task is to predict the future output for an input ICU stay.
I have tried to make it a sequence to sequence problem where the output is a sequence shifted by a one-time step. However, the results were not good enough.
The question is there a way to better tackle this problem? (like maybe windowing ?)


texts: Is confusion matrix possible in one column

texts:
I am performing anomaly detection using K-Means. I am working with only one column, plotting those values and then within this column I am adding some anomalies.

My question is

if it is possible to perform confusion matrix for this ? If so what is the approach ?
Asking this since I have seen that confusion matrix needs y axis and x axis


texts: How to check and correct misspelling in the data of pairs of words?

texts:
I have user generated text containing names of ports often containing typos and the actual port names. I would like to correct the misspelling of user generated text containing the names of ports. Can anyone help me think of some ideas or steps to take?
Perhaps using word2vec? RNN?

thank you in advance.


texts: Morphological Analyzer for Hindi and English in Python

texts:
I am unable to find morphological analyzers for English or Hindi. I am looking for a tool that can split the word into morphemeses e.g.
Independently = In + dependent + ly


texts: What are the equations involved in calculation of the parameters of embedding layer?

texts:
I'm trying to perform sentiment analysis on some data using keras.I'm using embedding layer and then LSTM. I know that embedding layer decreases the sparsity of the one hot encodings of the words and its parameters are trained while back-propagation, but I don't know the mathematics of its implementation.

Thanks in advance.


texts: Triplet loss when positive or negative examples are missing

texts:
I am trying to use Siamese network for text classification, where I have to output whether two texts agree or disagree with each other, i.e. both the texts are of the same context but have opposite relation
However in my training data there are not many points where we have two example text mapping to the same anchor statement and having a relation of agree and disagree.
So, I wanted to ask if it is possible to train the triplet loss function by passing the anchor as the positive and negative example and 0 as the negative example when either of them are missing.
Like when positive is missing, we do
L = f(a, a) - f(a, n) + margin
and when negative is missing
L = f(a, p) - f(a, a) + margin


texts:
I am trying to explore models for predicting whether the a team will win or lose based on features about the team and their opponent. My training data is 15k samples with 760 numerical features. Each sample represents a game between two teams and the features are long and short term statistics about each team at the time of the game (i.e. avg points over last 10 games).

My thought was to use a binary classifier as a multi layered perceptron.  Each layer has batch norm, dropout and an relu activation. At the output layer there is a sigmoid activation. I am also using principal component analysis to reduce the dimensionality of my dataset.

Im am using an automated hyper-parameter tuner with randomized search and population based training to find optimal accuracy hyper-parameters. I am able to tune the hyper-parameters such that training loss and validation loss both converge to a very small value but when I look at accuracy I am getting very minimal improvements.

When looking at the initial accuracy after one epoch of training, it seems to be around 60% across all initial hyperparams, and it only reaches a max of about 70% after 30 epochs.

Im not really sure what to do, does this imply my data is to noisy / random? Should I reconsider my architecture?  Would I get better results with a regression instead of classification? Also, since the data is sequential in nature this might lend itself to a recurrent architecture but Im not familiar with any RNN that is trained on multiple sequences as opposed to a single sequence.


texts:
From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated.

What I don't know is if the original embedding - the original thing you put into BERT - could or should be a vector. People wax poetic about how BERT or ALBERT can't be used for word to word comparisons, but nobody says explicitly what bert is consuming. Is it a vector? If so is it just a one-hot vector? Why is it not a GLoVE vector? (ignore the positional encoding discussion for now please)


texts: Dense? or TimeDistributedDense? after LSTM layer in Keras

texts:
Dense and TimeDistributedDense, which one is suitable after LSTM layer in Keras?

For example,

input = Input(shape=(12,N_indepen), dtype='float32', name='inci_input')
lstm = Bidirectional(LSTM(100, activation='elu',return_sequences=False))(input)

# Dense? or TimeDistributedDense?
dense = layers.Dense(30, activation='elu')(lstm)
dense = layers.TimeDistributedDense(30, activation='elu')(lstm)

m_y = layers.Dense(1, activation='linear')(dense)

I read some articles, most of them used Dense, but some used TimeDistributedDense. I hope to know what is proper theoretically.


texts: Can I use categorical and numerical data variable at the same time in randomForest in r?

texts:
I have data in which few columns contain categorical data whereas remaining columns contain numerical data. I want to use random forest regressor from the randomForest library in r. So does this library work with such data out of the box (assuming I have preprocessed the data and is ready to directly feed in the regressor)?

DATA (something along this line) :

Target | product | status | revenue | MS
  0.8      abc       NC      1000    0.5
  0.5      abc       UR      200     0.2
  0.2      pqr       NC      800     0.04


texts: Are Support Vector Machines still considered "state of the art" in their niche?

texts:
This question is in response to a comment I saw on another question.
The comment was regarding the Machine Learning course syllabus on Coursera, and along the lines of "SVMs are not used so much nowadays".
I have only just finished the relevant lectures myself, and my understanding of SVMs is that they are a robust and efficient learning algorithm for classification, and that when using a kernel, they have a "niche" covering number of features perhaps 10 to 1000 and number of training samples perhaps 100 to 10,000. The limit on training samples is because the core algorithm revolves around optimising results generated from a square matrix with dimensions based on number of training samples, not number of original features.
So does the comment I saw refer some real change since the course was made, and if so, what is that change: A new algorithm that covers SVM's "sweet spot" just as well, better CPUs meaning SVM's computational advantages are not worth as much? Or is it perhaps opinion or personal experience of the commenter?
I tried a search for e.g. "are support vector machines out of fashion" and found nothing to imply they were being dropped in favour of anything else.
And Wikipedia has this: http://en.wikipedia.org/wiki/Support_vector_machine#Issues . . . the main sticking point appears to be difficulty of interpreting the model. Which makes SVM fine for a black-box predicting engine, but not so good for generating insights. I don't see that as a major issue, just another minor thing to take into account when picking the right tool for the job (along with nature of the training data and learning task etc).


texts: How many images are generated when ImageDataGenerator is used, and when data augmentation is included as a part of the model?

texts:
Is there any way to know the number of images generated by the ImageDataGenerator class and loading data using flow_from_directory method? I searched everywhere for the same but couldn't find anything useful.
Also, if I use image_dataset_from_directory fuction, I have to include data augmentation layers as a part of the model. In that case, how many images are generated?
My ImageDataGenerator code:
train_datagen = ImageDataGenerator(rescale=1./255,
                                   horizontal_flip=True,
                                   zoom_range=0.2,
                                   shear_range=0.2,
                                   rotation_range=15,
                                   fill_mode='nearest')

My code with data augmentation as a part of the model:
data_aug = Sequential([
                         layers.experimental.preprocessing.Rescaling(1./255),
                         layers.experimental.preprocessing.RandomFlip('horizontal'),
                         layers.experimental.preprocessing.RandomZoom(0.2,0.2),
                         layers.experimental.preprocessing.RandomRotation(0.06)
])

# in the model:

input_layer = layers.Input(shape=(150, 150, 3))
x = data_aug(input_layer)


texts:
How can I create a VM instance with GPUs on Google Cloud Platform?
When I go to https://console.cloud.google.com/compute -> CREATE INSTANCE, I only see CPUs and no GPUs, as shown in the video below. I did select a region+zone that is supposed to have GPUs according to https://cloud.google.com/compute/docs/gpus

https://cloud.google.com/compute/docs/gpus (mirror):

I see that some VMs from the marketplace comes with GPUs but I'd prefer to configure the VM myself.


texts: How to clean messy column and reshape data structure in pandas

texts:
I have a dataset that looks like this.

There are 122 columns of coordinates excluding the date (Time) column and 7000 rows.

What I want to do is split the lat and lon into two columns, and then reshape the data so that the lat and lon become the index, and the dates become the columns.

So the header would look like: lat -- lon -- date1 -- date2 -- date3...

I am new to Pandas and I have looked for solutions online but I can't find any solution to my issue. So far my progress is on splitting the lat and lon, now I'm stuck figuring out the next step - do I create two columns to store the lat and lon first, or do I reshape the data structure first?


texts:
I would appreciate if you could let me know how to resolve this error:
Code:
X = np.array(pd.read_csv('my_X_table1-1c.csv',header=None).values)
y = np.array(pd.read_csv('my_y_table1-1c.csv',header=None).values.ravel())
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)

def Ridgecv(alpha):
    return cross_val_score(Ridge(alpha=float(alpha), random_state=2),
                           X_train, y_train, 'mae', cv=5).mean()

The error is related to X_train, y_train:
ValueError: Found input variables with inconsistent numbers of samples: [1052, 1052, 3]


texts: What is the best approach for send time optimization?

texts: I could no find a lot information about how the companies doabout send time optimization, either for push notifications or email campaigs. having historical data about clicks and sends what would be your approarch?


texts: How to create model for selecting a set of categories with a set of attributes?

texts:
I have a couple of hundred categories where each of these categories has a specific set of attributes having different values (historical).

The problem I need to solve is to select the best set of categories out of a smaller group which meet some constraints.

I'm new to datascience and was wondering how do I go about solving this problem?

One option I thought was to use multiple regression for the different attributes to assign a weight to each category and then use these to generate a random forest on the historical groups of categories to train and test them.

Does this make sense?


texts: Whether to replace NaN values in result column

texts:
I have a training dataset where we have to predict "Result" based on features "A", "B", "C" and "D" using machine learning. For a few rows, the "Result" is empty (7/19612).
While for the other features I have filled the NaN values with their mean, I don't understand whether or not to do so with the result column. Will it better to drop the rows entirely instead?


texts: What method is recommended after outliers removal?

texts:
I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials.
I was thinking of applying outlier removal on my data. and the data does look better (I used a Matlab function which removed all data above of below 3 IQR's above and below median).
After doing that I got a histogram which is more similar to a normal distribution (below an example picture of one of my sessions).
My question is:

After applying my outlier removal, how should I analyze the remaining data?
Should I consider the Median (together with IQR as standard error mean)?
Or should I consider the mean (together with $ \frac{\sigma}{\sqrt n}$ as standard error mean)?

Remark: I have very little knowledge in statistics, so, I there are mistakes above (for example my standard error mean definition as IQR or $ \frac{\sigma}{\sqrt n}$ is not correct), I would be grateful if you'll let me know.
Thanks!
Edit: The purpose of my analysis is to show that under certain conditions, the response times of the mice will be faster then under other conditions.
fig 1: Data before outlier removal

fig 2: Data after outlier removal


texts: Feature Selection with non-linear numerical and categorical variables

texts:
I have a dataset of 45 non-linear numerical values and 2 categorical values. I am making a feature selection to predict categorical variables one by one or together. I used the correlation ratio and kendall rank correlation coefficient to calculate the strenght of the relations.

Among these two methods, which one should I use as the primary method to sort the strenght of variables? For example the table below is sorted by correlation ratio. Should I sort the table with the kendall coefficient instead?


texts: Handling aggregate and individual data input to neural networks?

texts:
Consider a scenario where I have a lot of aggregate data as well as individual data that makes up that aggregated data. For example, Rotten Tomatoes, where aggregate data would be movie tags and overall critic and audience rankings, while the individual data would be each review.

Is it possible to treat this as input to a neural network? Do I just repeat the corresponding aggregated data when feeding in each individual review to be classified? Is there a better way to do this?


texts: How to print out to a file using Standford Classifier

texts:
I am using Standford Classifier for my project.
This project takes training data to tune the algorithm then test data to classify text inputs into categories.

So the format for test and training data is tab-delimited text which means predictor -TAB- input text

The software prints out the output to stdout (command line).
Is there anyway to output to a text file?

I searched the javadoc of the project site, and I found:

But I don't know how to use this property.

I tried -csvoutput=%1%n%c on command line
But it gives me java null pointer exception error when I try to run it.
Any ideas?
Thank you so much.


texts: What kind of algorithms can be used as a stacker in stacked generalization?

texts:
In stacked generalization, several algorithms (I use some random trees, booster trees, etc.) are first trained and used to make the predictions which are used as input for another algorithm. However, can I use any kind of algorithms, or is there a preference?
P.S.: I often see people using linear models in this case.


texts: How to increase accuracy and decrease loss of my model

texts:
https://jovian.ai/casella0798/badmodel
I created the model above to predict red wine quality. I have 6 classes, from 3 to 8. Dataset is unbalanced, with a lot of classes 5 and 6.
My model performs really bad, with a low accuracy, and a loss that does not decrease under 1.
How can I improve this model?
I tried changing #layers and #neurons, also the batch_size and the learning rate. Any suggestion, also related to hyperparameters optimization?
Dataset is available here: https://archive.ics.uci.edu/ml/datasets/wine+quality it is the red version


texts: Integer programming formulation: which algorithms

texts:
I have a complex problem that I have simplified coming to a simple integer linear programming formulation.

Given the scalar $K > 0$, the vectors $v(t) \in R^n$ and $b_i \in R^n, \forall i=1,\ldots,K$ are known. In particular $v(t)$  is the measured value (ex: every 10 minutes) while the $b_i$ are fixed (computed ones, previously).
Consider that I have: $$K >> n,$$
i.e., I have to identify the presence of $K$ (ex: $K=20$) elements, each one of dimension $n$ (ex: $n=2$).

So, for every fixed $t$, the problem that I have to solve has the following formulation:

$$ \newcommand\norm[1]{\left\lVert#1\right\rVert}
\begin{equation*}\begin{aligned}
& \underset{x_i}{\text{minimize}}
& & \norm{v - \sum_{i=1}^{K} b_i x_i}_{2} \\
& \text{subject to}
& & x_i \in \{0,1\},\ \forall i = 1, \ldots, K.
\end{aligned} \end{equation*}$$

Do you know which algorithms solve that formulation? In particular I am looking for an algorithm implemented in Python.


texts: Notation of Transposed Convolution Operation in Equation

texts:
How do I notate a transposed convolution operation (as it is used in deep learning), in a math equation?
A convolution operation for example is often notated as $\hat{y} = x \circledast W$ where $W$ is the weight and $x$ the input. Is there a similar notation of the transposed convolution operation?


texts: Why does a belief network need to be represented using a directed acyclic graph (DAG)?

texts:
I would have thought that it's because DAGs preserve the dependency relationships between the variables, but I am currently unsure.

Thanks


texts: Null-invariant measures of association in R

texts:
A usual way to find association rules in R is the "arules" package, which easily let's use calculate some rules based on the apriori algorithm.

However, for the data i'm using, I have a lot of NULL cases (baskets where no product A o B is present). This means that I need to calculate some null-invariant measures (kulcynski, for instance).

Does anyone know of any package or workable code that let's me implement this as opposed to writing from scratch the entire algorithm?


texts: Detecting specific points in (noisy) dataset

texts:
In my recent work I've came across a problem where I need to find certain points in quickly oscillating data. Let's work with syntethic data instead of real measurements and ignore the noise for the time being. These points are called stationary phase points (SPP for short). For human eye they are quite easy to spot: the oscillation slows down locally. An example:

Some important things to note:

It's always a local maxima or minima (so a reliable peak detection algorithm can help a lot, which I've been  working on recently)
The graph is ideally normalized to [-1, 1], in real measurements it's a little distorted
Usually the Nyquist–Shannon sampling theorem is not satisfied at the borders of data and the oscillation is avaraged out. (In the synthetic data I created it's satisfied, see below for a real-world dataset.)
The general number of SPPs are 1 or 2, 3 or 4 are really rare, but exist.
A SPP can continiously move around in the x direction as conditions are changing, can vanish at the borders of x-axis, and also one SPP can separate into two different SPPs which are moving away from each other.

The main goal would be reliably monitoring these points movement (ideally in real time, but first the SPP detection from x-y data is the target).
There are some other generated graphs with indication for SPPs, and the last is from real data.

To solve this, my first thought is to detect all the extremal points in data, then generate their consecutive distances, and where that distance exceeds a certain threshold value, it's a SPP. But setting a good threshold, correctly identifying extremal points seems quite hard and unique for each graph.

My question is:
 Do you think it would worth using a machine learning tool there, and if yes, which algorithm would be the most suitable? As a physicist I'm inexperienced, I only have basic knowledge about ML (but willing to learn!)
In general, can you think of any useful algorithm or idea for such situation?

Note: Mostly I'm working in Python with numpy/scipy.

As far as local max and min detection concerned I use scipy.signal.find_peaks and scipy.signal.find_peaks_cwt along with scipy.signal.savgol_filter.


texts:
I have a problem I would like to solve using machine learning. I would like to use some sort of classification to know if a just added change in a tree data structure is "good" or is "bad".

Let's say I have this tree:

        (A)
        / \
       /   \
     (B)   (C)

And I apply a change to it (a "good" change, so the algorithm should associate this change with the "good" changes). The updated tree would be like this:

       (A)
       / \
      /   \
    (D)   (C)
    /
   /
 (B)

Added a certain node (D) above another node (B) would be classified as a "good" change.

So when I have the learner with the correct data, the algorithm should be able to know that if I add a node of type D above a node of type B, it is a "good" change.

I would like to work with XML files that keeps the tree structure, a simple classifier like a naive bayes would not work, because it wouldn't be able to recognise if a node is added above another one, it only would be able to know that a node has been added.

I don't know how which algorithm/technique I should use and I don't know how I should pass the data to the learner, because the context in this scenario is important.

What is the best technique/algorithm to compare trees changes?


texts: unsupervised outliers detection - possible solution?

texts:
I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $

The goal is to identify anomalies in transactions data of traders. For example to find suspicious transactions.

As I dont have information how real outliers look like so it is unsupervised algorithms.
As usual unsupervised algorithms I tried k-mean, z-score, lot, one class svm and different things with moving average etc.

Ive got results, but I was able to use only volume and date for every trader.

Is there any algorithm that can help me to include sector data/stock data/buy-sell data to do my model better? Can I include this information in above unsupervised algorithms or Im missing something?

How would you tackle this task, are there any other solutions maybe?

Thanks in advance.


texts: How does the inertia cython implementation in scikit-learn for kmeans work?

texts:
Specifically, what do the & symbol stand for? and why is the column index always 0?
    cpdef floating _inertia_dense(
        np.ndarray[floating, ndim=2, mode='c'] X,  # IN
        floating[::1] sample_weight,               # IN
        floating[:, ::1] centers,                  # IN
        int[::1] labels):                          # IN
    """Compute inertia for dense input data
    Sum of squared distance between each sample and its assigned center.
    """
    cdef:
        int n_samples = X.shape[0]
        int n_features = X.shape[1]
        int i, j

        floating sq_dist = 0.0
        floating inertia = 0.0

    for i in range(n_samples):
        j = labels[i]
        sq_dist = _euclidean_dense_dense(&X[i, 0], &centers[j, 0],
                                         n_features, True)
        inertia += sq_dist * sample_weight[i]

    return inertia


texts: What is Continuous Ranked Probability Score (CRPS)?

texts:
I came across some evolution metric at Kaggle:

Continuous Ranked Probability Score (CRPS):

Mathematically,

$C = \frac{1}{199N} \sum_{m=1}^{N} \sum_{n=-99}^{99} (P(y \le n) -H(n - Y_m))^2,$

where P is the predicted distribution, N is the number of plays in the test set, Y is the actual yardage and H(x) is the Heaviside step function $(H(x)=1$ for $x≥0$ and zero otherwise).

Can anyone be kind enough to explain this evolution metric and implementation with Python?


texts: When does it make sense to choose gradient descent for SVM over liblinear?

texts:
I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver.
But in the case that you are not using a kernel and simply treating it as a linear separation problem, when does it make sense to use gradient descent as your solver?
As I see it, liblinear is $O(N)$ time and doesn't require hyperparameter tuning.
In some past tests, liblinear has converged to a lower error rate at much faster rate than gradient-based methods.
Yet, Sklearn's own tests show it can be faster in many cases.
When is it optimal to use gradient-based methods with the SVM? Is it with a certain sized dataset or data that is highly linear and convex? What heuristics or explanations are available?


texts:
I am looking at this tutorial: https://www.dataquest.io/mission/75/improving-your-submission

At section 8, finding the best features, it shows the following code.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif

predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked", "FamilySize", "Title", "FamilyId"]

# Perform feature selection
selector = SelectKBest(f_classif, k=5)
selector.fit(titanic[predictors], titanic["Survived"])

# Get the raw p-values for each feature, and transform from p-values into scores
scores = -np.log10(selector.pvalues_)

# Plot the scores.  See how "Pclass", "Sex", "Title", and "Fare" are the best?
plt.bar(range(len(predictors)), scores)
plt.xticks(range(len(predictors)), predictors, rotation='vertical')
plt.show()

What is k=5 doing, since it is never used (the graph still lists all of the features, whether I use k=1 or k="all")?
How does it determine the best features, are they independent of the method one wants to use (whether logistic regression, random forests, or whatever)?


texts: Traversing trees in SQL: JOINs vs imperative algorithm

texts:
I have a table representing posts on a message board. Posts may or may not have parents.

What is the most common way to get all posts starting from a given post, or to find the root post of any given post?

I can think of using JOINs to join parents and children, but how can I know how many joins I need before I stop?


texts: I have developed an iterative process via which I can collect data in batches. The data are points in a predefined 3D space. I am trying to explore and locate clusters in that 3D space based on my data. After some batches have been collected I can locate and create the clusters. As I collect even more data though, small refinements are being made and the new results are not really worth the effort. How can determine whether a new batch is "worth" collecting? Is there some metric I can use to measure how much the batches I have are "similar" or whether a new random batch will affect the overall process?


texts: Predict the loss amount for insurance policies using the historical trends and features?

texts:
Anyone who has worked with insurance policy datasets, please guide me to an appropriate dataset for this problem .

Also, what is this 'loss amount' and what model will be appropriate for it?


texts:
What is the best way to regularize latent embeddings, I have two solution in my mind but I'm not sure which one to use over other.

In batch-wise training regularize over the whole embedding matrix, but this would be too expensive computation wise
Regularize over the item embeddings in the current batch but as real world datasets tends to have non-uniform distribution of items in the dataset, perhaps model will penalize commonly occurring items more over the long tail items

Is there any better way to deal with such cases or any improvisation over the above methods?


texts: Correlation between time to event data and continuous data

texts: I want to measure the correlation between the survival time which is a time to event data and the patient's activity count which is measured on continuous scale. What type of correlation coefficient is available to measure the strength of these two variables?


texts: I am trying to understand sentiment analysis and how to apply it using any language (R, Python etc). I would like to know if there is a good place on internet for tutorial that I can follow. I googled, but I wasn't very much satisfied because they were not tutorials but more of theory. I want theory and practical examples.


texts: Search for similar wikipedia articles based on a set of keywords

texts:
I want to solve two questions:

Which wikipedia articles could be interesting to me based on a list of keywords that are generated by the search terms I normally use in google(received by google takeout)?
Which wikipedia articles could be interesting to me based on what is not on a list of keywords that are generated by the search terms I normally use in google(received by google takeout)?

I am looking for a how to do context search on wikipedia articles - preferrably via api so I don't have to download and process terabytes of wikipedia articles - using a/the mentioned set of keywords.


texts:
Oversampling of under-represented data is a way to combat class imbalance. For example, if we have a training data set with 100 data points of class A and 1000 data points of class B, we can over sample the 100 A data (may be with some sophisticated oversampling methods) to generate 1000 A data to mitigate the data imbalance.

Now, let's say we have 1100 data points of class B, and class A has 2 subclasses, A1 and A2, which have 100 and 10 data points, respectively. And we are still interested in binary classification.

In this case, how should I over sample data of class A to address class imbalance? Should I over sample A1 to 1000 and A2 to 100, or over sample both A1 and A2 to 550?

Besides running an experiment, is there any theoretical analysis of this kind of class imbalance problem?


texts: Why predicted proababilities from this binary classifier does not sum up to 1?

texts:
I have a C5.0 model that is trained to predict binary class (C1/C2) on a dataset with 20 features. The model is configured to perform boosting (10 trials) and it has a miss-classification cost function (100:1 where 100 is the cost for miss-classifying a Negative sample as Positive and 1 is the cost for miss-classifying Negative as Positive).

Looking at predicted probabilities generated by the model, I can see that it ranges from 0 to 1 for each class. i.e, I have instances where the predicted class (C1) has a probability lower than 0.5 ( for example: predicted class=C1 and predicted probability=0.1 ) - This is where the question arises: if P(C1) < 50% why is it classified as C1 (there are only two classes, C1 and C2)

Based on my understanding of decision trees, the predicted probabilities are often generated based on the percentage of test cases on the leaf node that were labeled in each class divided by the total number of instances hitting that leaf node. This method will dictate the probability for two classes must sum up to 1.

My question is why does the model classify an instance in class C1 if it has only got 0.1 confidence in it. Does a predicted probability of 0.1 on class C1 mean that there is a 0.9 confidence in it belonging to class C2? If so, why does it classify an instance in class C1 is it has less than 0.5 confidence in it?

My own theory is that this might be due to boosting and miss-classification cost and the way they are influencing the predicted class.


texts:
Suppose I have a data frame

name = ['A', 'B', 'C']
score = [2,4,6]

I want to create a scatter plot with the following conditions, color the bubble as green if the score is greater than 3 and red otherwise. I'd also like to label the bubble with its respective name.

I'm only able to create a scatter plot with the bubble having the respective name.


texts: Decision tree and random forest over fitting

texts:
I am working on a real state data set to predict the price of buying a house in Dubai based on area, no.of bedrooms, number of baths and the town which the house is in.

All variables are numerical except the town variable which I transformed, using one hot encoding and target transform.

Then I applied linear regression, decision tree, random forest and I got the same results for both transforming methods.

Algorithm            Train score        Test score

linear regression    0.50               0.45
decision tree        0.93               0.79
random forest        0.94               0.77

From the results, I can read(but not so sure) that the data has information and has a high correlation scores but the model is over fitting.
I used grid search to optimize the hyper parameters of the decision tree but the result did not improve.

So, the question is, what am I doing wrong?


texts: Can you explain what is finite precision? Why is finite precision a problem in machine learning?


texts:
I'm using XGBoost (through the sklearn API) and I'm trying to do a binary classification.

False Positives are much worse for me than False Negatives, how can I take this into account?

The API confuses me a bit and I found two arguments that might be related to this:

objective & eval_metric

Which one is it? And how to use it?


texts: Why does English ELMo model give embeddings for non-English words?

texts:
Here's the code from my notebook:
%tensorflow_version 1.x
import tensorflow as tf
import tensorflow_hub as hub

elmo = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
tf.logging.set_verbosity(tf.logging.ERROR)

def elmo_vectors(x):
    embeddings = elmo(x, signature="default", as_dict=True)["elmo"]
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        sess.run(tf.tables_initializer())
        return sess.run(embeddings)

Output for non-English language: (Hindi in this example)
words = ['गोकुल']
v = elmo_vectors(words)
print(v.shape) # (1,1,1024)
print(v[0][0])
# Output: [ 0.3731584   0.5700774  -0.48072845 ... -0.1241736   0.5961436 -0.6986947 ]

The documentation of the pre-trained ELMo on Tensorflow Hub shows that it was trained only on the English language.
That is, the dataset from 1 billion word benchmark is based on monolingual English data. (Source)
So, how/why am I getting embeddings for non-English vocabulary words from ELMo using the TF Hub model?


texts: How to plot prediction intervals in seaborn tsplot?

texts:
I am plotting a group of time series plots with CI's in seaborn using tsplot:

These were produced from 10 traces each:

The confidence bars shown (99% confidence) appear to be the CI's for the mean. However I am really after 99% confidence for the prediction interval. For the examples above these should be much wider and cross the red dotted line on occasion.

Can I plot 99% prediction intervals in seaborn tsplot?


texts:
I noticed that feature scaling can destroy completely neural networks performance in some cases. Below are my results that you can reproduce easily.

I use a neural network to approximate function $g$ on $[0,100]$ as follows:

$\mathcal{N}\mathcal{N}(X; \theta) \approx g(X)$

where $X \sim \mathcal{U}([0,100])$. Then I randomly sample some $X$ ($(X_i)_i$) to be my training set, and use $Y_i=g(X_i)$ as targets.

Typically $\mathcal{N}\mathcal{N}$ is a Keras Sequential model, and has a few hidden layers with 15-20 neurons (elu), and a single output layer with no activation function. Parameters are initialized using kernel_initializer='normal'.

Now I noticed the following while testing my approximations in 1dim (before trying to generalize to higher dimensions):

For $g(x)=\max(0, X-50)$, scaling $X$ by dividing it by 100 does not really change the performance of the $\mathcal{N}\mathcal{N}$, which estimates $g$ relatively well.

However, when I use other functions such as $g(x)=1-\mathbb{I}_{x\in[40,60]}$ or $g(x)=(x-50)^2$, it fails completely.

Do you have an explanation?

This is very disturbing because how can you trust your network to approximate high-dimensional functions if it fails completely for simple 1-dimensional examples?

Below is an example implementation in Python:

from keras.models import Sequential
from keras.layers import Dense
import numpy as np
import matplotlib.pyplot as plt

d=1
nn_value_batch_size=128
nn_value_epochs=50
nn_value_training_size=10000

def nn_value_get_architecture(layers_sizes):
        nn_value = Sequential()
        nn_value.add(Dense(layers_sizes[0], input_dim=d, #d=1
                        kernel_initializer='normal', activation='elu'))

        # hidden layers
        for nb in layers_sizes[1:]:
            nn_value.add(Dense(nb, kernel_initializer='normal',
                               activation='elu'))

        # output layer
        nn_value.add(Dense(1, kernel_initializer='normal'))

        nn_value.compile(loss="mean_squared_error", optimizer='adam')
        return nn_value

def nn_value_optimize(nn_value, objective_function,
                          x_normalize=True):
        # fit the model
        x_distribution = lambda size: np.random.uniform(0, 100, size)
        for epoch in range(nn_value_epochs):
            X_train = x_distribution(nn_value_training_size)
            y_train = objective_function(X_train)
            if x_normalize==True:
                X_train = X_train/100
            nn_value.fit(X_train, y_train,
                         batch_size=nn_value_batch_size,
                         verbose=1)

        return nn_value

def g(X):
        #return 1-(np.maximum(0., X-40)/(X-40) - np.maximum(0., X-60)/(X-60))
        return (X-50)**2
        #return np.maximum(0., X-50) #only example that works

nn_value = nn_value_get_architecture(
                layers_sizes=[20,15,15,15,15])
g = np.vectorize(g)
normalize=True
nn_value = nn_value_optimize(nn_value, g, normalize)

# plot results
x = np.arange(0,101,1)
if normalize==True:
  plt.plot(x, nn_value.predict(x/100))
else:
  plt.plot(x, nn_value.predict(x))
plt.show()

No scaling:

With scaling:


texts: What is the difference between public test and private test on Kaggle

texts: I am trying to use the datasets from a competition held on Kaggle in which the dataset contains fer2013.csv file having columns emotion, pixels, and usage where usage contains 3 values i.e "training", "privatetest" and "publictest". So I am not getting what is publictest and privatetest. It would be appreciated someone could explain to me what it is?


texts: Multinomial Naive bayes giving wrong result

texts:
from sklearn.naive_bayes import GaussianNB,MultinomialNB
xx = [[1],[1],[1],[2],[2],[3]]
yy = [1,1,1,0,0,0]
clf = GaussianNB()
# clf = MultinomialNB()
clf.fit(xx,yy)
clf.predict(xx)

The expected result is [1,1,1,0,0,0]
but code output is [0,0,0,0,0,0].


texts: Are GPTs close to real intelligence or just another Data In -Data Out -Data Permutation and Combinations?

texts: Use cases and Solutions surrounding GPT's have taken NLP world with storm and started the GPT-Best vs GPT- Not So Best war on the internet. There are solutions been derived from API's provided by HF. What would be the impact and how feasible would be GPT development and deployment in the real world use case across domains?