texts: Python Code to find the number of hapax legomena in a Text or Words_List

texts:
In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, either in the written record of an entire language, in the works of an author, or in a single text. The term is sometimes incorrectly used to describe a word that occurs in just one of an author's works, but more than once in that particular work. Hapax legomenon is a transliteration of Greek ἅπαξ λεγόμενον, meaning "(something) being said (only) once"

Hapax_legomenon
enter link description here


texts: Decision Tree Regressor: domain of the y variable

texts: just wondering about a thing. suppose you fit a Decision Tree Regressor and your training y variable has got a domain that spans from 0 to 1 included, is it possible that the predicted y variable of the fitted model on the test set to exceed these boundaries?


texts: Classifying whether a comment or review is a complaint or appreciation of product and extracting the Topic?

texts:
I need to classify whether a given review or comment is a complaint or appreciation. This is planned to be used in multiple places, product review pages of own site as well as facebook and twitter. Suggestions on how to approach please.

The Problems that are confusing me:

In FB/Twitter I don't know which product it is for, I need to extract that from text as well.
I need to extract the complaint/appreciation part and group similar ones together, (like good color reproduction and great clarity into just good display)
Articles(each document) will be differently sized.
Data availability is none, I will prepare data by going through our FB etc.

My initial thought was LSTM based classification, but point 3,4 make that hard. Even with 3,4 solved. How do I go about 1,2. I only have played with word2vec a bit and done some twitter sentiment analysis dummy projects. Point 1,2 seem Information Retrieval, Need pointers for that.


texts: how to make new class from the test data

texts:
I have a list of accounts as data set and I need to group the accounts that refer to the same user using many features.

I'm thinking to use machine learning( but I'm new in this domain), because I know the group of each account for the training data set.
ex of training data:

account-id   Feature1    Feature2    class(Group)
1            T1          P4          Gr1
2            T2          P4          Gr1
3            T3          P2          Gr2

The problem is in the testing of data and when a new account arrive for a new group not learned before in the training set.

ex of testing data:

account-id   Feature1   Feature2
4             T5         P5
5             T6         P5
6             T3         P2

The groups of the testing data should be as following:

account-id   Feature1   Feature2   class(Group)
4             T5         P5         Gr3
5             T6         P5         Gr3
6             T3         P2         Gr2

The accounts 4 and 5 are in a new group (Gr3) which is not learned before in the training data.

My question is how could I group the new data under a new class that is not defined before in the learning phase ? and which algorithm can I use to solve this issue ?


texts: monte carlo simulation for impact to revenue by adjusting credit limits

texts: I've been tasked with prioritizing projects for the year. One of them is to estimate the impact to net revenue if we were to adjust the purchase limits for our users. Our platform is transactional in nature - our users can purchase items and provided they're not high risk, we float them money that may take time to deposit. From some initial research it feels like I should be able to leverage monte carlo simulations but I'm not quite sure how to go about it. I want to be able to say, "if we adjust the limit to X, on average, here's the impact to net revenue" where X can be in some range. I have all the transactional data, the limits for each user and the net revenue from each user. Any help would be greatly appreciated!


texts: Train error vs. Test error in linear regression by samples analysis

texts:
I have run a multivariate linear regression model on a small set of about 3500 samples. While the model's error is as large as expected, I also ran a bias vs. variance analysis by comparing the train set error vs. the test set error using different sample sizes. I was expecting something like this:

But instead I found that the train error doesn't plateau at any point.

The code that generates this graph is the following:
def loss_by_sample(X, Y, testX, testY, learning_rate):
    samples_list = list()
    train_loss_list = list()
    test_loss_list = list()
    m = X.shape[0]
    for i in range(50, len(X), 50):
        samples_list.append(i)
        weights, loss = gradient_descent(learning_rate, X[:i], Y[:i])

        X2 = tf.concat([tf.ones([X.shape[0], 1]), X], 1)
        train_loss = (1/(2 * m) * tf.tensordot(tf.transpose(h(X2[:i], weights) - Y[:i]), (h(X2[:i], weights) - Y[:i]), axes=1))[0][0]
        train_loss_list.append(train_loss)

        X2 = tf.concat([tf.ones([testX.shape[0], 1]), testX], 1)
        test_loss = (1/(2 * m) * tf.tensordot(tf.transpose(h(X2, weights) - testY), (h(X2, weights) - testY), axes=1))[0][0]
        test_loss_list.append(test_loss)
    // plot train_loss, test_loss

I have tried different learning rates and the one I picked is the one that minimizes the loss using mean squared error:

(values to the right of the x axis after the last datapoint are inf.
Is there something I can conclude from this? Any reason why the train error can surpass the test error?


texts: Data representation (NoSQL database?) for a medical study

texts:
Problem description

I have a data set about 10000 patients in a study. For each patient, I have a list of various measurements. Some information is scalar data (e.g. age), some information is time series of measurements, some other information can be even a bitmap. The individual record itself can be quite thick (10kB to 10MB). The data is to be processed practically in two steps:

Preprocessing at the level of individual records (patients), i.e. to extract some features in raw data, store them. Calculate some slopes in time series etc. All this can be done on individual level and it can be very easily distributed.
On top of the preprocessed data (extracted features), I will need to calculate some aggregated things such as e.g. average age, but also some machine learning tasks.

The question

Obviously, this is very suitable to be addressed in Apache Spark (or any map-reduce architecture). At the most general level, my question is: what is the most appropriate NoSQL database for this situation?

So far, I have considered two basic options:

MongoDB - to take advantage of the document oriented storage where everything is on the same place. However, I am not sure about the performance on the larger binary data (pictures, time-series).
Cassandra -  this may have some better storage of binary data, but the joins will be necessary (even if optimized by indexing all data by "patient id").


texts:
I understand boosting is a sequential learning technique and it use the prediction from previous model as a dataset for new model ,after adding weight to the misclassified data points.
The point which was not clear how the weights are added for misclassified ones and diminished for the correctly classified ones. It would be great if veterans can help me to understand this

Thanks in advance


texts: How to binary encode multi-valued categorical variable from Pandas dataframe?

texts:
Suppose we have the following dataframe with multiple values for a certain column:

    categories
0 - ["A", "B"]
1 - ["B", "C", "D"]
2 - ["B", "D"]

How can we get a table like this?

   "A"  "B"  "C"  "D"
0 - 1    1    0    0
1 - 0    1    1    1
2 - 0    1    0    1

Note: I don't necessarily need a new dataframe, I'm wondering how to transform such DataFrames to a format more suitable for machine learning.


texts: My ADALINE model using Gradient Descent is increasing error on each iteration

texts:
I have used the Iris Dataset's 1st and 3rd Column for the features. and the labels of Iris Setosa (-1) and Iris Versicolor (1). I am using ADALINE as a simple classification model for my dataset. I am using gradient descent as the cost minimizing function. But on every iteration the error increases. What am I doing wrong in the python code?

import numpy as np
import pandas as pd

class AdalineGD(object):

    def __init__(self, eta = 0.01, n_iter = 50):
        self.eta = eta
        self.n_iter = n_iter

    def fit (self, X, y):
        """Fit training data."""

        self.w_ = np.random.random(X.shape[1])
        self.cost_ = []
        print ('Initial weights are: %r' %self.w_)
        for i in range(self.n_iter):
            output = self.net_input(X)
            print ("On iteration %d, output is: %r" %(i, output))
            errors = output - y
            print("On iteration %d, Error is: %r" %(i, errors))
            self.w_ += self.eta * X.T.dot(errors)
            print ('Weights on iteration %d: %r' %(i, self.w_))
            cost = (errors**2).sum() / 2.0
            self.cost_.append(cost)
            print ("On iteration %d, Cost is: %r" %(i, cost))
            prediction = self.predict(X)
            print ("Prediction after iteration %d is: %r" %(i, prediction))
            input()
        return self

    def net_input(self, X):
        """Calculate net input"""
        return X.dot(self.w_)

    def activation(self, X):
        """Computer Linear Activation"""
        return self.net_input(X)

    def predict(self, X):
        """Return class label after unit step"""
        return np.where(self.activation(X) >= 0.0, 1, -1)

####### END OF THE CLASS ########
#importing the Iris Dataset
df = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data", header = None)
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)
X = df.iloc[0:100, [0, 2]].values
#Adding the ones column to the X matrix
X = np.insert(X, 0,  np.ones(X.shape[0]), axis = 1)
ada = AdalineGD(n_iter = 20, eta = 0.001).fit(X, y)


texts: Is a 100% model accuracy on out-of-sample data overfitting?

texts:
I have just completed the machine learning for R course on cognitiveclass.ai and have begun experimenting with randomforests.

I have made a model by using the "randomForest" library in R. The model classifies by two classes, good, and bad.

I know that when a model is overfit, it performs well on data from its own trainingset but badly on out-of-sample data.

To train and test my model I have shuffled and split the complete dataset into 70% for training and 30% for testing.

My question: I am getting a 100% accuracy out of the prediction done on the testing set. Is this bad? It seems too good to be true.

The objective is waveform recognition on four on each other depending waveforms. The features of the dataset are the cost results of Dynamic Time Warping analysis of waveforms with their target waveform.


texts: I am doing a project on fruit disease recognition and classification. Anyone have an existing dataset of fruit diseases? Can you help me to find one?


texts: Gensim LDA model: return keywords based on relevance (λ - lambda) value

texts:
I am using gensim library for topic modeling, more specifically LDA. I have created my corpus, my dictionary and my lda model, and with the help of pyLDAvis library I visualize the results. When I print the words with the highest probability on appearing to a topic with pprint(lda_model.print_topics()) I have results for the first topic similar to:

0.066"car" + 0.032*"gas" + 0.031*"model" + 0.031*"top" + 0.024*"CO2" + ... + 0.012*"investment"*

The results are good as are indicative about the topic, but when I interact with the relevance parameter (λ - lambda value) provided by pyLDAvis, I can have results that are more specific about the topic, for example setting λ=0.2 the top 5 words are:

car, horsepower, torque, speed, V8

My question: is there any function or parameter in gensim that can return the pair probability - word given a specific lambda value?

Thank you in advance and I am sorry if I have missed an already open thread.


texts: I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style embeddings to feed the entities to a CNN, but I was wondering if a Transformer (à la BERT) would be a better alternative & provide a better way of capturing the semantics of the entities involved. I can't seem to find any articles (let alone libraries) to apply sth like BERT to non-textual sequence data. Does anybody know any papers about this angle? I've thought about training a BERT model from scratch and treating the entities as if they were text. The issue with that though is that apparently BERT is slow when dealing with long sequences (sentences). In my data I often have sequences that have a length of 1000+  so I'm worried BERT won't cut it. Any help, insights or references are very much appreciated! Thanks


texts:
I am trying to create a PivotChart in the form of a bar chart and I would like to add a group average line on the chart.

I have this (sample) table in my PowerPivot Data Model:

Group averages are A: 20, B: 50 and C: 80. My bar chart looks like this.

How would I do this? Do I need to add a measure? If so, how? I am not that great with DAX functions yet.


texts: Better way to deal with downsampled MNIST images

texts:
model = tf.keras.models.Sequential([
    tf.keras.layers.MaxPool2D(4, 4, input_shape=(28,28,1)),
    tf.keras.layers.Conv2D(32, (5, 5), padding='same', activation=tf.nn.relu),
    tf.keras.layers.MaxPool2D(2, 2),
    tf.keras.layers.Dropout(0.25),

    tf.keras.layers.Conv2D(128, (3, 3), padding='same', activation=tf.nn.relu),
    tf.keras.layers.Conv2D(128, (3, 3), padding='same', activation=tf.nn.relu),
    tf.keras.layers.MaxPool2D(2, 2),
    tf.keras.layers.Dropout(0.25),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(512, activation=tf.nn.relu),
    tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])

optimizer = tf.keras.optimizers.RMSprop(lr=0.00020, rho=0.99, epsilon=1e-8, decay=0.0)

model.compile(optimizer=optimizer,loss='sparse_categorical_crossentropy',metrics=['accuracy'])

So, the MNIST images are downsampled from 28*28 to 7*7 from the first line. Using that,I want to get a good accuracy and the maximum I'm getting is 89% with 40 epoch and 6000 test images. How can I improve this without removing the first line?


texts:
One of the problems I often encounter is that of poor data provenance.

When I do research I continuously make modifications to my code and rerun experiments. Each time I'm faced with a number of questions, such as: do I save the old results somewhere, just in case? Should I include the parameter settings in the output filenames or perhaps save them in a different file? How do I know which version of the script was used to produce the results?

I've recently stumbled upon Sumatra, a pretty lightweight Python package that can capture Code, Data, Environment (CDE) information that can be used to track data provenance. I like the fact that it can be used both from the command line and from within my Python scripts and requiring no GUI. The downside is that the project seems inactive and perhaps there's something better out there.

My question is: what is a good lightweight data provenance solution for my research? I'm coding small projects mostly in Python in the terminal on a remote server over SSH, so a command line solution would be perfect for me.

EDIT 2019: The Sumatra project seems inactive and other, more mature project have come along. DVC looks very promising and I've been in contact some of its authors, who have proven very helpful and supportive.


texts: What happens if GBM parameters (e.g., learning rate) vary as the training progresses?

texts:
In neural networks there is an idea of a "learning rate schedule" which changes the learning rate as training progresses.

This made me ask the question, what would be the impact of varying parameters in a GBM as a function of the number of trees?

Take the learning rate for example.  For GBMs using the MART algorithm, the contribution of each tree is weighted by a function of the error and the learning rate.  Trees fit early on have a higher impact; trees fit later on have less impact.  What if the learning rate was a function of $N$ such as $\exp(-a N)$ where $a$ would be the decay parameter of the learning rate?

Other parameters could vary as well.  For example the max depth of each tree could start out high and then decrease as training progresses.  Going beyond just the tree parameters, other examples are the subsample percentage if using bagging or parameters of a loss function (e.g., Huber loss parameter $\delta$).


texts: CNN Multi-class vs Binary Class Image Classification

texts:
Suppose we have a training set of 3 classes of image: 1.Cats, 2.Dogs, 3.Neither cats nor dogs.
We're only really bothered about detecting whether an image is a cat/dog, or neither, but we don't care whether it's a cat or a dog, just that it's one of the two.

Is it inherently "better" to train a binary class CNN by merging the Dog and Cat classes into a single "positive" class rather than train a 3-class CNN?

If it is, does it then follow that if we did care about whether the image was a dog or cat, it would also be better to first run the binary classifier to decide if it was a dog/cat or not, then run a second binary classifier trained on just cats and dogs to decide which of the two it actually is?


texts:
It's clear how LSTM works with 1 feature. But what happens if the number of features is > 1?

According to the answer proposed here,

Keras creates a computational graph that executes the sequence in your bottom picture per feature (but for all units). That means the state value C is always a scalar, one per unit.

But if it executed the process per feature, the result would be number of units x number of features. Instead, we get only the number of units (if return_sequences=False; otherwise, number of timesteps x number of units).
What happens to features and how are they processed? At which step and how are they merged?


texts: Interpreting lasso logistic regression feature coefficients in multiclass problem

texts:
I have a dataset with a large number of text features, where the target variable has three classes.

I have encoded the features using tf-idf. This has resulted in a dataset with an extremely large number of features.

I have tried performing classification using lasso logistic regression (in sklearn, LogisticRegression(penalty='l1')), which automatically sets some features' coefficients to 0 and thus performs some kind of feature selection for me.

The sklearn implementation of LogisticRegression() performs multiclass classification using 'one-versus-rest'. That is to say, it trains three separate classifiers which predict one particular class against the other two, and then assigns each data point to the class whose predictor gave the highest probability to.

The coefficients of some of the features from my dataset were set to $0$ in all three classifiers. Others, however, are set to non-zero numbers in all three.

Is it correct to think of those with coefficients set to $0$ by all three classifiers as uninformative, in the sense that they add no new information given the presence of all the others?

Does it make sense to compare the magnitude of the non-zero coefficients, across the three classifiers, in order to find out which features are most informative?


texts: Relation of Tensors to artificial neural networks

texts:
When dealing with artificial neural networks, one is bound to come across the concept of Tensors. In such applications a Tensor is essentially just a data grid, just like a scalar (rank=0), a vector (rank=1) or a matrix (rank=2), except it has any finite rank. However, this is not how a Tensor is actually defined. In mathematics a Tensor is an abstract multilinear function which maps one algebraic object to another. While all matrices (or "data grids") always represent a certain type of Tensor, I fail to see how they semantically represent a function in case of artificial neural networks, as they are essentially only used as data storage for computations in between the computation graph.

I found that in tensor-network-theory, a network of Tensors is used to represent a geometrization of the brain, but in case of artificial neural networks, there is no actual network to begin with. To me an artificial neural network is just some function which maps one algebraic object to another and consists of a concatenation and combination of piecewise partially differentiable functions.

So my question is if Tensors as used in artificial neural networks actually have any semantic meaning. Do they represent some kind of function in relation to the inputs? Are artificial neural networks themselves Tensors (or tensor fields)? Or is the term Tensor just used due to lack of a proper name for a data grid without any other meaning?


texts: How exactly do I go about extracting features from timestamps for machine learning?

texts:
My dataset has a timestamp column with the following format: 06/24/18 0:56
How exactly do I convert this information into features that can be used for classification algorithms like logistic regression?


texts: How to visualize Ensemble Models ( Random Forest) with 1000 estimators

texts:
I am working on classification problem where I need to categorize  the user in buy/ non-buy category. I have around 100 + features or predictors to predict the behavior of user.

I tried to implement with Random forest and Gradient Boosting to get better prediction compare to decision tree. I am getting better performance when I evaluate against performance parameters like roc_auc,accuracy, precision and recall when using ensemble techniques.

I also extracted important features that are responsible for my predictions but I am not able to visualize the model fully. Some how random forest works as black box where i am not getting what is the contribution of each tree, which features are been considered in each trees, etc.

Is there any way through which I can find out more information from  Random Forest Model?


texts: Dataset of extremely low-dimensional images for PCA

texts: I am looking for a public data-set of images that differ from each other only slightly, so that after applying PCA they can be reconstructed with a small error from very few PCA coefficients. It can be any type of images, the purpose is only to demonstrate an extreme example of PCA.


texts: Two-sample K–S test Kolmogorov–Smirnov test in Python

texts:
For my evaluation, I have three different time-series data of the following format with different characterstics where the first column is timestamp and the second column is the value.

0.086206438,10
0.086425551,12
0.089227066,20
0.089262508,24
0.089744425,30
0.090036815,40
0.090054172,28
0.090377569,28
0.090514071,28
0.090762872,28
0.090912691,27

For reproduciability, I have shared the three time-series data I am using here.

From column 2, I wanted to read current row and compare it with the value of the previous row. If it is greater, I keep comparing. If the current value is smaller than the previous row's value continue, I take the difference. Let me make it clear. For example in the above sample record I provided, the seventh row (28) is smaller than the value in the sixth row (40) - so it will be (40-28 =12).

Here is my sample code.

import numpy as np
import pandas as pd
import csv
import numpy as np
import scipy.stats
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import norm
from statsmodels.graphics.tsaplots import plot_acf, acf

protocols = {}

types = {"data1": "data1.csv", "data2": "data2.csv", "data3": "data3.csv"}

for protname, fname in types.items():
    arr = []
    arr1 = []
    with open(fname, mode='r', encoding='utf-8-sig') as f:
        reader = csv.reader(f, delimiter=",")
        for i in reader:
            arr.append(int(i[1]))
            arr1.append(float(i[0]))

    arr, arr1 = np.array(arr), np.array(arr1)
    diffs = np.diff(arr)
    diffs1 = np.diff(arr1)
    diffs1 = diffs1[diffs > 0]
    diffs = diffs[diffs > 0] # To keep only the increased values
    protocols[protname] = {
        "rtime": np.array(arr1),
        "rdata": np.array(arr),
        "data": diffs,
        "timediff": diffs,
    }

## change in time
for protname, values in protocols.items():
    d = values["rdata"]
    t = values["rtime"]
    d = np.diff(d, 1) #/ d[:-1]
    t = np.diff(t, 1)
    plt.plot(t[d < 0], d[d < 0], ".", label=protname, alpha=0.5)
    plt.xlabel("Changes in time")
    plt.ylabel("differences")
    plt.legend()
    plt.show()

This gives me the following plots

However, this doesn't seem to give me a good option to classify these time-series data. I wanted to do it using Kolmogorov–Smirnov test. How can do it using a two-sample KS test in Python?


texts: euclidean distance between true value and estimated value as input for knn

texts: If we apply knn on euclidean distance between true value and estimated value  will the results be better in case of data classification?


texts:
I am working on a drug reaction problem in which I need to extract tweets and label the tweets (binary-reaction due to drug or not). But since I don't have domain knowledge, and clustering would also not help me in labelling them, I thought of using semi-supervised learning as one labelled small labelled dataset of around 400 tweets is available. So I will be training my model on those 400 tweets and then make predictions on the tweets that I extracted for labelling.

I know since 400 tweets is very less so its predictions on my extracted tweets would be really poor. But ignoring that, I am facing another problem.
In NLP, we map every word in the dataset during preprocessing. Suppose bag-of-words or create word embedding, word2vec or tf-idf vector. So during prediction when I run the preprocessing on my extracted tweets, it throws an error for the new words in this extracted tweets dataset. As in my extracted dataset there will be names of new medicines or some English word that was not present in the labelled dataset. And suppose I had created bag-of-words, the structure of bag-of-words changes with the input, so in that case model prediction would be wrong as inputs (bag-of-words) are not based on the same words.

One thing that I have planned to do is combine extracted and labelled dataset and preprocess them together. Then train only on labelled and predict on extracted. This way maybe the predictions would be more accurate and it shouldn't throw any error. Still, is there any specific NLP technique that I can use in preprocessing to deal with this error and get better results?


texts: Hierarchical method for handling large class classification

texts:
I am looking for an ML algorithm which is appropriate for the following situation:

I am trying to pick out specific invoice numbers from remittance advice images which contain lots of irrelevant numbers which I wish to ignore. There are about 500,000 possible invoices to pick from, however, once I know one some information (customer name), the number of possible choices drops to a small number (between 1 and 50). There are other clues such as dates.

The possible invoice # values to pick from is continually changing, as once an invoice is paid, it will not be paid again and is no longer a possible answer.

One approach would be OCR and then try regex to match on available numbers.


texts:
Good day, I had this question set as optional homework and wanted to ask for some input.

Suppose an individual was to take a data set, divide it in half into training and test data sets and then try out two different classification procedures.
First they use logistic regression and get an error rate of 20% on the training data and 30% on the test data. Next they use 1-nearest neighbours and get an average error rate (average over both test and training data sets) of 18%. According to these numbers, which method would you prefer to use for classification purposes (of new observations)? Why?

I am inclined to say kNN as it is a rather flexible approach and provides a lower error on average. But this doesnt sound formal enough and may likely be a flawed and naive conclusion. Any input?


texts:
For a project at my university I have to develop a simple chatbot. Since I am new to machine learning, it should be really simple and work like a customer support. The chatbot has to recognize about 10 intents related to some really simple topic. For example I thought that it could give information about a party, so the questions would be something like "Where is the party?, When does the party start? How much does the entry cost?"

The chatbot should classify the question, and out of the classification it should give back a prepared answer. If it cannot assign the question to the one of the 10 intents, it should do nothing or just answer that it does not understand the question. The same may happen if the question has spelling mistakes.

So it should be really simple, but be programmed with the deep learning approach.

My question is, how much data should I have, to train this bot to assign questions to the intents? Would I be able to generate the data by my own, if I reformulate each question on, lets say 20 ways? Or do I have to  feed the algorithm with a very large amount of data, even for such simple task?


texts: Convolutional neural networks for non image dataset

texts:
Can we use Convolutional Neural networks for a non image dataset for prediction?

The dataset is a record of student academic details

I know that CNN is mostly used in  computer vision and image processing for analyzing visual imagery.
And it is also used in natural language processing for text classification.

I am new to this please guide me through

Thank you!!


texts: When to enhance a dataset of images with flips and rotations of the images?

texts:
I am a beginner in machine learning, so I'm sorry if my question is a bit trivial.

Suppose I have a dataset of images and which I want to classify, say using a neural network. It makes sense to me to try to enhance my dataset by e.g. flipping and rotating the images, so to obtain more training observations.

At some point, I want to split my dataset in a training set and a test set, and maybe also to additionally subdivide the training set for cross-validation. My question is: when should I enhance my dataset with the flipped/rotated images? If I do it before I split my dataset in training and test sample, then the test sample will contain e.g. rotations of observations that also are in the training sample, so I have the feeling it might be "contaminated" and under-represent the test error. Same thing with the split for cross-validation. Is there a consensus on how to proceed? Also, is this a valid approach?


texts: What is a good method to generate the KS-statistic in R?

texts:
Kolmogorov-Smirnov (KS) statistics is one of the commonly used measures to assess predictive power for marketing or credit risk models.

The KS statistic is usually published for logistic regression problems to give an indication of the quality of the model.

A Wikipedia page (https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) gives this explanation:

The KS statistic calculation below is from page 5 on https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf

require(ROCR)
set.seed(7)
prd=runif(1000)
act=round(prd)
prd[sample(1000,500)]=runif(500) #noise

pred<-prediction(prd,act)
perf <- performance(pred,"tpr","fpr")
#this code builds on ROCR library by taking the max delt
#between cumulative bad and good rates being plotted by
#ROCR see https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf
ks=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]])
plot(perf,main=paste0(' KS=',round(ks*100,1),'%'))
lines(x = c(0,1),y=c(0,1))
print(ks);
auc <- performance(pred, measure = "auc")
auc <- auc@y.values[[1]]
print(auc)

It gives a KS of 0.4939511 and an AUC of 0.7398465. Area under the curve is obviously correct as one can see.

Is the R code for the KS statistic correct?
If not, what should it be?

Update

Warning: From a sample decision tree output which is very clumpy, the Sharma method gives a better estimate of the KS score while the ks.test gives a very bad one.

     auc         ks.score.D^+ sharma.ks
    0.6153846    0.7045455    0.2307692

The ks.test function gives this error: 1: In ks.test(prd, act, alternative = "greater") :  cannot compute exact p-value with ties

act=c(1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1,
  0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0,
  0, 0, 1)
prd=c(0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.352941176470588,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.352941176470588, 0.352941176470588, 0.914893617021277,
  0.914893617021277) ##this only has two values
ks.test(prd, act, alternative='greater')$statistic
#gives 0.7045455 which clearly isn't correct


texts: XGBoost Feature importance - Gain and Cover are high but Frequency is low

texts:
I have read this question: How do i interpret the output of XGBoost importance? about the three different types of feature importances: frequency (called "weight" in Python XGBoost), gain, and cover.

In my case, I have a feature, Gender, that has a very low importance based on the frequency metric, but is the most important feature by far based on both the gain, and cover metrics.

I know gender should be important for what I'm predicting. If I plot only gender vs. the target, there is a clear correlation. I am not surprised that it's the most important feature. I just want to know why it's not considered an important feature based on the frequency metric.


texts: Use serialized model without installing dependencies

texts:
I prototyped an ML model consisting of preprocessing + multiple stacked regressors. I would like a colleague of mine to develop an API that will query the model. Is there any way to query the model (sklearn pipeline) without having to download all the dependencies (XGBoost, LGBM, CatBoost, ...).

I tried to serialize it with Joblib but when we deserialize it on another machine it requires to have dependencies installed.

The goal is really to transform the sklearn's pipeline to a complete inert black box that requires minimal setup. Is it possible?


texts: My question is about dependency between hidden states for Back Propagation Through Time in RNN

texts:
In one video lecture, professor Ali Ghodsi of University of Waterloo says that the first node of S(t)(hidden state of RNN at time t) has an effect only on the first node of S(t+1)(hidden state of RNN at time t+1), the second node of S(t) has an effect only on the second node of S(t+1), and so on. I do not see how that is correct cause the weight matrix W connecting the state at one time step to the state at the next time step is (p,p) dimensional which means each node of S(t) is connected to every node of S(t+1). What I have come to understand from various blogs on the internet about RNN is that nodes in the hidden state of some time step are connected to every node in the hidden state corresponding to the next time step which is contradictory to what professor Ghodsi is stating in the lecture.

What I want to clarify is whether the hidden nodes in RNN are usually connected in the same way as fully connected networks throughout the temporal dimension, or are they connected elementwise as the professor says.

I am providing a link to the video https://www.youtube.com/watch?v=AvyhbrQptHk&t=1704s where the claim is made a 43:02.

I had posted this question on StackOverflow, but that doesn't seem to be the correct platform for such topics.

I am new to this site. So, please be a little patient if you do not like the format of the question and let me know how I can improve the quality of my question.

Thanks in advance.


texts: What knowledge do I need in order to write a simple AI program to play a game?

texts:
I'm a B.Sc graduate. One of my courses was 'Introduction to Machine Learning', and I always wanted to do a personal project in this subject.

I recently heard about different AI training to play games such as Mario, Go, etc.

What knowledge do I need to acquire in order to train a simple AI program to play a game? And what game do you recommend for a beginner?

This is what I know in Machine Learning so far -

Introduction to the course and to machine learning. K-Nearest Neighbor algorithm, and K-means algorithm
Statistical Inference
Gaussian Mixture Model (GMM) and Expectation Maximization (EM)
Probably Approximately Correct (PAC) model, including generalization bounds and model selection
Basic hyperplane algorithms: Perceptron and Winnow.
Support Vector Machines (SVM)
Kernels
Boosting weak learners to strong learners: AdaBoost
Margin-Perceptron
Regression
PCA
Decision Trees
Decision Trees pruning and random forests


texts: Jupyter notebook - Converted Python dataset to R Dataframe. Got stuck in converting back from R to Python Dataframe

texts:
I am able to Install Anaconda and able to write Python and R code without any issues based on below help.
https://gist.github.com/simecek/019d87c55fec3839d95bbf8489dde61d

I extracted my data from MS-SQLServer via pypyodbc
Converted the Python dataset to R and maniputed the data (as per my usecase)
Now I am ready to convert my R dataframe back to Python -> Got stuck

The question is..is it possible to convert R dataframe to Python dataframe (as a new dataframe)

Here, my final preparations should be done via Python.

I search for above issue for 3 hours and no luck. Please help.


texts: In U-Net, is there a non-linearity (relu) in up-convolution layer?

texts:
I am doing semantic segmentatio using U-Net. I was wondering whether to include 'relu' activation or not in the up-convolution layer?
x = Conv2DTranspose(filters, kernel_size) (x)

OR

x = Conv2DTranspose(filters, kernel_size, activation='relu') (x)

```


texts: Which ML method would be best for deriving a rough formula for prediction based on existing data?

texts:
Which ML method would you say is the easiest to derive a mathematical formula from based on already existing data of predictor stats and outcome?

I have this data:

Opponent 1:

Strength: x
Battle Score: y

I also have a model that I put against the opponent:

Opponent 2:

Strength: z
Battle Score: k

Finally, all outcomes of fights are written into a database (which currently has around 2800 outcomes) and look something like this model:

Fight:

Strength: x - z
Battle Score: y - k
Outcome: win/lose

I would want to get proper weights for Strength and Battle Score, so I can derive a simple formula from it and thus somewhat predict whether the next fight will be won or lost.


texts: Convey time lag information to a linear regression model

texts:
I am using a simple linear regression to predict the number of units an item has moved and price of the item is one of the input parameters.

For a few items, the older prices are not relevant and hence this results in incorrect predictions. The definition of old price varies from item to item.
Is there a way to make the linear regression know that recent prices are more relevant?


texts: How to build a classifier for determining if two pairs are a match?

texts:
I'm trying to build a classifier that can determine if two addresses are a match or non-match.

Let's assume I have a data set of address pairs that have a match or non-match label.

I'm new to ML so something may be going over my head but is this how I'm thinking of approaching the task:

Approach for building classifier

Segment each address into it's subcomponents such as streetname, zipcode etc
Construct a comparison vector with one dimension for each field. The fields holds the value from a string similarity method such as jaro-winkler.
All my comparison vectors with an associated match label can now be fed into training a classifier

Predicting if a pair is a match

Build a list of candidate address pairs by using a method such as blocking
Construct a comparison vector the same way as the classifier was built above
The comparison vector for each candidate is now fed into the classifier and this will resolve to either a match or a non-match

My approach was inspired by research paper  Machine learning innovations in address matching: A practical comparison of word2vec and CRFs but a lot of the things are vaguely described for a beginner.

To sum up I'm looking for any input on whether this approach holds up or I'm misunderstanding something completely


texts: Decision tree on big categorical dataset

texts: I would like to use algorithm ID3 in order to find a decision tree of my dataset. I would like to see which of the attributes and values lead to the different value of rating (1<= x <= 5). Do you think it is possible to do it? I got stucked but I don't know if it is because it is not a possible thing to do or it is my fault as a beginner. Can you give me some advise or hints please?


texts: Classification training using probabilites and not raw classes (factors)

texts:
I have a problem where instead of having classes, i.e. a vector of 0s and 1s, I have the probability of an observation belonging to a class.

A vector with 0.1, 0.95, 0.2, 0.3, etc.

The obvious approach is using regression and it works relatively well, but I'm interested in an approach that trains on these probabilities that an observation belongs to a class and classifies them.

A Multinomial Classification approach has also been tried. The problem with this approach is that it doesn't take into consideration the order of the classes  (factors) which causes in some cases that the highest predicted class for an observation is at one end (let's say 0.2) and the second one is at the other end (0.8). Further, the more classes there are the less scalable this approach it becomes. Conversely, if there are too few classes gaps in predictions start to appear.

So my question is: Is there a classification algorithm that supports training probabilities instead of just factors (0s and 1s)?
Alternatively, is there another approach that is not regression nor classification that can solve this problem?


texts: Do C 4.5 or C 5.0 perform multi way split or binary split?

texts:
Suppose that I am using a continuous variable as an independent variable (although a bad choice) for C 4.5 or C 5.0 (tree based classifier). I am having a hard time figuring out does C 4.5 or C 5.0 does a binary split or a multi way split by calculating entropy?

If it does a multi way split how does it decides on optimal number of splits?

If it does a binary split for continuous variable using Entropy or Gini Index, then it's alright to regard it's splitting criteria the same as of CART?


texts: Features used by MAUI key phrase extraction tool

texts:
I have been trying out keyphrase extraction for a while and I want to know what are all the features that MAUI MAUI github uses for training the keyphrase extraction?
Having read this its hard for me to understand every feature that she uses can someone help me give a brief about the features that she is using? and some references for the same if any.


texts: CNN model for more than 3 channels input

texts: Are there any special cnn architectures for data, which has more than 3 channels? For example, satellite imagery, where we can consider different wavelenghts. It's clear, that we can readily use unet or something like that, but when there are 20 channels, for instance, it seems, that approach should differ. Not a neural network model, which can get feature-vector consisted of pixels from each channel, isn't sufficient perhaps, because spatial characteristics are important too.


texts: How does DQN solve Open AI Cartpole - v0?

texts:
Context

I am confused about how a DQN is supposed to solve the cart pole problem since the rewards are so dense. I have been using pytorch example. I am aware of some solutions, but I have issue with the basic principle of the env.

Unlike the tutorials, I converted the state space representation into just the 1x4 returned state as opposed to an image. Also, I converted the action output to be a binned output. So the action 1x2 becomes 3x2 when binning is set to 3. So instead of getting the max action row-wise, I get a max action column-wise. I am using fixed targeting (training a primary and target DQN).

Question / Concern

My main issue with the env in general is that keeping the pole vertical is no different to the DQN as holding the pole to near-failure. How does the DQN get better if it is getting +1 reward regardless? My hypothesis is that keeping the pole tilted creates more samples in the memory. Then after, when optimizing the model, the tilted pole states get higher reward than the vertical pole states since there is a higher distribution of tilted pole states. How are we expected to expect a DQN to do well with this kind of reward set up? Wouldn't it be better to have Cart pole produce +1 reward only for if the pole is near vertical?

Extra Information

The goal here is to use the cart-pole for debugging a RL model, then shift it to multi-joint robot control.

The state is normalized to the expected min/maxes:
Current State [[0.45564668 0.51196048 0.53126856 0.52450375]] The action input is Raw Action tensor([[0.9477, 0.9471]])
Bin Action [1.0, 0.0]. I am just using simplest action rep for testing below. I have also tested double dqn's, dueling dqns, and the use of PER. I have also been testing dropping the state space down to 1x1 via just inputting the angle of the pole.

X axis is the number of steps during an episode. Y axis is the number of episodes.


texts:
I have a beam loaded with random loads, for each timestep I obtain accelerations and maximum moments on the beam.

My prediction should receive a timeseries of accelerometer data and output the maximum moment at the end of that time. What type of NN should I use?

I've found that LSTMs and GRUs predict on a future time step of the input data (here next accelerometer value) but I need to predict the moment. Note that the simulation lasts for more than 0.5 s.

When training the network both accelerometer and moment data is available but when testing only
accelerometer timeseries should be provided to the prediction.

The logic is: after 0.5 s of these displacements I should expect a maximum moment of...


texts: Help solving Bigram Model with the following probabilities


texts: Identifying potential customers based on their Rank and Value

texts:
I have a dataset which has demographic data available for a list of new customers.
the data does'nt include transaction data of the customers.
I want to identify the top 100 potential customers among these customers. Im aware that we can make use of clustering to segment these customers.However, I have two more variables in my data which are Rank and Value.
What approach should be taken when rank and value of customers are given.How do we interpret the clusters given these 2 variables.
Need some guidance on this


texts: How do I use Machine Learning to analyze process statistics of automated test results?

texts:
(I am newbieeee in machine learning)

At the end of every automated test (Python), I collect a lot of information from our product. The information includes CPU, Memory Usage, Disk Activity, Postfix Queue statistics, and various other stats.

I want to come up with a script/tool(Python) that leverages machine learning and automatically analyze the results for me, instead of me looking at the results manually every day. I want to train the tool on what acceptable levels are for various components like CPU, Memory, depending on product's specifications, and various inputs.

From what I read online, I think I need to implement supervised classification in my tool, build a model, train it for a few days. But what I don’t know is where to start. When I looked at Supervised learning, I see a bunch of stuff and don’t really know what is really relevant to me. I appreciate if I can get some suggestions on what I need to focus on and how to implement this.

All the data can be in memory before my original script completes, or I can get the data from a database to which my original script writes to.


texts: Which machine learning algorithm should I apply for differentiate question difficulty level with users' result

texts:
Here's the scenario, There's a database with thousands of single-option questions for testing a specific skills, and a large number of users (either professional or amateur in this skill), each of which answer 10 random questions from the database.

The only thing that I can think of is to differentiate question difficulty level according to the correct rate of each question. But how could I take fully use of other informations like:

the correct rate from the user's perspective and feedback to its own influence to the difficulty level of questions (if user A answers 9 out of 10 questions correct, then the incorrect one (question_10) is more likely to be harder, than an user B answers 1 out of 10 questions correct and question_10 is in the incorrect set)
the answer time for each question by each user

Could anyone give me some ideas on this model, like where should I delve more into to make the difficulty level of question more accurate? Great thanks!


texts:
Recently I deployed a program using libtorch (PyTorch C++ API). The program run as expected but its gives me a warning.

Warning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().

How do I disable the warning ?


texts: Regression: how to interpret different linear relations?

texts:
I have three datasets, let's call them X and Y1 and Y2. A scatterplot is produced out of them, with Y1 and Y2 sharing them same X dataset (or support).

My question: if the two regression lines are different in both slope and intercept, is there a way to evaluate if the X dataset has more influence on Y1 or Y2?

Based on the image below, this is to say - which Y dataset is more influenced by the X dataset?

Blue slope (Y1): -112
Red slope (Y2): -90

EDIT

It is visible that an increase in X produces a decrease in both Y1 and Y2. My question could be interpreted as follows: which Y dataset decreases the most, given the increase in X? Is the slope everything I need?

Image:


texts: input and output to fully connected layer

texts:
I have an input vector of size 384. It is to be inputted to FC layer with one hidden layer. The output should be a vector of 3.
How is it possible? What is the math behind the calculations in FC layer?


texts: Lower Variance vs. Higher Validation Scores

texts:
So I'm trying to compare between two models, say model(1) has training accuracy of 90% and validation accuracy of 86%, while model(2) has training accuracy of 87% and validation accuracy of 85%.
Now, model(1) has a better validation score, but with high variance, and model(2) has a lower variance, but a slightly worse validation score.
Which one should I pick? assuming these are the best results we'll ever get.
I'm new in this, but my intuition is pushing me towards picking the more stable model with lower variance, but I would like to get feedback from more experienced professionals.


texts: Probability of event given two depandant events

texts:
Is there anyway, to compute the probability of the event given two dependent events?
I know that Bayes can help if those events are independent, but what if condition events are dependent?


texts: Question Answering - Was there any direct model before BERT?

texts:
This is a follow up question after question-answering-without-bert-transformers . While in search of a question answering mechanism without transformers, I am hitting dead ends.
Old question answering techniques I could figure out are finding similar known questions from a dataset and responding with a already known answer. This is in contrast to actual semantic understanding of question and paragraph, and then pointing to start and end word of answer span in document.
So the question to experienced NLP researchers is that, was there any working technique devised before invention of transformers (BERT), which was capable of pointing answers spans in the context document?
Would appreciate, if some code repo/article links can be posted here please.


texts: Correlate an array of categorical features to binary outcome

texts:
I have a data set that looks like this:

target,items
1,[i1,i3]
1,[i4,i5,i9]
0,[i1]
...

The variable target is 0-1 outcome. The feature "items" is a set of items (variable length). Each item is a categorical variable (one of: i1, i2, .., i_N). There's no order/relationship between the items. A business example would be "set of products in a cart, outcome whether the customer abandons cart".

The size of data is approx. 1,000,000 by 5,000 (I have ~1 million examples, and N is approximately 5,000)

I want to do the following analysis. I want to find the items that influence (or lead to) target = 1. I don't have extra features to add. What is the type of statistical analysis or machine learning modelling technique that I should use?


texts:
I am all new with ML. I try to understand what is Kfold and cross_val_score.
I made this model:
RandomForestRegressor(max_depth=17,n_estimators=93,criterion='mse')
modelfinal.fit(xtrain1, ytrain1)
mrse_test = np.sqrt(mean_squared_error(y_pred=modelfinal.predict(xtest1), y_true=ytest1))

It gave me:

Train/test
R2
mae
mrse

Train
0.94
13.52
22.07

Test
0.605
34.57
56.592

Because on the train, it gave me 0.9 as R2 and only 0.6 on test.
I have been told  I should use Kfold cv to see if I have over overfitting or not.
But I just can't understand how it will help me, and what I did is right or not.
This is what I did:
cv=ShuffleSplit(5,test_size=0.2)
score=cross_val_score(modelfinal, X, Y, cv=cv)

This is what I get:
array([0.59618812, 0.57799759, 0.55214982, 0.63904499, 0.65027832])

What can I conclude from this?


texts: How to convert a text to lower case using tm package?

texts:
I am using the below R code to convert text to lower case:

movie_Clean <- tm_map(movie_Clean, content_transformer(tolower))

However I end up getting the below error:

  Error in FUN(content(x), ...) :    invalid input 'I just wanna watch
  Jurassic World í ½í¸«' in 'utf8towcs'.

Please help how to overcome this error.


texts: Finding a pattern in reponses in R programming

texts: I did a study in which around 1000 participants took a test (100 questions). In this study participants were asked in each question to choose between two texts (text 1 and text 2) and decide which text is easier for them. Now in R I want to check if there are any participants who followed a pattern. For example, he or she have only chosen texts 1 or text 2. I also want to examine response string screening for participants that alternated left/right/left/right etc... 20, 30, 40, 50, 60, 70, 80, 90, 100 times in a row. Can anyone help me that how I can do it in R?


texts: Reducing search iteration over millions of data

texts:
The problem goes like this, with a story.

You have an application that contains a search field. When you search some input, there's an auto-complete component that pops up showing up similar results to your input.

Each result is a location in the database and this is the structure of location:

id | name                    | lat  | lon
-----------------------------------------
0  | New york street test 51 |34.123| 38.245

Every location has lat and lon coordinates.

Your system has a function that knows how to receive 2 locations and return the distance in km.

You also have a table called stores which contains id, name, lat, lon of the shop.

When you select a place, your application finds 20 closest stores to the location that you have selected.

The problem

Everything works fine for a start, but now the application has grown and filled up with new stores in many locations and you have over one 500 million stores registered in the database. Every search is heavy and overloading the system and it takes a few seconds to a minute to get a result.

How can you make it better, efficient?

Spoiler - My idea to answer the question

   My idea is to split the world map into regions matrix of size Constant where constant is set to 10km x 10km. Each region will have its ID and every shop you add through your platform will get the region id of where you add it to. When you search for lat and lon, the system will go through all regions searching what region contains that coordinate and return the region id, and then you can grab all shops of that region id. But wait, you have a problem, what if there's a closer store in the neighbor regions? Simple, you have 4 points of your region's corners, you can look up for all neighbor regions and compare if there's a closer store. And if you don't have enough stores and you must get 10 stores out of all regions, use the 4 coordinates of the corners as-well and go recursive on all regions until you reach 10 closest.

I am open and happy to hear everyone's thoughts. Note this is not a real story, it was asked in some interview and I took it very interesting to see if there are better and more interesting solutions.


texts: Theano logistic regression understanding code

texts:
I am used to C/Java like programming, and sometimes I am getting a headache on understanding the Python notation.

On the logistic regression code available online, I am trying to understand this line of code:

-T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]), y])

It is basically saying perform this average:
$\begin{align}
J(\theta) = - \left[ \sum_{i=1}^{b} \sum_{k=1}^{K}  1\left\{y^{(i)} = k\right\} \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\right]
\end{align}$ where $b$ is the batch size.

So is it related to theano code, or it is just a python notation ?
I am interested exactly on this piece of code:

T.log(self.p_y_given_x)[T.arange(y.shape[0]), y]

If you could please give me some explanation.


texts: How do I validate this Kalman model for estimation of undocumeted covid cases?

texts:
Tensorflow recently made a tutorial titled Estimation of undocumented SARS-CoV2 cases. It replicates 6th March 2020 paper by Li et al titled Substantial Undocumented Infection Facilitates the Rapid Dissemination of Novel Coronavirus (SARS-CoV2). It is a compartment based SEIR model where the population is represented by a state with compartments Susceptible, Exposed, Undocumented Infectious and Documented Infectious.
I am trying to test the validity of this model by training it with data for 30 days, and forecast the state for next 15 days. I did find the optimal parameters of the model and current state, but I am not sure how I can use it to predict future states. I have been attempting this for almost a week now. The programming style in the notebook is quite unfamiliar to me, hence, I am struggling to figure out how exactly to do it.
I request someone with more experience to go through the notebook once and give me suggestions about how to predict the states.
Thank you!


texts: What happened to the accuracy before and after 75th epoch. Why its unstable at first, then stepped up after the 75th epoch?

texts: What happened to the accuracy before and after 75th? Why before 75th, its unstable and suddenly stepped up after that? I am using RESNET and CIFAR-10 dataset.


texts: Python SkLearn Gradient Boost Classifier Sample_Weight Clarification

texts: Using Python SkLearn Gradient Boost Classifier - is it true that sample_weight is modifying how the algorithm penalizes errors made on that particular class, rather than feeding more data into the trees by oversampling from that class. If you have any links to code that confirms this that would be ideal. Thanks


texts:
I have a multiclass-classification dataset with the target (dependent) variable highly imbalanced. While using the randomForest package in R, I usually use the parameters sampsize & strata to account for the imbalance in training data. Are there any similar options in xgboost package also?

Summary of the number of datapoints available in each class.

Factor 1     : 667
 Factor 2     : 676
Factor 3    :7807
 Factor 4       : 850


texts: Forecasting via LSTM or XGBoost... is it really a forecast or

texts:
I guess I understand the idea of predictions made via LSTM or XGBoost models, but want to reach out to the community to confirm my thoughts.

This tutorial does a nice job explaining step by step of what to do:
"How to Develop Multi-Step LSTM Time Series Forecasting Models for Power Usage"

However, when it came to forecasting, the author held out portion of the data and then used that data to predict future values... In my mind, it is not really a forecasting (something that is done via ARIMA, VAR, etc - where you specify number of time periods and then don't specify anything else and the model gives you some forecasting for the future)
You can see my comments to the author and basically the predictions via LSTM or XGBoost models will be based on the input values and not on the future data range, i.e. I would need to supply temp, humidity, wind, etc to get the forecast of the power consumption for the house hold.

That being said, in what extend LSTM or GXBoost are used in forecasting? Given these values predict what would be the final outcome? Hence if I need to forecast something in the future and have no clue about the other input values, just stick to traditional VAR model?

By looking at this question/answer by @Fnguyen, "If you have more input variables you need a way to forecast or impute these because to make a prediction your model needs all inputs that build the model. " it seems that I should use ARIMA to forecast other input values and only then use LSTM... makes me wonder if forecast would be that accurate now that it depends on ARIMA that would predict future values of the input of LSTM and then LSTM model that would make final prediction. Then again, accurate forecast is fairly complicated thing to do as we are trying to predict the future.


texts:
I used the following code to pass a json string as a parameter in requests.post method,
import json
data = {'Key': 'Value',"title": "hello world"}
# get string with all double quotes
json_string = json.dumps(data)
json_string

output is
'{"Key": "Value", "title": "hello world"}'

How to remove the single quotes around the dictionary without using
print(json_string). So that I can pass this in requests.post like this
results=requests.post(url=url,data=json_string)

If I give the json_string without removing single quotes, it gives me internal server error.


texts: How can we get gradient with some other loss function apart from MSE?

texts:
In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE.
Can we have an example, where we did not use derivative of MSE, but used derivative of some other loss function?
If so, please show the Python code.


texts: Pretrained models for Propositional logic

texts:
Are there any pretrained models which understand propositional logic?
For example, the t5 model can do question-answering. Given a context such as "Alice is Bob's mother. Bob is Charlie's father", t5 can answer the question "Who is Charlie's father" correctly, but it cannot say "Who is Charlie's grandmother".
Is there any model that has been/can be trained to do this kind of deduction and answer the question?


texts: Markov Chains: How much steps to conclude a Transition Matrix

texts:
I have just learned Markov Chains which I am using to model a real world problem. The model comprises 3 states [a b c]. For now I am collection data and calculating transitional probabilities:-

T[a][b] = #transitions from a to b / #total transitions to a

However I am stuck at determining the correct Transition Matrix. As I am getting more data, the matrix is changing drastically. So when do I finalize Transition Matrix? Does that mean that my data is too random and cannot be modelled or I am doing some mistake here?


texts:
I have a database containing sets of words. So for example, I have a database that has:

{happy, birthday, to, you}
{how, are, you}
...

Given a query set, lets say {how, was your, birthday}, I want to find the top K sets in the database that is most similar to my query. Similarity metric used can be something like Jaccard's Index. Right now, I go through the database one by one and calculate the Jaccard Index and keep track of the top K scores found so far. I was wondering if there are any data structures or methods that would allow me to more efficiently find the top K scores. Right now it's a linear search. Thanks


texts: Is shuffling training data beneficial for machine learning?

texts:
I was curious to know if shuffling ML training data is beneficial to better results?

Sorry not a lot of wisdom here, but I have been reading a post from pythonprogramming.net for this topic.

I copied this function from the post and modified to just save my shuffled data to csv file.

def Randomizing():
    df2 = df.reindex(np.random.permutation(df.index))
    df2.to_csv('C:\\Users\\Machine-Learning-Electric-Data\\randomized.csv')

Randomizing()

What appears to happen is only the index gets shuffled and all other data stays the same. I have many columns in my pd dataframe where I would need to keep all rows the same. (randomly shuffle all rows, its time series data) If this is beneficial can someone give me a tip on how to randomly shuffle my data more than just the index?


texts: How to decide on using xgboost with imputation or without it and keeping missing values?

texts:
I have a large genetic dataset that I am using xgboost on to score most likely disease causing genes - giving the genes a score between 0-1 of likelihood.

I try to avoid features with a lot of missing data but this can be hard for genetic data, the largest amount of missingness I have for a feature is roughly half of values in a feature column are missing.

Currently I run my xgboost model in 2 versions, one with random forest imputation of missing values and one without where xgboost is handling the missing data directly. The imputation model performs at an r2 of 0.7 and the model with missing values performs at 0.8 on nested cross-validation.

My question is how do I choose which version to take on to further work? Can I trust that the higher 0.8 r2 with missing data is because xgboost is finding patterns in the missingness and finding this useful? Are there rules around missing data I should be trying to abide by? I have a biology background so I'm unsure what is best practice from a data science perspective for handling missing data, most resources about this that I've found online conclude this is a case by case problem, which I find hard to interpret into what I should be specifically looking into. Any help would be appreciated.


texts: Feature name extraction directly from dataset

texts:
I would like to visualize my features using the code below. However, I am getting an error that my features are being recognized as "nan" instead of their actual names.

Instead of the below Feature = [and then writing the features that I want], I am assigning  feature = data [0,1:]. This is the first row of my data that has the features in it. I have many features that I don't want to write as string but would just like to extract from the data file directly. How can I do this and get the names of the features instead of 'nan'?

# Load the dataset
data = load_data('credit')

# Specify the features of interest
features = [
        'limit', 'sex', 'edu', 'married', 'age', 'apr_delay', 'may_delay',
        'jun_delay', 'jul_delay', 'aug_delay', 'sep_delay', 'apr_bill', 'may_bill',
        'jun_bill', 'jul_bill', 'aug_bill', 'sep_bill', 'apr_pay', 'may_pay', 'jun_pay',
        'jul_pay', 'aug_pay', 'sep_pay',
    ]

# Extract the instances and target
X = data[features]
y = data.default

from yellowbrick.features import Rank1D

# Instantiate the 1D visualizer with the Sharpiro ranking algorithm
visualizer = Rank1D(features=features, algorithm='shapiro')

visualizer.fit(X, y)                # Fit the data to the visualizer
visualizer.transform(X)             # Transform the data
visualizer.poof()                   # Draw/show/poof the data


texts:
I'm learning how to use the Recurrent Neural Network model (RNN). I'm not entirely sure about the feed-forward procedure in RNN. It includes, for example, input, hidden state, and output. As far as I know, the hidden state is a type of multi-layer perception (MLP). However, in this case, a hidden state is derived from both current input and a previously hidden state. Unfortunately, "we can note that everyone reported the total number of memory cells, but no one specified the number of neurons inside each memory cell," which confuse me.
Second, I am confused by the RNN backpropagation procedure. I searched Google, and everyone only mentioned the generic steps (calculations of gradients) but no one conducted step-by-step backpropagation on example. I'm desperate for the entire RNN training process (two iterations is sufficient), even on a single layer with three to four memory cells.
Can anyone available concrete examples of the RNN model training process?


texts: Impact of a new word on word embedding vectors

texts:
Question
What is the impact of a new word on the word embedding vectors already trained before the word is invented?
For instance, at November 2019, there existed multiple pre-trained models from Hugging Face which contained word embedding vectors for the words that existed in the training+testing data used. Now there is a new word COVID-19. What will happen to the vectors? For instance the vector for vaccine will change if the model is re-trained on the new data including COVID-19?
I suppose the locations and the distance between two words is a representation of the relation between them. Will COVID-19 being added relocate the vectors of existing and related words such as vaccine, flue, virus?
If a fine-tuning (transfer learning) has been done already on the model without COVID-19, I suppose we need to import the new model and re-run the fine-tuning. Is this correct? A NLP system which has been using the model without COVID-19 needs to be updated with the new models too?


texts: Navigating the jungle of choices for scalable ML deployment

texts:
I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model was built with Python and Keras. It takes in a CSV table of inputs and spits out the corresponding CSV table of predicted outputs.)

My naive "vision" is that I'd have the model reside on a single (master) machine, whereas the data would be equally distributed among several units (whatever unit means: nodes in a "cluster", CPUs, GPUs, ... ?) The model would be projected onto these units and the learned parameters would somehow all synchronize back to the master unit. Similarly, in the case of serving, the same model would be applied to the data that resides on the different units. Does this "vision" sound reasonable? (I have had some experience with parallel computing with MPI and I vaguely remember that's how things work.)

If I were to start from a blank slate, what architecture/infrastructure should I choose to deploy my model in a scalable fashion? Below are some of the confusingly many options I have read about. (I hardly master what each of these things do, so please forgive me if it looks like a laundry list of disparate technologies.)

Kubeflow (i.e., Kubernetes)
PySpark
Amazon SageMaker
Distributed TensorFlow
Dask
Apache Mesos
REST API
Hadoop
... and dozens more

As a pure ML guy (read: Python, Keras, pandas guy coding on a laptop), I'm out of my depth with all the infrastructure jargon that comes with the above links. It's therefore overwhelming to find a starting point, or some kind of "Hello World" example I could relate to. All I want is an architecture from which I can transpose my code and have it run in an efficient and scalable manner. Which one does the job? Despite all the hype around ML, there does not seem to be any comprehensive map or comparative review of all these solutions. I did find some comparisons, for example between Spark and SageMaker, or Spark and Dask, but given my illiteracy in these subjects, they only add to the confusion.


texts: Duplicate QUORA question detection:Kaggle Dataset

texts:
I have tried to use 2 BILSTMs along with the attention layer but the validation accuracy is not improving at all. Could anyone suggest an alternative to increase the accuracy?
Layer structuring:
>Preprocessing
>GLOVE
>BILSTM
>ATTENTION
>BILSTM

Current Validation Accuracy for 10 epochs is in the range of 0.62 to 0.63
Code:
## Embeddings matrix is the set of vectors representing our words
from keras.initializers import Constant

# Length of the maximum sentence in the df
sequence_length = 470
DROPOUT=0.1

question1 = Input(shape=(max_len,))
question2 = Input(shape=(max_len,))
q1 = Embedding(len(word_index)+1,
                 embedding_dim,
                 embeddings_initializer=Constant(embedding_matrix),
                 input_length=max_len,
                 trainable=False)(question1)
q1 = Bidirectional(LSTM(300, return_sequences=True), merge_mode="sum")(q1)
q2 = Embedding(len(word_index)+1,
                 embedding_dim,
                 embeddings_initializer=Constant(embedding_matrix),
                 input_length=max_len,
                 trainable=False)(question2)
q2 = Bidirectional(LSTM(300, return_sequences=True), merge_mode="sum")(q2)
## Applying Attention
model = dot([q1,q2], [1,1])
model = Flatten()(model)
model = Dense((max_len*300))(model)
model = Reshape((max_len, 300))(model)
model = add([q1,model])
##Applying BiLSTM
model = Bidirectional(LSTM(300, return_sequences=True), merge_mode="sum")(model)
model = Flatten()(model)
model = Dense(200, activation='relu')(model)
model = Dropout(DROPOUT)(model)
model = BatchNormalization()(model)
model = Dense(200, activation='relu')(model)
model = Dropout(DROPOUT)(model)
model = BatchNormalization()(model)
model = Dense(200, activation='relu')(model)
model = Dropout(DROPOUT)(model)
model = BatchNormalization()(model)
model = Dense(200, activation='relu')(model)
model = Dropout(DROPOUT)(model)
model = BatchNormalization()(model)

is_duplicate = Dense(1, activation='sigmoid')(model)

final_model = Model(inputs=[question1,question2], outputs=is_duplicate)
final_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

history = final_model.fit([x1_train, x2_train],
                    X_train.is_duplicate.values,
                    epochs=10,
                    verbose=2,
                    validation_data=([x1_test, x2_test], X_test.is_duplicate.values),
                    batch_size=64,
                    callbacks=callbacks)


texts: how to make new columns using another column in pandas?

texts:
i have a dataframe like this

i want to add new column and my desire format is like this

i use indexing eg
 dll.loc[:6,'category'] = "sectowise"
dll.loc[7:11,'category'] = "productwise"
dll.loc[12:16,'category'] = "collateral wise"
but which is risky because data index can change anytime
is there any method to do this?


texts: Creating Data model for mahout recommendation engine

texts:
I am trying to build an item-item similarity matching recommendation engine with mahout. The data set is as in the following format ( attributes are in text not in numerals format )

name : category : cost : ingredients

x : xx1 : 15 : xxx1, xxx2, xxx3

y : yy1 : 14 : yyy1, yyy2, yyy3

z : xx1 : 12 : xxx1, xxy1

So in-order to use this data set for mahout to train, what is the right way to convert this in to numeric (as CSV Boolean data set) format accepted by mahout.


texts: distinguish users for recommender system

texts:
How can you calculate better video recommendations for a SmartTV app which is used by multiple users in a household. I don't know which user is currently watching the video because the account for the app is shared. So the current user might get a recommendation based on video usage of an other user.

I was thinking of using principal component analysis to find out how many users live in the household and which user is likely to watch the video now. I could map the usage data to this specific user and then build recommendations for this hypothetical user.

What would be the best approach to solve this problem. Are there best practices or papers which describe a solution?


texts:
I am new to machine learning.
Does it make sense that my model works when I do both Standardise and Normalise? What does this say about my data?

Or do I do have to select one or the other? My goal is to build a binary classifier which analyses CNNs and GIST features?


texts: Discovering dis-associations between periods of time-series

texts:
I'm interested in discovering some kind of dis-associations between the periods of a time series based on its data, e.g., find some (unknown number of) periods where the data is not similar with the data from another period.

Also I would like to compare the same data but over 2 years (something like DTW?).

I get my data Excel as a two-column list:

c1=date (one per each day of the year), c2=Data To Analyze

So, what algorithms could I use and in what software?

Update/Later edit:
I'm looking for dates as cut-off points from which the DataToAnalyze could be part of another cluster of consecutive dates. For example:

2014-1-1 --> 2014-3-10

are part of Cluster_1 based on DataToAnalyze. And:

2014-3-11 --> 2014-5-2

are part of Cluster_2 based on DataToAnalyze, and so on. So, clusters of consecutive dates should be automatically determined based on some algorithms, which is what I'm looking for. Which ones (or which software) would be applicable to this problem?


texts: How to apply hierarchical cluster analysis using the Hellinger metric to an LDA model?

texts:
I am doing an LDA analysis, I have the topics, but I need to cluster the topics according to Hellinger distance.I need to group the 20 topics generated by the LDA model and present it in a dendrogram.
I share part of the code.
'''
textos <-select(Base_Articulos, Articulo, Evento, Ano)
textorder <- textos[order(textos$Ano),]

bd_duplicados <- textos[duplicated(textos),]
bd_unicos <- unique (textos)
bd_unicos <- na.omit(bd_unicos)
ap_td <- tibble(textos)
ap_td

tidy_articulo <- ap_td %>%
  unnest_tokens(word, Evento)

espstopwords <- tibble(word = c(stopwords(kind = "es")))
enpstopwords <- tibble(word = c(stopwords(kind = "en")))

miastopwords <- tibble(word = c("colombia", "study", "bogota",
                                "colombian", "colombiano", "t",
                                "medellin", "n", "k", "b", "hom",
                                "cc", "92", "85", "m", "1",
                                "l", "sp", "50",
                                "155.000", "155", "59",
                                "64", "70", "80",
                                "18", "ri", "2", "3", "4", "5", "6", "7","8", "9"))

tidy_articulo <- tidy_articulo %>%
  anti_join(espstopwords)
tidy_articulo <- tidy_articulo %>%
  anti_join(enpstopwords)
tidy_articulo <- tidy_articulo %>%
  anti_join(miastopwords)

ap_td <- mutate(ap_td, Evento = as.character(ap_td$Evento))

tidy_articulo %>%
  count(word, sort = TRUE)

word_counts <- tidy_articulo %>%
  count(Articulo, word, sort = TRUE) %>%
  ungroup()

word_counts

desc_dtm <- word_counts %>%
  cast_dtm(Articulo, word, n)

desc_dtm

ap_lda <- LDA(desc_dtm, k = 20, control = list(seed = 1234))

ap_lda

ap_topics <- tidy(ap_lda, matrix = "beta")
ap_documents <- tidy(ap_lda, matrix = "gamma")'''


texts: Dynamic time warp z score normalisation not working

texts:
I have data that looks like this

A bit of background, these are soil moisture graphs of different depths. I wish to investigate how long it takes for water to drip down from one depth to another, and usually there is a bit of lag between when the data jumps upwards (when the soil gets watered) so I'm trying to calculate the lag of the blue graph compared to the red graph. Both graphs share the same x axis and start and end at the same time value.

Below is a z score normalisation attempt, but at the end of the graph, they aren't lined up very well, so this happens:

Does anyone know how to normalise this better so that the last part of the graph isn't below the other?

As well as this, it does not match well here either.
A picture of the whole warp

At the start of the warp:


texts: What possible data products can be built using this dataset

texts:
I have a dataset which contains information about when do people enter and leave a premise. I have the following information in the dataset :

Person Id
Time of Entry
Time of Leaving

The dataset has around 50 unique persons. Each person will have multiple entries corresponding to multiple visits. The data spans over a year so I have quite a lot of entries (around 1 million).

These people can be classified on the basis of the department they work under (2 departments - mutually exclusive) or on basis of  their role (4 possible roles - all mutually exclusive)

I was wondering what kind of data analysis can be done with this kind of dataset. I am not looking for straight-forward things like "who spent the most time in building". However things like finding correlation between visits of 2 people would be interesting. So if person A visits the premise, what is the probability that person B would also visit. Since I have only around 50 unique visitors, I think such an analysis is feasible.

Another line of thought was to apply some interval-pattern mining techniques but I am not much familiar with them.

Can someone give me some pointers/ideas about what kind of data products can be build using this or what kind of techniques can be used with such data.

Edit : As discussed in comments, I call it a product in the sense that I do not want some simple or trivial analysis. And I am not looking for any commercially viable idea - just some cool fun idea :)


texts: What is the difference between concept class and hypothesis

texts:
Formal definition that I have seen of concept class is

class of all true functions

mathematically :
$f:X \rightarrow\{0,1\}$
and that of hypothesis is:
$h:X \rightarrow\{0,1\}$
But most of the times they are used together. For example in definition of PAC

A concept class  is PAC learnable by a learner  using hypothesis space  if for all concepts ∈, distributions over , true error probability 0≤≤1/2, failure probability 0≤≤1/2, learner  outputs a hypothesis ℎ∈ such that
True error less than or equal to
Computational time is polynomial in 1/,1/, representation size of data object, and representation size of concept

What is the difference?


texts:
I have a time series. Data points are available for each year from 1966 to 2000. Using R, I want to decompose this time series into trend, seasonal and random components. When I run the decompose command, I get the error "time series has no or less than 2 periods". Since my data is annual I have specified a frequency of 1. What am I doing wrong?

Here is the R code that I am using:

dat=c(37.2,37,37.4,37.5,37.7,37.7,37.4,37.2,37.3,37.2,36.9,36.7,36.7,36.5,
      36.3,35.9, 35.8,35.9,36,35.7,35.6, 35.2, 34.8, 35.3,35.6,35.6, 35.6,
      35.9,36,35.7, 35.7, 35.5, 35.6, 36.3, 36.5)

whts <- ts(dat, frequency = 1, start=1966, end=2000)
is.ts(whts)
plot.ts(whts)
whtimeseriescomponents <- decompose(whts)


texts: Dask error when reading data from a large zip file

texts:
I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this:

import dask.dataframe as dd

df = dd.read_csv(file_names, storage_options={'fo': folder_path})

Where each file_name.csv is prepended with a 'zip://' tag and folder_path is folder_name.zip.

When I then try to perform any operation such as

df.set_index('new_index')

or

df.groupby(['column']).mean().compute()

I often get the following error stack:

Traceback (most recent call last):
  File "<input>", line 5, in <module>
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\base.py", line 166, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\base.py", line 437, in compute
    results = schedule(dsk, keys, **kwargs)
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\threaded.py", line 84, in get
    **kwargs
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\local.py", line 316, in reraise
    raise exc
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 115, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 115, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\dask\bytes\core.py", line 184, in read_block_from_file
    return read_block(f, off, bs, delimiter)
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\site-packages\fsspec\utils.py", line 258, in read_block
    b = f.read(length)
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\zipfile.py", line 930, in read
    data = self._read1(n)
  File "C:\Users\Vlad\Miniconda3\envs\pvforecast\lib\zipfile.py", line 1006, in _read1
    data = self._decompressor.decompress(data, n)
zlib.error: Error -3 while decompressing data: invalid block type

The weirdest thing is that it sometimes works perfectly fine for the same dataset without changing any code. It also always takes a different amount of time to raise the error and does it for different parts of code. Seems like it could be a bug within dask, but I don't get why it works at one time, but not the other?


texts:
What are some good error metrics for multi-label (not mutli-class) problem in industry?

http://scikit-learn.org/dev/modules/multiclass.html


texts: Simple example of genetic alg minimization

texts:
I have been looking for a while for examples of how I could find the points at which a function achieves its minimum using a genetic algorithm approach in Python. I looked at  DEAP documentation, but the examples there were pretty hard for me to follow. For example:

def  function(x,y):
     return x*y+3*x-x**2

I  am looking  for some references on how  I can make a genetic algorithm in which I can feed some initial random values for both x and y (not coming from the same dimensions). Can someone with experience creating and using genetic algorithms give me some guidance on this?


texts: Regression coefficients vs feature_importances_ vs none

texts:
On looking at various machine learning methods at the scikit-learn, it appears that some modules such as LinearRegression provide coefficients (coef_), others such as AdaBoostRegressor provide feature_importances_ while some e.g. BaggingRegressor and BernoulliNB do not provide either of these.
Why this difference? Are coef_ similar to feature_importances_ to assess the contribution of a variable in prediction? How to assess the feature importance for modules where neither of these is available?


texts:
I understand Conv2D filters. I think I understand Conv1D filters as well but have not seen any examples of the filters like what we have in 2D filters, e.g. Sobel as shown in the following image. Are there any standard one dimensional filters?

Sobel 2D Filter example


texts: Why does normalization improve my decision tree performances?

texts: I have a regression problem for which I have to try several models, so I normalized my data and then tried to use a decision tree regressor (from sklearn.tree) and I noticed very good results (explained variance higher than 0.999). After some research I found out that for decision tree normalization is irrelevant, so I tried to skip the normalization process and explained variance went down (in several tries it stays between -0.006 and 0.006). Is it possible than normalization affects the decision tree performances or maybe there is some coding issue or something like that?


texts: I am using the rpart function (in the "rpart" package) in R to fit a cost-sensitive classification tree. I have read the corresponding vignette, which says that misclassication costs are incorporated via altered priors (section 3.2.2 on page 8). However, I am not sure that I understand exactly how these altered probabilities affect the splitting criteria/impurity index. Does anyone know?


texts: NER at sentence level or document level?

texts:
Should NER models (LSTM or CRF) take input training data at sentence level or paragraph level?

Let's say we have this input text, and we would like to do Named Entity Extraction:

  GOP Sen. Rand Paul was assaulted in his home in Bowling Green,
  Kentucky, on Friday, according to Kentucky State Police. State
  troopers responded to a call to the senator\'s residence at 3:21 p.m.
  Friday. Police arrested a man named Rene Albert Boucher, who they
  allege "intentionally assaulted" Paul, causing him "minor injury".
  Boucher, 59, of Bowling Green was charged with one count of
  fourth-degree assault. As of Saturday afternoon, he was being held in
  the Warren County Regional Jail on a $5,000 bond.

Paragraph level: we can take it as one record and each token is marked by the entity label. Model has ONE record with LONG sequence.
Sentence level: we first intelligently split the paragraph into 5 correct sentences, and each token in each sentence is marked by the entity label. Model has FIVE records with shorter sequences:

  0) GOP Sen. Rand Paul was assaulted in his home in Bowling Green,
  Kentucky, on Friday, according to Kentucky State Police.

  1) State
  troopers responded to a call to the senator's residence at 3:21 p.m.
  Friday.

  2) Police arrested a man named Rene Albert Boucher, who they allege
  "intentionally assaulted" Paul, causing him "minor injury".

  3) Boucher, 59, of Bowling Green was charged with one count of
  fourth-degree assault.

  4) As of Saturday afternoon, he was being held in the Warren County
  Regional Jail on a $5,000 bond.

Which one gives the NER modeling a better NER performance?

I tend to think sentence level is better, however, shouldn't LSTM memory cells be trained to remember or forget states automatically if given long paragraphs? Especially when the sentence splitting can also potentially make mistakes, for instance:

  1) State troopers responded to a call to the senator's residence at 3:21 p.m. Friday.

could have been

  1) State troopers responded to a call to the senator's residence at 3:21 p.m.

  2) Friday.


texts: How to generate sports tracking data using deep learning?

texts:
Data: I have a 2D Numpy array that contains tracking data for football. Each row has the (x,y) coordinates for all players + the ball. That's 22 players and 1 ball = 46 columns. The frequency is 0.1 seconds i.e., every row shows everything's positions 0.1 seconds after the previous row. I have a total of a few thousand such arrays.

My Goal:
I was thinking of generating artificial data based upon these sequences. I got the original idea from this paper - Generating Play-by-Play Basketball Data

What I've tried My initial efforts were with a GAN. Since it was already a 2D array, I thought I could convert it to images and then use a GAN to generate images based on them (I'd read a lot about GANs to generate faces and handwritten digits). However, that didn't work out for me. Here's what any one of the actual real images look like -

And here's what the generated image looks like (after 29000 epochs) -

Obviously, a lot of noise

The dimensions are all 46x46 which is why the pictures are so tiny.

I read up a bit and from what I understood, generating temporal data is slightly different from normal images so I'm back to square one.

If someone could suggest some ideas to help me in understanding -

What's a general idea to go about achieving this task with GANs (temporal sequential data instead of image arrays),
What generative models/ networks/architectures could I use to do so,
Any other tips/suggestions as well.


texts: Is using unsupervised learning to setup supervised classification reasonable?

texts:
I've got a biological dataset describing genes. The overall idea is that there are thousands of these genes to sort through, so if ML can rank them I can then know which should be going into the lab for functional research first. Currently, I make labels for supervised classification of these genes based on their known biology (so for example some genes interact with drugs related to a disease so I label them as 'most likely to cause the disease' and this goes down until I have a final 4th label of 'unlikely to cause the disease').  The way I make these labels seems impossible to not be biased, since I'm making all the decisions, so I'm wondering if I can compare my decisions with seeing how an unsupervised model would group the data (e.g. I've got 4 labels but if the model finds 5 groups then it shows how far off I am potentially?).

Would it even also be possible to use unsupervised learning to create the labels by itself or would this too be unreliable as you can't know why it's grouping certain genes together? Or would doing this step  alone actually make the supervised step redundant anyway?


texts: Ordinal features to decision tree in Python

texts:
I have a data set with ordinal features.Each feature might have 6 to 7 levels.
Based on my search for R if you have ordinal data, rpart treats ordinal and nominal differently.
https://stats.stackexchange.com/questions/94502/decision-tree-splitting-factor-variables

But now I'm implementing the decision tree with Python and there nothing comparable to rpart to handle ordinal data. It seems Python sklearn does not handle categorical data well and I have to use one hot encoding. In this case, the order of level like level 1  to level 2 to level3......to level 6 will just disappear.

https://stackoverflow.com/questions/38108832/passing-categorical-data-to-sklearn-decision-tree

Any suggestions?
Thanks.


texts: Computational Complexity of Tucker decomposition

texts:
I am currently doing background reading for my Masters Thesis. I am working with tensor decompositions, whereby tensor I simply mean a multi-dimensional array. The aim of my master's project is to potentially find a new decomposition algorithm or to improve on existing algorithms. So I am quite interested in the computational complexity of decomposition algorithms.

I am reading about the Tucker Decomposition at the moment. The method is simple enough to understand and implement, especially the Higher-Order Orthogonal Iteration(HOOI) Algorithm. However, I am not going to spend a lot of time on this algorithm. So I do not want to include a whole proof and derivation of its computational complexity in my thesis.

Are there references out there that mention or derive the computational complexity of the HOOI algorithm?


texts: Using random forest to learn Imbalanced Data (rare disease)

texts:
I met a question when I ran the random forest. I used "V1", "V2", "V3" to predict a binary outcome (1: sick; 0: no) with random forest.

I got a very high accuracy score (99%) however, when I check the confusion matrix, it shows that none of sick individuals were caught in testing data set (30% of entire data set). Here is the confusion matrix:

[[856   0]

[  9   0]]

This result means that 0 out of 9 people was detected as sick and it causes my attention. Maybe because the data set is imbalanced (very few sick individuals)?

I would like to see if there is any other ways to detect sick individuals rather than a high accuracy rate, which means it is OK it has higher false positive rate but I would like to catch all 9 (true positive) individuals.

Thanks!