texts: Outputting risk groups for a logistic regression model

texts: I have a problem with outputting the terms for a logistic regression model in R.  For a given list of independent values, say list l of terms {w,y,z} to determine dependent variable {x}, I want to find out what the biggest regressor is when we pair two terms together.  I want to be able to group multiple independent variables together and say "when a record has this combination of values, then they have a very strong chance of predicting X".  I tried to just add the interactions when calling the glm function like glm(x~y + w + z + w:z + y:z + y:w, data = l).  But the results come out very hard to explain, because they of how they are measured between themselves and not just measured against the mean. Does anyone know a way to do this?


texts: How to convert the categorical features into one-hot encoded dense vectors before applying the min-max scaler of sklearn?

texts:
like the question says I am trying to figure out how to convert one-hot encoded vectors into dense vectors, because my data-set contains both continuous and categorical features, I used the pd.get_dummies() on all the categorical features to get the one hot encoded vectors. My code so far is as follows:
data[categorical_features] - data is the dataset
for feature in categorical_features:
    dummy_columns = pd.get_dummies(data[feature])
    new_columns = [f"{feature}_{item}" for item in dummy_columns.columns]
    data[new_columns] = dummy_columns.values
    data = data.drop(labels=categorical_features, axis=1)
data

I dropped all the categorical features after creating the dummies with the respective column names. Now, I believe, before creating a training set or applying mix-max scaler on the training set, I need to transform these one hot encoded vectors into dense vectors, but I can't figure out how, I searched for resources on the internet but nothing helped.
Once I create the dense vectors of new_columns, I would like to make the train-test split and apply mix-max scaler on training set, train it and so on...
Appreciate any thoughts or code examples to create these dense vectors, thank you


texts: Implementing Scikit Learn's FeatureHasher for High Cardinality Categorical Data

texts:
Background: I am working on a binary classification of health insurance claims.  The data I am working with has approximately 1 million rows and a mix of numeric features and categorical features (all of which are nominal discrete). The issue I am facing is that several of my categorical features have high cardinality with many values that are very uncommon or unique.  I have plotted 8 of my categorical features below which had the highest counts of unique factor levels:

Alternative to Dummy Variables: I have been reading up on feature hashing and understand that this method is an alternative that can be used for a fast and space-efficient way of vectorizing features and is particularity suitable for categorical data with high cardinality. I plan to utilize Scikit Learn's FeatureHasher to perform feature hashing on my categorical features with more than 100 unique feature levels (I will create dummy variables for the remaining categorical features with less than 100 unique feature levels).  Before I implement this I have a few questions relating to feature hashing and how it relates to model performance in machine learning:

What is the primary advantage of using feature hashing as opposed to dummying only the most frequently occuring feature levels?  I assume there is less information loss with the feature hashing approach but need more clarification on what advantages hashing provides in machine learning algorithms when dealing with high cardinality.
I am struggling to understand how to best determine n_features in Scikit Learn's FeatureHasher. Clearly higher hashing dimensions will encode more information and provide better model predictions at the cost of computationally more expensive training.  When researching FeatureHasher I was looking at it as a method to reduce the dimension of the encoded table generated (I have several categorical features with well over 1,000 unique levels) while preserving information by not dropping uncommon or unique features levels.  With that being said, how can best determine the optimal n_features for each column in my dataset with the goal of obtaining an encoded table that does not exponentially increase the number of features being fed into my classification models?
I am interested in evaluating feature importance after evaluating a few separate classification models. Is there a way to evaluate hashed features in the context of how they relate to the original categorical levels? Is it possible to reverse hashes or does feature hashing inevitably lead to loss of model interpretability?

Sorry for the long post and questions. Any feedback/recommendations would be much appreciated!


texts:
I'm building binary classification models on my company's dataset. The problem I'm having is that I haven't been able to increase the accuracy of my models. I have trained, tuned, cross validated models such as logistic regression, knn, neural networks and they all have results with the same accuracy. I feel that I have tried everything. The dataset is about 100 data points with 21 features. I'm aware of the curse of dimensionality so I have tried using only subsets of the features but the accuracy is the same no matter which subsets of features I use.
Is it possible that with this dataset, no further improvements can be done? Should I just tell my boss that this is the maximum accuracy possible given the amount of data we have?


texts:
What is regularization in machine learning?
Why do we need this in machine learning?


texts: I am new to Data science and I am trying to build a neural Network to recognise Musical Instruments .. I am Kind lost here on how to choose the CNN filter size ?


texts: What is semantic gap? Why it exists in AI

texts:
The semantic gap characterizes the difference between two descriptions of an object by different linguistic representations, for instance languages or symbols. The semantic gap can be defined as "the difference in meaning between constructs formed within different representation systems".
I need to find a clear answers to the second part


texts:
I am new to ML and looking to learn with some project. I have a medical imaging dataset where an image (image is a time series of an object so multiple images) has been looked at by radiologist and they have graded it on a scale of 1-5 for some pathology.

Now, I would like to basically use this to predict the pathology on new images. I am guessing there are multiple approaches one could take to do so. Could someone point me to some methods I could try (simple to more advanced) as I would like to also learn about them.

Another issue is that different images are of different sizes. Is this usually a problem for these approaches? I could try something where I can register them so that they are of the same size.


texts: How to rename columns that have the same name?

texts:
I would like to rename the column names, but the Data Frame contains similar column names. How do I rename them?

df.columns

Output:

Index([ 'Goods',
       'Durable goods','Services','Exports', 'Goods', 'Services', 'Imports', 'Goods', 'Services']

Here, there are three goods columns that have similar names. How can I rename a specific column?


texts:
I have a user_data table with various fields, some of them are based on geography.

I'd like to enrich the data with additional columns, like expected_income_in_region, city_population, life_expectancy_in_state, etc...  for each user record.

I'd like to use these extended fields with my sql query (prior to additional python manipulations)

Is the standard way of getting hold of this data , or do I need to scrape from the US Census bureau and format it myself ?


texts: how to save deep learning model and test it after training?

texts: I have a CNN model written using tensorflow for python, the model is for classifying lung CT images (cancer/no-cancer), after training the model with training and validation data and get a reasonable accuracy, after all, that I need to test the model with test data, but I don't know how to do that? how to save the model and using it for testing?


texts: How can I use the shape of a distribution as a feature in machine learning ? Do I use something like the standard deviation ?


texts: Dataset - Sample pdfs for text processing?

texts:
I'm looking for a rather large amount of pdf files for testing my text processing program. Tried looking for an open site to get like some thousand pdfs, but wasn't able to find anything. I don't really know if that is the right place to ask (probably not) but maybe one has a good tip for me.

Thanks in advance.


texts: Why does the same algorithm give very different metrics on similar datasets?

texts:
I used Random Forest and hypertuned the parameters for a binary classification problem on a dataset (dataset A). I got a F1 score of 0.78. I then used a second dataset (dataset B). It was very similar to dataset A (same variables and the distribution of classes in the target variable). I again built and trained a different Random Forest algorithm for dataset B. I expected the f1 score to be around 0.78, but the f1 score for dataset B was 0.50.

Why could there be such a large difference between the f1 scores of the 2 datasets?

Both datasets (A & B) are very similar to each other and I trained separate models on both of them.


texts: A Simple Explanation of ZooKeeper in Hadoop

texts:
Can Anybody Explain me the role of ZooKeeper and ZooKeeper Quorum in a BigData structure?

I'd appreciate if you explain me with a simple practical example instead of abstract theoretical definitions.

Please

As you would explain for a non-technical person


texts: Implementation of Association Rules in Javascript

texts:
I have implemented an interactive visualization with d3.js, javascript to explore the frequency and various combinations of co-occurring item sets. I want to complement the interactive exploration with some automated options.
Does someone know an efficient javascript implementation of the association rules mining ?
My typical scenario will have just up to 30 different items.
There are some good web site with implementations of frequent item set mining (improvements from the initial apriori algorithm): http://www.borgelt.net/apriori.html
Any help is greatly appreciated.


texts:
I'm trying to train a neural network-based model to play a game similar to Pac-Man, except there's no maze. i.e., the player is in a 2-dimensional grid, with dots of food in some locations, and the player need to figure out walking to the cells with the food.
I'd like the player to only see the 11x11 cells around him, i.e. an array of 11x11 cells, with our player in the center, and either a food or no food in each cell.
I tried feeding that array as-is as input to my neural network, but it's learning poorly. This makes sense, as this smells like a problem that you'd use convolution layers for, because this array is kind of like an image.
But unlike an image, we don't care about contours at all. Unlike an image, the 2 cells to the left and right of the centermost cell have vastly different meanings: The position on the left and the right of the player. Food in the former should encourage going to the left, and food on the latter should encourage going to the right. Not so with any similarly-spaced cells in the input. Therefore I'm not seeing how CNN would be a good fit.
Can you think of a good way to represent this input to a neural network?


texts: Is it possible to guess a pretrained CNN accuracy beforehand?

texts:
I am given a dataset of 2D medical images. I am asked to extract image descriptors from the hidden layer of the neural network pre-trained on the ImageNet dataset.
I consider to use two networks: ResNet-50 with imageNet accuracy of X and VGG-16 with imageNet accuracy of Y (X < Y). I wonder if descriptors from ResNet 50 will always be better than the ones from VGG-16 in our pipeline or if it's not possible to know what descriptors are better on the dataset beforehand and if I should rather evaluate both.
I come from an NLP background, don't hesitate to explain to me as if I was 5 years old :p


texts: Understanding a dataset (prior to applying ML models) with no metadata given

texts:
How do you understand a dataset when there is no metadata given (no details about the attributes given in the dataset)? It is difficult to comprehend the attribute names as only the short forms are given.

It's given to me that 'pm2.5' is the target variable. How do I understand which independent variables will affect this target variable?


texts:
I want to perform feature selection, having 128 real-valued standardized features and 1/0 labels. Below are feature a5 density histograms for Classes 1 and 0. The data is skewed, so that Class 1 is about 5% weight.

Next I subtract right curve from left, so that to remove "normal" (relative to this feature) distribution.  I assume that if 50% of feature values lay in e.g. [0.1-0.2] in BOTH classes this means this range is nothing special, regarding my classes. That's why I make subtraction. This hopefully gives me ranges (positive values of resulting curve) where this feature contribute to choosing Class0. Is this assumption correct?

Then I build graphs this way for all features. Below are two different features. I suppose that leftmost is better than rightmost, giving clearer distinction in correlating to one class or another (right is noisier) and the rightmost could be removed, if need to reduce feature number. Is this correct?


texts: How do I convert strings in CSV into integer in Pandas?

texts:
for my supervised classification problem,

I have a train dataset which contains past purchase data of customers and 5 new products are purchased by these customers. I have a test dataset which contains past purchase data of customers. They never bought from these 5 products.

I want to train a predictive model, I need to convert integer purchased data and new products. There are so many different products in my purchased data.

I saw that the mapping was made for true-false to convert 1-0, but in this case what can I do that?


texts: Is this a tried alternative to word embedding for NLP?

texts:
I'm searching for research related to my idea, but apparently cannot articulate it well enough to the search engines to show me what's been published on this.
My idea: in a deep learning context (text classification), instead of inputting the text as integer vectors representing words in a vocabulary, which get substituted with dense vectors (word embeddings), what if instead, you represent each phrase almost like how a vector of "pixels" represent an image (then use 2d convolutions)? For example, each character, digit, and symbol get an integer value, just like how pixels are represented. So abc becomes 0, 1, 2. The phrase could be formatted as a rectangle at word boundaries or something, but the gist of it, a phrase is formulated as a 2d array where each element is a character from some character set.
Is this a known/tried approach, and what name does it go by? From its apparent lack of popularity, I assume this formulation is far worse than word embeddings, but I'm curious about it.


texts: Linear Regression Loss function for Logistic regression

texts:
I was attending Andrew Ng Machine learning course on youtube Lecture 6.4
He says what a cost function will look like if we used Linear Regression loss function (least squares) for logistic regression

I wanted to see such a graph my self and so I tried to plot cost function J with least square loss for a losgistic regression task.
Here is my code
import matplotlib.pyplot as plt
import numpy as np
import math

x = np.random.rand(10000)
# x = np.array([0.1, 0.2, 0.7, 0.4])
y = np.round(np.random.rand(10000))
b = 1

J_list = []
w_list = []
for w in np.arange(-500.5, 500.5, 0.05):
    J = (1/10000)*np.sum((1/2)*np.square(((1/(1 + np.exp(-1*(w*x + b)))) - y)))
    J_list.append(J)
    w_list.append(w)

df = pd.DataFrame()
df['w'] = w_list
df['J'] = J_list

import seaborn as sns
sns.lineplot(x='w', y='J', data=df)

The output of lineplot is

Note: w in my code in theta in Andrew Ng's lecture
If anyone can help me spot my mistake, would be really appreciated.


texts: Reformatting data- Giving each value in a list it's own row while retaining the list's ID

texts:
I am looking to reformat some data. It currently looks like this:
<img src="https://i.stack.imgur.com/Ts8Cn.png" alt="Site_ID Section_ID
1, "A100/2020, B100/1001, C130/2000"
2, "A100/2021, ZW00/2002, W300/0999"
3, "A100/2022, TS100/4000, RW100/0000"" />
Using this as an example, the below is the format i'm trying to achieve:

So each element in the list gets it's own row, but the original site_ID is retained.
A solution in Python would be ideal as this is the only language I am currently comfortable with.


texts:
I am working in emotion analysis of tweets. I collected close to .6 million tweets. I dont want to label them manually, instead i coded a bunch of complex rules to arrive at their label.

Before I use these algo generated labels for my ML solutions, I want to check/analyze how good is this labeling. Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code.


texts: What to do when two algorithms perform better on different set of classes while doing multi-class classification?

texts:
If an algorithm X performed better on a set of classes U and another algorithm Y works better on set of classes V where U and V don's share any class, what improvements can be done from a data scientist view?

Also, please highlight the case where U and V share few classes.


texts: Time series classification with handcrafted features

texts:
I have a dataset with handcrafted statistical features such as follows:
    UserID     AccelX_mean1   accelX_mean2   accelX_mean3  accelY_mean1  accelY_mean2 ....    target

Using sensor data such as accelerometer and gyroscope, statistical features have been created and they are window (3 days, where mean 1, mean 2 and mean 3 are day  1,2,3 accordingly). Every row is a different user.
I need to perform classification, which is binary, and the target can be either 1 or 0 for different users.
Are LSTM/GRU suitable for timeseries data that have windows and are of time series classification problem?
What would an appropriate simple architeture look like, if using recurent networks or are CNN more suitable?
I have read and seen implementations with time series predictions, but the data fed has usually been raw so I am unsure about a suitable method.
Any guidance is appreciated.
Thank you


texts: hyperparameter tuning with validation set

texts:
For what I know, and correct me if I am wrong, the use of cross-validation for hyperparameter tuning is not advisable when I have a huge dataset. So, in this case it is better to split the data in training, validation and test set; and then perform the hyperparameter tuning with the validation set.

In the case that I am programming I would like to use scikit, the yeast dataset available at: http://archive.ics.uci.edu/ml/datasets/yeast; and for example to tune the number of epochs.

First, I have separated my training, validation and test set by using the train_test_split twice according to one answer that I saw here. The loss plot that I got is the following for 1500 max iterations:

Then I wanted to use my validation set with a list of different values for the hypeparameter of max iterations. The graph I obtained is the following (with some warning messages of non-convergence for max_iter values less than 1500):

So, I have the first question here. It seems that for a value of max_iter of 3000 the accuracy is 64% approximately, so I should choose that value for the max_iter hyperparameter; is that correct? I can see from the graph that also the red line of 3000 has a less value of loss than the other compared options.

My program so far is the following:

import numpy as np
import pandas as pd
from sklearn import model_selection, linear_model
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.neural_network import MLPClassifier
import matplotlib.pyplot as plt
from sklearn.model_selection import GridSearchCV

def readFile(file):
    head=["seq_n","mcg","gvh","alm","mit","erl","pox","vac","nuc","site"]
    f=pd.read_csv(file,delimiter=r"\s+")
    f.columns=head
    return f

def NeuralClass(X,y):
    X_train,X_test,y_train,y_test=model_selection.train_test_split(X,y,test_size=0.2)
    print (len(X)," ",len(X_train))
    X_tr,X_val,y_tr,y_val=model_selection.train_test_split(X_train,y_train,test_size=0.2)
    mlp=MLPClassifier(activation="relu",max_iter=1500)
    mlp.fit(X_train,y_train)
    print (mlp.score(X_train,y_train))
    plt.plot(mlp.loss_curve_)

    max_iter_c=[500,1000,2000,3000]
    for item in max_iter_c:
        mlp=MLPClassifier(activation="relu",max_iter=item)
        mlp.fit(X_val,y_val)
        print (mlp.score(X_val,y_val))
        plt.plot(mlp.loss_curve_)
    plt.legend(max_iter_c)

def main():
    f=readFile("yeast.data")
    list=["seq_n","site"]
    X=f.drop(list,1)
    y=f["site"]
    NeuralClass(X,y)

Second question, is my approach valid? I have seen a lot of information over the web and all point to cross validation for hyperparameter tuning, but I want to perform it with a validation set.

Any help?

PD. I have tried early stopping and the results are poor compared to the ones obtained with the method I programmed.

Thanks


texts: Using Amazon Personalize to build a Recommendation System

texts:
I would like to build a recommendation system based only in the items metadata.

I have an input vector with some desirable topics that the user want to read about, for example:  (self-help, yoga, sports)

On the other hand I have a dataset with books with Title, Description, among other fields.

Up to now I am building this model locally with Python, using clustering to group books by similarity.

However, I would like to build a Recommendation System using Amazon Personalize for this case using only item metadata. I do not want to add other info as ratings for now.

Is it possible?
Do you know some example?


texts: Is probabilistic machine learning just the mathematical background of machine learning?

texts:
I wanted to begin with machine learning, I went through the contents of the course on ML by Andrew Ng and found that though the course was based on mathematics, but wasn't too much on the probability or statistics.
But in many university curriculua the book: "Probabilistic Machine Learning: An introduction" by K. Murphy is being used, and it seemed to be full of probability, statistics etc.
I wanted to know if the ML - which is being used in the industry, like the neural networks or other ML techniques - has any statistical or probabilistic foundation which is often not bothered or is it a new emerging field with mathematical basis which is not from probability or statistics?
I really want to know if the ML in academia is really the foundation part of the ML used in the industry or is it the ML in its old form?
And finally, are the contents of the book: "Probabilistic Machine Learning An Introduction", relevant in the ML being used today?
I am aware that these are quite broad and open ended questions from someone entering into the field, I would be happy to informative answers or links?


texts: How to choose sample from a large dataset such that each unique row from the dataset is selected at least once in the sample? Is there a way of doing this in python?


texts: Replacing column values in pandas with specific column with multiple database operation?

texts:
Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can make in one column with multiple operations.

Using Boolean methods to justify results but how can I do in one line code of python to get a replacement of refining/ categorized values to a specific column.

  import pandas as pd
         db = pd.read_csv('Database Path')
         db.head(2)

stoppage time in minutes  | Activity
-------------------------120 | Stopped
-------------------------240 | Stopped

#Need to refine below code for specific requirement as mention above:

db['New Column(Time Category)'] =

db['Stoppage time in minutes'] < 120 OR
db['Stoppage time in minutes'] > 120 & db['Stoppage time in minutes'] < 240 OR
db['Stoppage time in minutes'] > 240 & db['Stoppage time in minutes'] < 360 OR

# So Need Result like as below:

stoppage time in minutes  | Activity | New Column(Time Category)
-------------------------120 | Stopped  | <2 HRS
-------------------------240 | Stopped  | 2 - 4 HRS

Require this solution due to a pool of high numbers of databases.


texts: Text classification and predictive model

texts:
I have collected a lot of data that I would like to analyse and classified. Unfortunately, they are not already labelled, so I am going to do manually.
The dataset consists of texts in Italian and I have not found a lot of models that I could use as a training model for labelling them and classifying them between True and False.

Suppose that I have 30000 of texts, which percentage could be enough to build a model for predicting the rest from that?
Do you have any model that I could build/use once labelled them?


texts: Does hierarchical agglomerative clustering with centroid-linkage suffer from chain-effect?

texts:
It is known that the results of hierarchical agglomerative clustering using single-link method in order to determine the inter-cluster distance suffer form the chain-effect (natural clusters tend to extend trough a line of few points, like in the image below).

Does centroid-linkage have the same disadvantage?


texts: Interpreting Machine Learning Classification Metrics

texts:
I'm trying to understand the results of a classifier I used to predict two possible classes. Here is what I get:

     Precision     Recall     F1-score     Support

 0      0.97        1.00        0.98        341091
 1      0.60        0.18        0.27        12629

 Avg    0.96        0.97        0.96        353720

If I'm trying to predict Class 1, does this model have any value? Can I improve the results of Class 1 prediction?


texts: Why the accuracy of my bagging model heavily affected by random state?

texts: The accuracy of my bagging decision tree model reach up to 97% when I set the random seed=5 but the accuracy reduce to only 92%  when I set random seed=0. Can someone explain why the huge gap and should I just use the accuracy with highest value in my research paper or takes the average with random seed=None?


texts: Ground-truth and feature extraction for predictive modelling

texts:
I have a dataset of users, each user has has daily information about his activities (numerical values representing some measurements of his physical activities).

In addition, each user in each day has a boolean value that represents if he/she took a particular action.

The dataset looks as follow

+------+----------+----------+----------+-------+
|userID|      date| activity1| activity2| action|
+------+----------+----------+----------+-------+
| user1|2016-06-05|       5.3|         6|  false|
| user1|2016-06-04|       3.1|         8|   true|
| user1|2016-06-03|       2.0|        13|  false|
| user1|2016-06-02|       4.7|         1|  false|
| user1|2016-06-01|       1.3|         9|  false|
| user1|   ...ect.|       ...|       ...|    ...|
| user2|2016-06-05|       0.6|         5|   ture|
| user2|2016-06-04|       3.0|         5|  false|
| user2|2016-06-03|       0.0|         0|  false|
| user2|2016-06-02|       2.1|         3|  false|
| user2|2016-06-01|       6.3|         9|  false|
| user2|   ...ect.|       ...|       ...|    ...|
| user3|2016-06-05|       5.3|         0|  false|
| user3|2016-06-04|       5.3|        11|  false|
| user3|2016-06-03|       6.8|         5|  false|
| user3|2016-06-02|       4.9|         2|  false|
| user3|   ...ect.|       ...|       ...|    ...|
+------+----------+----------+----------+-------+

Note that the dataset is not fixed, so one new row is added for each user on every new day. But the number of columns is fixed.

Goal

Build a model that predicts which user is likely to take the action in the near future (e.g. in any of the next 7 days).

Approach

My approach is to build feature vectors representing the activity values for each users over a period of time, and use the action column as a source of ground-truth. Then I feed the ground-truth and the feature vectors to a binary classification training algorithm (e.g. SVM or Random Forest) in order to generate a model able to predict if a user is likely to take the action or not.

Problem

I started by the positive examples that are the users who took the action. To extract the feature vector of a positive example, I combined the activity values in the X (30 or 7 or 1) days preceding the action (the day of taking the action is included).

When I moved to the negative examples, it gets less obvious, I am not sure how to select negative examples and how to extract features from them. This has led me actually to re-question if my way of selecting positive examples and building my the feature vectors was correct.

Questions

How to build the ground-truth of positive (users who did take the action) and negative (users who didn't take the action) examples?
What is a negative example in this case? is it the user who didn't take the action in a fixed period of time? What if he didn't take the action in this fixed period, but he just took it right after?
What are the possible approaches of selecting the ranges of dates to extract feature vectors from.

Rational Question

Is there more suitable approaches (other than classification) to solve this kind of problems?


texts: Numpy ndarray holding string with unknown data type

texts:
I ran the following code in a jupyter notebook cell

ndarrs=np.array(["1.2","1.5","1.6"], dtype=np.string_)

print(ndarrs.dtype)

It returned |S3 as shown below.

Can someone help me understand the meaning of this symbol?


texts:
I am wondering the calculation of perplexity of a language model which is based on character level LSTM model. I got the code from kaggle and edited a bit for my problem but not the training way. I have added some other stuff to graph and save logs. However, as I am working on a language model, I want to use perplexity measuare to compare different results. In tensorflow, I have done it via this answer and it was easy. I have looked for a way doing it in PyTorch and literally no related result on Google. I need some help, and it is really appreciated.

Here is the related code, I believe:

criterion = nn.CrossEntropyLoss()

# create training and validation data
val_idx = int(len(data)*(1-val_frac))
data, val_data = data[:val_idx], data[val_idx:]

if(train_on_gpu):
    net.cuda()

counter = 0
n_chars = len(net.chars)
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    for x, y in get_batches(data, batch_size, seq_length):
        counter += 1

        # One-hot encode our data and make them Torch tensors
        x = one_hot_encode(x, n_chars)
        inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

        if(train_on_gpu):
            inputs, targets = inputs.cuda(), targets.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output, targets.view(batch_size*seq_length))


texts: Audio Analysis : Segment audio based on speaker recognition

texts:
I have audio clips of people being interviewed and am trying to split the audio clips using python such that all speech segments of the interviewee are outputted in one audio file (eg .wav format) & that of the interviewer in another audio file. Speaker recognition needs to be performed using unsupervised learning.

So far I have found a few libraries that perform the speaker diarization task (pyAudioAnalysis, aalto-speech) but none that combine the different speaker segments and output it in separate audio files. How do I segment the audio files & combine them based on the speaker?


texts:
I’m working on a project involving fitting planes to 3D point clouds. The actual plane fitting part is working fine, but I’m trying to decide the best way to actually bound the fitted planes by the data points they are fit to. I need bounded regions of the planes, not just their equations, so I have to decide which points in the dataset “belong” to which plane after the fitting process. The planes are oriented more or less randomly, and can intersect, which complicates things.

It’s easier to picture in 2D with lines instead of planes, so I made two example plots. The first has the raw data in blue. I can find the equations of the lines that match the linear clusters of data. Given these equations, I want to end up with the distinguished clusters in the second plot. I've got something working using a running variance calculation and taking all the points that are below some threshold, but it's not ideal and I know there has got to be a much better way to do it. My gut tells me I need an algorithm that considers the geometry of the clusters beyond just the distance from each point to the plane. Taking all the points within some threshold distance from each plane is problematic for a number of reasons, so I'm looking for something better than that.

Any ideas?


texts: Is it possible to get an ROC curve using Relu activation?

texts: Based on my understanding, given that Relu doesn't provide probabilities unlike Softmax, it's not possible to plot an ROC curve. However, is there some way to convert the output from a Relu to something like an ROC curve. I'm interested in doing so because it's important for my project (image classification) to be able to adjust the classification thresholds.


texts: I need help understanding RNN and toxic classification

texts:
How can an RNN be used for detecting toxic spans (spans of words containing toxic language) in a social media comment? Specifically, what should be the input to the RNN at each time step t? How many outputs (i.e., yˆ) are produced given a comment containing n words? What is each yˆ(t) a probability distribution over?
How about using RNNs for frame classification? Given an article containing n words as input, what should be the input to the RNN at each time step t? How many outputs are produced? What is each yˆ(t) a probability distribution over?
I need help with RNN.


texts: in binary classification where class labels are {-1, 1} is preprocessing needed?

texts:
In machine learning we convert labels using LabelEncoder to convert string ex:{"malignant", "benign"} -> {0, 1}
I am wondering if converting Labels to any other numbers matter, in my scenario to {-1,1}.
If it matters, reason provided will be helpful (or direction of helpful resource)!
Thanks in advance!


texts: Dataset to be used to learn NN regressions in classroom experiments

texts:
Can you suggest me datasets with intermediate size of about 1000-10000 to work with neural network regressions in classroom experiments?
It seems that there are much more datasets for classification tasks than for regression tasks.
I know two good datasets:

Boston Housing dataset

Auto MPG Data Set

But they actually seem to be very small. On the other hand, I know other datasets such as Zillow price but they seem to be very large to explore different types of NN regressions in classroom experiments.


texts: Hi there, My question is about how to read ECDF graphs. I am still quite unsure what the jumps / zig-zags in the graph mean and what is happening when there is a horizontal line and so on. I would be happy if someone can explain me how I am suppose to read this graph and what information I can get from it. Thank you


texts: Looking for modern algorithms using NN Language Model implementations addressing polysemy in NLP tasks, including text classification, question answering and topic modeling. Transfer/Zero-short learning methods are most interesting to find. Any working solutions with BERT and Hugging Face Transformers libraries?


texts:
Principal Component Analysis is a means to reduce the dimensionality of data, if I understand correctly.

So if I have a 1000 sample point 12 dimensional matrix and reduce it to a 1000 sample point 2 dimensional one, then are the values of the sample points themselves changed in some way? or are simply 10 dimensions/columns thrown out and 2 of the original remain?


texts: Clustering: How to cluster multiple CSV files that each represent a Steam user

texts:
I am currently about to do clustering analysis regarding Steam users activity. So I have a thousands of CSV’s, each representing a Steam user and his/her purchased games (with ID and genre). I am planning to use k-modes clustering because my data is categorical, and I have found a python library k-modes. This is what a CSV file looks like, representing a random steam user.

I have researched a lot online, but it seems that many people have all their data in one CSV file, just like this example of stock exchange data.

How could I read each file and represent it as one unique data then do clustering on all the data among the thousands of CSV files?


texts: How to fill missing values by looking at another row with same value in one column(or more)?

texts:
Let's say we have a 6*4 data frame in which third and fourth column contain missing value

1 2   3   L1
4 5   6   L2
7 8   9   L3
4 8   NaN NaN
2 3   4   5
7 9   NaN NaN

I'd like to fill the missing value by looking at another row that has the same value for the first column. So, in the end, I should have:

1 2   3   L1
4 5   6   L2
7 8   9   L3
4 8   6   L2    <- Taken from 4 5 6 L2 row
2 3   4   L4
7 9   9   L3    <- Taken from 7 8 9 L3 row

How can we do it with Pandas in the fastest way possible?


texts:
we are currently performing a K-MEANS under scikit-learn on a data set containing 236027 observations with 6 variables in double format (64 bits).

According to our calculations, the complexity of the algorithm is O(n * k * v * i), with n the number of observations, k the number of clusters, v the number of variables and i the number of iterations which is 300 maximum.

Let (236027 * 3 * 6 * 300) * 64 (size of the double in bits): we have 81570931200 bits which is ~10.1 GB of memory.

However, by running the algorithm on a 600 GB VM, it crashed due to lack of memory.

Are we wrong in our calculation of the space required for KMEANS?

Is KMEANS suitable for this volume?

What solutions are available to us?

Thanks you


texts: Regression of citations by article title

texts:
I have a dataset of some lines (article titles) and the number of citations for an article with this title, what is the best way to build a regression model? I have an idea to launch TF-IDF and use a linear model, but I don't yet understand how true this is. There are about 5000 titles, all of them from the same topic.
Maybe there is some common name for such a task?


texts: Twitter Data-Analyse: What can I do with the data?

texts:
I retrieve data to a specific topic from Twitter and did my sentiment analysis on it. I never did anything in NLP, etc. So what else can I do with that? "Main goal" would be to find out if the Twitter community is against this "topic" or not.
I am also struggling with cleaning the data and I mean by that, that I am unsure how much should I clean on that Tweet.
I would be also glad to get any advise on books, articles, communities, videos...


texts: Why is my keras cnn model giving me the same result for every prediction?

texts:
I have a simple cat or dog cnn model.
My model has an 80% validation accuracy and an 84% training accuracy

import keras
import tensorflow as tf
from keras.models import Sequential
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPooling2D
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
import numpy as np
from keras.models import load_model

classifier = tf.keras.Sequential()

classifier.add(Conv2D(32,(3,3),input_shape = (64,64,3),activation ="relu"  ))

classifier.add(MaxPooling2D(pool_size = (2,2)))

classifier.add(Conv2D(32,(3,3),activation ="relu"  ))
classifier.add(MaxPooling2D(pool_size = (2,2)))
classifier.add(Flatten())
classifier.add(Dense(units = 128, activation = "relu" ))
classifier.add(Dense(units = 1, activation = "sigmoid" ))
classifier.compile(optimizer = "adam", loss = "binary_crossentropy", metrics = ["accuracy"])
from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

training_set= train_datagen.flow_from_directory('dataset/training_set',
                                                 target_size=(64, 64),
                                                 batch_size=32,
                                                 class_mode='binary')

test_set = test_datagen.flow_from_directory('dataset/test_set',
                                             target_size=(64, 64),
                                             batch_size=32,
                                             class_mode='binary')
classifier.fit(training_set,
               steps_per_epoch=int(8000/32),
               epochs=25,
               validation_data=test_set,
               validation_steps=int(2000/32))
print(training_set.class_indices)
from keras.preprocessing import image
test_image = image.load_img("dataset/training_set/cats/cat.1.jpg",target_size= (64,64))
test_image = image.img_to_array(test_image)
test_image = np.expand_dims(test_image, axis = 0)
result = classifier.predict(test_image)


texts: How to predict custom value after using linear regression?

texts:
I'm new to machine learning, and I'm currently practicing by playing around with datasets that I find on Kaggle. Currently I'm trying to predict the price of an Audi, based on the model, mileage and manufacturing year, using a slighly modified version of this set (only columns I use are model, mileage, price and year).
I have the following code written down which makes use of linear regression.
import pandas as pd
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt

df = pd.read_csv('audi.csv', sep=";")
cars=pd.get_dummies(df['model'],prefix='car')

new_df=pd.concat([df,cars],axis='columns')
new_df=new_df.drop(['model'],axis='columns')

x_data=new_df.drop(['price'],axis='columns')
y_data=new_df['price']

model=LinearRegression()
model.fit(x_data,y_data)

This gives me a meagre model.score of 0.8290666609212749.
To test for a custom car that I find on one of the many used car websites, I do the following:
#Audi A3, 42100 mileage, 2016 built
model.predict([[42100,2016,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]])

This works, but as you can see it's a huge hassle to work with because of all the dummy variables. Is there a way of making this simpler?


texts:
I would like to develop an algorithm for Multi-Touch Time-Decay Attribution Model for an e-mail campaign. I have the research paper ready and I would convert the paper into R-script. I am struggling to find the data for the purpose.

Can anyone please give me some pointers on where to find such kind of data or what kind of features are considered for such model, so that I can try to simulate the data?

Thank you in advance.


texts: Consecutive Feature Selection-CV and Model Selection-CV

texts:
I want to ask a question about general workflow of algorithm development.

I want to include a "feature selection with Random Forest" step into my workflow but I have doubts about data leakage.

It is understood that feature selection shouldn't be applied on whole data initially. So, I am planning to implement a workflow like this:

0-)Take All Data

1-) Use a CV approach and for each fold gather the importance scores for all features

(Q1: Even though it is Random Forest and can handle categorical features, my test models will need one hot encoding features. I think I should apply one hot encoding here, too, to get importance scores of encoded variables. Is it true?)

2-) Get average scores for each variable and select the K best feature

3-) With selected K best features and ALL DATA again, use another CV to find best performing algorithm with best hyperparameters.

(Q2: Is it valid to use same ALL data to construct my main algorithm? To minimize data leakage I used CV for feature selection. Now, I am using another CV to select best algorithm with these selected features. It seems proper to me but I have doubts, if you have any suggestion, they are very welcome)

4-) After determining subset of features, best algorithm and best hyperparameters; final model should be constructing without any train-test split, just using all of the data for training. This model will perform the predictions on future data.

(Q3: Can feature selection improve MSE score in regression models? R^2 will not increase of course, but less features may mean a more stable model and increased predictive performance, I think. Or feature selection always only means decreased scoring performance (but increased stability - less overfitting etc.)

(Q4: Would a nested approach be valid for feature selection? In outer loop there will be feature selection CV. Inside each fold, after selecting k best features a model selection CV will be performed with these features. Then get the best performing algorithm among outer loops. But I have 2 limitations here:

Limitation 1: Inside each outer loop, different subset of features may occur and it looks like a show stopper. I can't figure out how to handle it. (I wouldn't use nested approach because of this limitation but I want to learn your opinions on it)

Maybe selected features is not important at this part, only selected model is an output of this stage. When I will train whole data for a final model, I can select best K features again before training. It seems valid to me.

Limitation 2: Of course time constraint. It will require 4 more GridSearch CV loops, which means huge additional time. )

Thanks for reading my long entry. Some of my questions seem too basic, sorry for that. I am a self-learner, I have to pull each valuable information from valuable people.


texts: H2o Model Classification - Confusion Matrix Vs AUC

texts:
I am running a Classification Model in Python using H2o

When I checked the model Performance on test dataset I got

  ModelMetricsBinomial: gbm
  ** Reported on test data. **

    MSE: 0.2166007413446628
    RMSE: 0.46540384758257297
    LogLoss: 0.6222820159893956
    Mean Per-Class Error: 0.3505506508514733
    AUC: 0.7082387540614046
    AUCPR: 0.7134050844687989
    Gini: 0.41647750812280915

    Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.36403865600341695:
              0      1          Error     Rate
    0   0   13060.0 32618.0     0.7141  (32618.0/45678.0)
    1   1   4298.0  46071.0     0.0853  (4298.0/50369.0)

I am little confused by the Output.I expected the accuracy to be 70.82%

But when I look at the confusion Matrix and calculate the correctly classified classes I only get 61.5%

      (13060.0  + 46071.0) /(13060.0+32618.0+4298.0+46071.0) = 61.5%

I understand that confusion Matrix looks at Max F1 to classify classes and AUC looks at an average of threshold to determine how is the classifier performing.

I am not sure how do I predict probabilities based on AUC and not Max F1 so that I get an accuracy of of 70%

I would really appreciate any help on this! Thanks


texts:
When we use a regression algorithm in out dataset it's because we assume that there is a relation between our input data and some quantitative value. This is expressed as :

$y = f(x)+\varepsilon $, where x is an input vector and $\varepsilon$ is the random error term.

Now, this random error term can have any probability distribution ?


texts: How to create new graph vertices from existing one with igraph in R?

texts:
I have the following DF in R representing a directed graph:

    >df

            from          to           timestamp duration
    1 1521695321  1066724476 2014-07-28 14:34:00     6279
    2 1523695522  3446234478 2014-02-24 13:36:00     3630
    3 7621695323  1066234477 2013-12-11 22:11:00     1045
    ...
  101 1521695324  1662354377 2014-08-02 00:49:00     5504
  102 1521695324  1066234478 2014-08-23 02:26:00     1276
  103 1521695324  3446234477 2014-05-03 04:36:00     2228

I have turned this into a graph with igraph:

g <- graph.data.frame(df)

All of the vertices have multiple edges to the other vertices they connected to.
Now, I want to turn some (random) connections of some (random) existing vertices with their respective existing connections and attributes into new, synthetic vertices so, that I will get new, similar vertices to the originals. But I have no clue how to separate the candidates and append to the vertex/edges list. (My task is to find identical entities under different nodes, this is why I want to produce some similar synthetic new vertices from existing ones.)

Additionally, I have to keep track of the original vertex names in the new artificial ones to figure out later which were the original ones (the goal is to recognize the similarity to the latter ones).

Thank you for any help.


texts: Document embedding vs locality sensitive hashing for document clustering

texts:
I would like to compare two methods: locality sensitivity hashing  and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a vector which I would like to use to find similar documents in a very large corpus (potentially more than 100 000 documents). Have anybody ever compared those two methods and what are the advantages of each of them.

Cheers in advance


texts: Compare job ads with a given set of categories (which each consists of terms)

texts:
For a recent research paper, I plan to perform the following, for which I'd kindly ask for your advice.

I obtained a set of a few thousand job ads. I now want to analyse how and whether these job ads include 'content' that has been previously specified in another research paper as individual 'categories'. To make things more precise, there are about 15 existing categories, each of which contains descriptions that explain the category in 2-4 sentences.

Now, I want to understand, which and how many job ads cover the aspects described in each of the 15 categories. A result could be, for example, job ad #1 contains content that matches (or comes close) to the descriptions of categories 2, 5, 8 but misses content that would allow any reference to the remaining categories.

In case you got any references or advice how to approach this task, please let me know. I would suspect that the best approach would be a supervised learning approach.

Best,

Spooz.


texts: About A technique mentioned on ESL to transform an unsupervised into supervised

texts:
I'm reading page 495 of Elements of Statistical Learning.

Here we discuss a technique for transforming the density estimation problem
into one of supervised function approximation. This forms the basis
for the generalized association rules described in the next section.

Let $g(x)$ be the unknown data probability density to be estimated, and
$g_0(x)$ be a specified probability density function used for reference. For example,
$g_0(x)$ might be the uniform density over the range of the variables.
Other possibilities are discussed below. The data set $x_1, x_2, . . . , x_N$ is presumed
to be an i.i.d. random sample drawn from $g(x)$. A sample of size $N_0$
can be drawn from $g_0(x)$ using Monte Carlo methods. Pooling these two
data sets, and assigning mass $w = N_0/(N + N_0)$ to those drawn from $g(x)$,
and $w_0 = N/(N + N_0)$ to those drawn from $g_0(x)$, results in a random
sample drawn from the mixture density $(g(x) + g_0(x)) /2$. If one assigns
the value $Y = 1$ to each sample point drawn from $g(x)$ and $Y = 0$ those
drawn from $g_0(x)$, then
$$μ(x) = E(Y | x) =\frac{g(x)}
{g(x) + g_0(x)}$$

can be estimated by supervised learning using the combined sample
$(y_1, x_1), (y_2, x_2), . . . , (y_{N+N_0}, x_{N+N_0})$
as training data.

I don't understand what are mass and mixture density. And I want to know
why $μ(x) = E(Y | x) = \frac{g(x)}{g(x) + g_0(x)}$.


texts: Weighted Binary Cross Entropy Loss -- Keras Implementation

texts:
I have a binary segmentation problem with highly imbalanced data such that there are almost 60 class zero samples for every class one sample. To address this issue, I coded a simple weighted binary cross entropy loss function in Keras with Tensorflow as the backend.

def weighted_bce(y_true, y_pred):
  weights = (y_true * 59.) + 1.
  bce = K.binary_crossentropy(y_true, y_pred)
  weighted_bce = K.mean(bce * weights)
  return weighted_bce

I wanted to ask if this implementation is correct because I am new to Keras/Tensorflow and the optimizer is having a hard time optimizing this. The loss goes from something like 1.5 to 0.4 and doesn't go down further. Normal binary cross entropy performs better if I train it for a long time to the point of over-fitting. Before anyone asks, I cannot use class_weight because I am training a fully convolutional network.


texts: AttributeError: 'str' object has no attribute 'keys'

texts:
While converting json file to csv,I got the above error
My code is shown below:

import json
import csv
data = open("SRInFlow.json",encoding="utf8")
rows = json.loads(data.read())
r=rows.values()
fieldnames = [b'dateTimeValues', b'timeSeries']
with open('C:/DeepLearning/tensorflowpoc/Codes/SRInFlow.csv', 'w') as f:
    dict_writer = csv.DictWriter(f, fieldnames=fieldnames)
    dict_writer.writeheader()
    for d in r:
  --->    dict_writer.writerow(d)

The error displays in the last line. How to rectify the error? Thanks in advance


texts: Gradient descent multidimensional linear regression - does learning rate affects concurrency?

texts: I wonder if gradient descent for multidimensional regression always finds the right result? I feel like this doesn't always have to be true. I have done some calculations and actually got correct results but not for all learning rates I tried to specify. It's not the case that it is too big, cause sometimes with even having too small learning rates I get wrong results. Please tell my if I make mistake in the calculations, or I'm wrong with divergence of the hypothesis.


texts:
I want to cluster groups, using K-Means, DBSCAN, etc. algorithms, based on lat-lng coordinates along with other features such as dummy variables, continues variables (in different units).
What would be an appropriate way to scale all these features and include them together in those algorithms.


texts: Does dropout require multiple passes of the same data set, as a sort of ensemble method?

texts: I'm a bit confused about dropout -- on one tutorial, it was described as basically an 'ensemble method' of sorts. This implies that you might need to create an ensemble of networks. Is this the case, where you would need to get a consensus from the ensemble at the end? Or is dropout run over one single network?


texts: Obtaining correctly gradient in neural network of output with respect to input. Is relu a bad option as the activation function?

texts:
My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus.

My main purpose is to obtain the gradient of the output w.r.t. the input as accurate as possible. Since relu has a step-like derivative, it's better using softplus as my activation function even if I got better predictions with relu?

Thanks for your time,
Alberto


texts: How to decide the shape of input features, when each data file is of different length?

texts:
To help me understand the benefits and shortcomings of decision trees, KNN, Neural Networks, I wanted to build a simple classifier that classifies into 2 classes (Bird Sound and Non-Bird Sound) using all of the above 3 methods.  So I downloaded a sound dataset from kaggle and was exploring pysoundfile as a module to read the sound file. So the following stub:

data, samplerate = sf.read('xc94652.flac')

returns a numpy ndarray. The shape of data varies with each file, some being of (8637686,) and some being (3227894,). Since each file is of different length the shape of data varies with each file. What are some ways, I can make the shape of data equal? I was planning to have the shape of data set to the smallest length from all the files. But it definitely does not do justice to the sound file dataset. I could lose so many features and so the model accuracy at the end.


texts: what happens to the depth channels when convolved by multiple filters in a cnn (keras, tensorflow)

texts:
I have a $15$-channel time series that I want to convolve using a $1$d CNN ($1\times n$ time-steps kernel). Now, let's say I want to have, as my first layer, $16$ filters. This would imply to my mind that the output would have a depth of $16 \times 15 = 240$, because each filter would be applied to each channel independently. However when I implement this in keras, (using Sequential) the filter dimensions in the summary do not reflect this. Here is a code fragment:

TIME_RANGE = 31
NUMBER_OF_CHANNELS = 15

model = Sequential()
model.add(Conv1D(input_shape = (TIME_RANGE, NUMBER_OF_CHANNELS),
                    filters=16,
                    kernel_size=7,
                    padding='valid',
                    data_format='channels_last'))
model.add(ReLU())
model.add(BatchNormalization())

model.add(Conv1D(   filters=32,
                    kernel_size=5,
                    padding='valid'))

and here is the corresponding summary output:

Layer (type)                 Output Shape              Param #

=================================================================

conv1d_1 (Conv1D)            (None, 25, 16)            1696

re_lu_1 (ReLU)               (None, 25, 16)            0

batch_normalization_1 (Batch (None, 25, 16)            64

conv1d_2 (Conv1D)            (None, 21, 32)            2592

as you can see output's shape along the time-wise axis decreases as expected due to the no-padding argument, from $31$ to $25$ to $21$, but the depth just reflects the number of filters-- so where have all my channels gone? At this point in the architecture I was expecting a depth of $15\times 16\times 32 = 7680$. It seems an implicit $1\times 1$ convolution is occurring somewhere, which I don't think I actually want-- I'd like to do my $1\times 1$ convolutions later on in the network. So what am I missing here?


texts: Finding the dual to an optimization problem on an unsupervised dataset

texts:
We consider the unsupervised dataset $x_1,..x_N \in R^d$ and the optimization problem: $$min_w \,\frac{1}{2}{\left\lVert w \right\rVert}^2,$$ subject to constraints:$$\forall_{i=1}^N: \phi(x_i)^Tw\geq1, \text{with}  \,\phi: \mathbb R^d \rightarrow \mathbb R^h \,\text{is a nonlinear feature map}. $$
Here are my questions:

What is the geometric interpretation of the optimization problem ? One can think of maximal separating hyperplane between the data point and the origin, right ?
How one can derive the dual formulation of the optimization problem ? I guess one will have to maximize a certain quantity, but how one can find it and specify the constrains, I am not sure. I was told  class 1 support vector machines (SVM) can be used, but I do not know how to do that. Can somebody provide some hint or a solution to the problem (in the best case by using first class 1 SVM) ? Many thanks.


texts: Why RANDOM noise images always predicted as BIRD?

texts:
Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%.

However when feeding into 5000 random noise images (Gaussian noise with the same mean and stddev of CIFAR-10), not as expected to be randomly sampled into 10 class, nearly all of them are classified as bird:

Then I examine the feature vector after the last conv layer (before FC layer), with one image corresponding to a 512-dim vector. For each of the 11 categories (CIFAR10+noise), I use the mean of all the 512-dim vectors of images under this category to represent this category. Cosine between two categories' representing vectors is used to measure the DISTANCE between two categories. ($d(v_1,v_2)=\frac{1}{2}(1-cos(v_1,v_2))$ so that the less $d(v_1,v_2)$ is, the closer $v_1$ and $v_1$ are) Then create a graph with 11 nodes, each two nodes are connected with an edge weighted $d(v_1,v_2)$. The MST of this graph:

Distance between bird and noise is exactly rather small.

This also happens if the network is only 2-classification trianed on cats and dogs. For the rest planes, planes, frogs, etc., over 70% of them are misclassified as cat, except for horses that 72% of them are classified as dog. As for random noises, 100% of them are classified as cat.

So some natural questions come to me :

If it's that cat images in training set are more clutter than that of dog? And bird is the most clutter among the 10 categories?
How can 'clutter' be mathematically represented? Spanning in a larger dimension space? Using manifold, PCA or K-means to calculate distance from centroid to bound? Over original images or feature maps?...
Maybe that birds in training set images always appear with a noisy background? So noise is exactly contributing positively for bird?
How to pull out what a neuron in the network really learned to classify the 10 categories?

I searched something about this and maybe it's called Out-of-Distribution Detection? What I'd like to do now is not to boost the accuracy but to figure out why this happens in the interpretable view.

Could you please give me some suggestions on my thoughts listed above or the further experiments designing? Some papers related to this is also appreciated.

Big thanks!


texts: Incremental training and Auto Machine Learning for big datasets

texts:
I built a NLP sentence classifier, which uses vectors from word embedding as features.

Training dataset is big (100k sentences).
Every sentence has 930 features.

I found the best model using an auto machine learning library (auto-sklearn); the training required 40 GB of RAM and 60 hours. The best model is an ensemble of the top N models found by this library.

Occasionally, I need to add some data to the training set and update the training.
Since this autoML library isn't suitable for incremental training, every time  I need to do complete retraining, using more and more memory and time.

How to address this issue? How to do incremental training? Should I quit the usage of this library? For memory and time usage, would it be better to parallelize the training?


texts: Does the mean/median of a set sentence embedded vectors represent anything?

texts:
Please bear with me as I am new to NLP.
I am specifically using tensorflow's universal sentence encoder: https://tfhub.dev/google/universal-sentence-encoder-large/3

I am clustering text based on the cosine similarity of the embedding produced by the model and I want to see what cluster a new text would most likely lie in.  I was going to compare the new text embedding to the mean/median of all the embeddings within a cluster to see which cluster it would most likely lie in.  Would taking the mean/median of the cluster's vectors "represent" the general idea of the cluster or will the vector not represent what I am looking for?


texts: How to obtain unique count of categorical variable based on another categorical variable?

texts:
I have a dataframe with date and cat_1 variables among other variables. I want to get number of unique values of cat_1, per date. Here is an example: Below is a toy dataframe.

         date foo
0  2019-04-21   a
1  2019-04-21   b
2  2019-04-23   c
3  2019-05-01   c
4  2019-05-01   c
5  2019-04-23   a

I want to get is the following:

2019-04-21    2
2019-04-23    2
2019-05-01    1

I got it, but I think it is a bit convoluted.
My code:

a.groupby(by='date')['foo'].apply(np.unique).apply(len)

Is there any other straightforward way to get the same answer?

Thanks


texts:
I have a dataset of past emails sent, when they were sent and if they were opened. What would be the best model to predict the best time to send email?

I was thinking of modeling it as time series with values 1(email opened), 0(email not sent) and -1(email not opened).


texts: Why is the area under the precision-recall curve not used as scoring function more often?

texts:
When I am training and evaluating classifiers or hyperparameter tuning I don't like to look at precision and recall metrics alone because those numbers depend on a threshold that I will set afterwards to cut the output probabilities and make the final prediction. This threshold usually defaults to .5. But that depends entirely on the problem and I always end up changing it. I like to decide its value at the end, once I have selected the ~3 best models from the previous evaluation phase. Sometimes I need a conservative classifier and sometimes a more relaxed one. So before going into that, I need something to use to select the best models and the metric I look at the most is the area under the precision-recall curve. If I had to choose only one metric to look at it would be that. However it is not that popular and I see people using precision and recall numbers alone (threshold=0.5) to evaluate their models.

Why isn't it used to grid-search/model selection more often? Maybe I am missing an important drawback?


texts: Convolutional layer dropout layer in keras

texts:
According to classical paper

http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf

dropout operation affects not only training step but also test step - we need to multiply all neuron output weights by probability p.

But in keras library I found the following implementation of dropout operation:

retain_prob = 1. - level

...

random_tensor = rng.binomial(x.shape, p=retain_prob, dtype=x.dtype)

...

x *= random_tensor
x /= retain_prob
return x

(see https://github.com/fchollet/keras/blob/master/keras/backend/theano_backend.py)

Why x is divided by retain_prob while it should be multiplied?
Or I'm just confused, and multiplying weights is equivelent to dividing output value?


texts:
I have the following image with me:

I want to identify the text, which is the amount mentioned at the bottom of the table. However, the bottom edge of the table goes through the text in this case.

How do I go about trying to extract this text?

PS: For now, I am just using tesseract. However, tesseract was unable to identify this text.


texts:
I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful distance result for the last feature.

For example:

f1 | 1 | 2 | 3 | 4 | 01/01/2019

f2 | 10 | 3 | 12 | 1 | 14/01/2019

Does anyone have any recommendations?


texts: How to combine two differently scaled, but equally important "running" signals into a reward function?

texts:
I asked this question on Artificial Intelligence, but got no answer, so I am moving it here.
I have two signals that I want to use to model a reward for a reinforcement learning algorithm.
The first one is the CPU TIME: running mean from this diagram:

The second one is the running mean of the MAX RESIDUAL from this diagram:

Both signals are equally important, but they have very different scales. I could combine the signals linearly together like this:
$r = w_\rho \rho + w_\tau \tau$
where $r$ is the reward function, $\tau$ is the CPU TIME: running mean, and $\rho$ is the MAX RESIDUAL. The problem is, how to set the weights $w_\tau,w_\rho$ to make the contributions equally important if $\rho$ and $\tau$ are on very different scales?
Reinforcement learning algorithms will learn policies based on the reward, and if one signal has values that are much smaller than the other, it will influence the reward much less, which is not the behavior I would like to model.
Edit: Dataset on Kaggle
Edit: comment from Pedro
It seems that a linear combination of signals is possible for the scaled mean CPU Time (mean to get rid of oscillations) and the scaled MAX residual:


texts: Does classification of a balanced data-set lead to any problem?

texts:
So I came across a bioinformatics paper, where I found a line which says:

  One potential problem with using a training set with
  equal numbers of positive and negative examples in cross-validation
  is that it can artificially inflate performance estimates
  because the number of false-positive classifications is
  proportional to the number of examples classified. So applying
  these methods to all proteins in an organism may
  result in a large number of false-positive identifications.

I am unable to understand how classification of balanced dataset is a problem. Can someone please explain this to me?


texts:
I have been training a multilayer perceptron using Keras to make a prediction on a function similar to that of a normal distribution. I have  input variables , and I have one output value .
When I set my input layer to have  neurons as such

model.add(Dense(4, input_dim=4, activation= 'relu'))

the model learns with a  accuracy.
When I tried to use  neurons in my input layer as such

model.add(Dense(35, input_dim=4, activation= 'relu'))

my model learns it with an  accuracy.
I'm not understanding the logic behind this. Surely you have to have only  neurons for the input layer; what is happening with the other  neurons


texts: How to write a loop for linear mixed effects model with a number of independent and dependent variables

texts:
I'm trying to run a number of linear mixed effects models with different independent and dependent variables. Since there is such a large number that need to be run, I understand that a loop would be the best option. I'd also like to store the relevant summary stats of each model in a new dataframe so that all of the models summary stats can be viewed easily.
Here is an example of a similar dataframe to the one I'm using:
df = data.frame(A = c(1, 2, 3, 4, 5, 6,7, 8, 9, 10, 11),
                B2 = as.numeric(c('56', '43','59','74','61','62','69','80','40','55','58')),
                C2 = as.numeric(c('0','1','0','0','1','1','0','1','0','1','0')),
                D = as.numeric(c('23','28','30','15','7','18','29','27','14','22','24')),
                E = as.numeric(c('1','3','2','5','8','2','5','6','7','8','2')),
                F = as.numeric(c('18','20','19','15','20','23','19','25','10','14','12')),
                G = as.numeric(c('5','3','3','5','2', '39','3','4','3','5','12')),
                H = as.numeric(c('5','4','3','4','2', '1','4','8','3','9','1')),
                I = as.numeric(c('5','7','3','5','2', '1','3','4','3','6','2')))

Basically, my model should look like this:
lmerTest::lmer(B2/C2 ~ D/E/F/G + H*I + (1|A)

where, I need all combinations of B2/C2 with D-G. Eg. lmer(B2 ~ D + H x I + (1|A), lmer(B2 ~ E + H x I + (1|A), etc.
I'm fairly new to this, so I don't even know where to start. Any help is appreciated!
Thanks!


texts: Sales Dataset to determine best model for predicting future sales

texts:
We have a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data AND active inventory, which can be discerned with the column titled "File Type".

We suspect that data science applied to the set--such as a decision tree analysis or logistic regression, or some other machine learning model---can help us generate a value (i.e., probability score) for each product, that can be used as the main determinant evaluating the inventory. Each row in the file represents one product.

It is important to note that we have MANY products in our inventory, and very few of them tend to sell (only about 10% sell each year) and many of the products only have a single sale in the course of a year.

The file contains historical sales data (identified with the column titled File_Type) along with current active inventory that is in need of evaluation (i.e., File Type = "Active"). The historical data shows sales for the past 6 months. The binary target (1 = sale, 0 = no sale in past six months) is likely the primary target that should drive the analysis.

The other columns contain numeric and categorical attributes that we deem relevant to sales.

Note that some of the historical sales SKUs are ALSO included in the active inventory.

A few comments about the attributes included, as we realize we may have some attributes that are unnecessary or may need to be explained.

SKU_number: This is the unique identifier for each product. Order: Just a sequential counter. Can be ignored. SoldFlag: 1 = sold in past 6 mos. 0 = Not sold MarketingType = Two categories of how we market the product. This should probably be ignored, or better yet, each type should be considered independently. New_Release_Flag = Any product that has had a future release (i.e., Release Number > 1)

Your data will be in front of the world's largest data science community. What questions do you want to see answered?

(1) What is the best model to use that will provide us with a probability estimate of a sale for each SKU? We are mainly interested in a relative unit that we can continuously update based on these attributes (and others that we add, as we are able).

(2) Is it possible to provide a scored file (i.e., a probability score for each SKU in the file), and to provide an evaluation of the accuracy of the selected model?

(3) What are the next steps we should take?

If it is possible for me to send you the file, please let me know.

Thanks very much for any suggestions you may provide.


texts: KNN Regression: Distance function and/or vector representation for datetime features

texts:
Context: Trying to forecast some sort of consumption value (e.g. water) using datetime features and exogenous variables (like temperature).
Take some datetime features like week days (mon=1, tue=2, ..., sun=7) and months (jan=1, ..., dec=12).
A naive KNN regressor will judge that the distance between Sunday and Monday is 6, between December and January is 11, though it is in fact 1 in both cases.
Domains
hours = np.arange(1, 25)
days = np.arange(1, 8)
months = np.arange(1, 13)

days
>>> array([1, 2, 3, 4, 5, 6, 7])
type(days)
>>> numpy.ndarray

Function
A custom distance function is possible:
def distance(x, y, domain):
   direct = abs(x - y)
   round_trip = domain - direct
   return min(direct, round_trip)

Resulting in:
# weeks
distance(x=1, y=7, domain=7)
>>> 1
distance(x=4, y=2, domain=7)
>>> 2

# months
distance(x=1, y=11, domain=12)
>>> 2
distance(x=1, y=3, domain=12)
>>> 2

However, custom distance functions with Sci-Kit's KNeighborsRegressor make it slow, and I don't want to use it on other features, per se.
Coordinates
An alternative I was thinking of is using a tuple to represent coordinates in vector space, much like we represent the hours of the day on a round clock.
def to_coordinates(domain):
    """ Projects a linear range on the unit circle,
        by dividing the circumference (c) by the domain size,
        thus giving every point equal spacing.
    """
    # circumference
    c = np.pi * 2

    # equal spacing
    a = c / max(domain)

    # array of x and y
    return np.sin(a*domain), np.cos(a*domain)

Resulting in:
x, y = to_coordinates(days)

# figure
plt.figure(figsize=(8, 8), dpi=80)

# draw unit circle
t = np.linspace(0, np.pi*2, 100)
plt.plot(np.cos(t), np.sin(t), linewidth=1)

# add coordinates
plt.scatter(x, y);

Clearly, this gets me the symmetry I am looking for when computing the distance.
Question
Now what I cannot figure out is: What data type can I use to represent these vectors best, so that the knn regressor automatically calculates the distance? Perhaps an array of tuples; a 2d numpy array?
Attempt
It becomes problematic as soon as I try to mix coordinates with other variables. Currently, the most intuitive attempt raises an exception:
data = df.values

Where df is:

The target variable, for simple demonstration purposes, is the categorical domain variable days.
TypeError                                 Traceback (most recent call last)
TypeError: only size-1 arrays can be converted to Python scalars

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-112-a34d184ab644> in <module>
      1 neigh = KNeighborsClassifier(n_neighbors=3)
----> 2 neigh.fit(data, days)

ValueError: setting an array element with a sequence.

I just want the algorithm to be able to process a new observation (a coordinate representing the day of the week and temperature) and find the closest matches. I am aware the coordinate is, of course, a direct representation of the target variable, and thus leaks the answer, but it's about enabling the math of the algorithm.
Thank you in advance.


texts: Test RMSE of polynomial regression drops when using more variables?

texts:
I am testing polynomial regression for a data set of 50 variables and a sample size of 5000. I ordered the coefficients of the linear model from high to low and then made different models using the p most explanatory variables. The RMSE values of these models are shown in the following figures. For polynomial regression of degree=2 everything seems to be normal, but for degree=3 something strange happens. (The dotted line shows the location of minimum RMSE)

I cannot understand why the test RMSE drops when using more than 27 variables? It seems to start underfitting rather than overfitting with more variables. Interestingly, this happens at the same time as the training RMSE becomes around 1e-14. Meanwhile, the adjused R-squared shows the following strange behaviour at that moment:

Am I using just too many dimensions for this polynomial regression? Or is there another reason why this is happening? I would love to understand what is going on. I will quickly move on to better algorithms, but I just wanted to make a benchmark for my ML project. Thanks in advance!


texts: Machine Learning Dataset: Easy enough for fully connected, but not easy enough for logistic regression

texts:
I was wondering if someone could direct me to a dataset for a classification task with the following conditions:

Multinomial logistic regression alone does not learn a good classifier
A series of fully connected layers is able to learn a good classifier
The task is not MNIST

Thank you


texts:
I am currently using a random forest model for classification, however I am unsure how the feature selection technique "varImp" works on R. I understand the context of variable importance, however when I implement it in R it doesn't seem to produce the results I expect.

When removing the most important variable (of 31 features), the model's accuracy does not decrease. I would expect it to as it should be contributing the most to the model's ability to classify.

Could someone please explain what this function is doing?


texts: SmoteBoost: Should SMOTE be ran individually for each iteration/tree in the boosting?

texts:
As per the paper on SmoteBoost, SMOTE is ran for each iteration of the boosting, generating N samples, which are further added to the original training data and the weight distribution of the instances are modeified acccordingly. However, this would result in increase in the size of the training data, wherein, we add N sythetic examples of minority class in each iteration.

So, if we have T trees for the booster, we would have added N*T sythentic examples for the minority class for Tth tree. Should we rather remove the N sythetic examples after each iteration(completion of each tree) or run SMOTE just for the first tree in the booster.


texts: Finding optimal threshold in extreme value theory - Peak Over Threshold-method (POT)

texts:
I am using r and doing a project with extreme values. Now I need to find optimal threshold based on charts outputs but I can't find any information how to interprete them. I am using ismev package:

u1=quantile(rainData,0.85)
u2=quantile(rainData,0.95)
ismev::gpd.fitrange(rainData, u1, u2)

ismev::mrl.plot(rainData)

How can I "read" optimal threshold from these two charts? What should I look at? It doesn't have to be the EXACT best thereshold.


texts: Why is this Binning by Median code wrong?

texts:
I was working on Binning by Mean, Median and Boundary in R.

# R CODE
a=c(20.5, 52.5, 62.6, 72.4, 104.8, 63.9, 35.3, 83.9, 37.4, 71.6, 74.6, 44.5, 66.6, 56.1, 45.3, 37.2)
a=sort(a)
binsize=4
median(a[1:4])
# BINS ARE
for(i in 1:length(a))
{
  if(i%%binsize==0)
  {
    print("HI")
    print(a[i-3])
    print(a[i-2])
    print(a[i-1])
    print(a[i])
  }
}

# BINNING BY MEAN
sum=0
for(i in 1:length(a))
{
  sum=sum+a[i]
    if(i%%binsize==0){
      avg=sum/binsize
      sum=0
      print(rep.int(avg,binsize))
    }
}

# BINNING BY MEDIAN
i=1
for(i in 1:length(a)){
  if(i%%binsize==0)
  {
    print(rep.int(median(a[i-3:i]),binsize))
  }
}

Can anyone tell me why binning by median is giving me wrong output.
The median(a[i-3:i]) for 1st bin returns a value which is not same as median(a[1:4]) for 1st bin. Why?


texts: Can some one please explain me what is the difference between one class SVM and SVDD(support vector data description)


texts: Machine Learning Identification and Classification, based on string contents: General advice

texts:
I have just very recently started to develop an interest in machine learning, and I have a particular problem in mind that I would like to start to explore.

I would like to train a system to automatically classify various attributes of an item, based on what's in a string.

Let's say I have a long list of various mutual funds, like:

Ticker  Fund Name
------  ---------
ABNAX   ABC Bond Fund, Inc: Bond Inflation Strategy
ALYSX   ABC Bond Fund, Inc: Credit Long/Short Portfolio; Advisor Class
AGRXX   DEF Bond Fund, Inc: Government Reserves Portfolio; Class 1 Shares
HIYYX   FGH Bond Fund, Inc: High Yield Portfolio; Advisor Class Shares
HIYAX   FGH Bond Fund, Inc: High Yield Portfolio; Class A Shares
...

… And so on.

I have a large data set that contains "complete" classifications, which have Fund Names similar to the ones above, and – in addition – a human has already given the training set items certain attributes. For example:

AIISX   Allianz Funds Multi-Strategy Trust: AllianzGI International Small-Cap Fund; Class R6 Shares

Which will have the associated attributes:

Strategy: Multi-Strategy
Geography: International
Capitalization: Small-Cap
Share class: R6

The challenge for the machine learning system will be to assign the right value to an attribute, when there are values "competing" on the same attribute. Let's say that a certain fund can have Strategy: Long-Short and Strategy: High Yield at the same time – and both terms are present in the Fund name. The system should select the right one, based on exposure to historical bias present in the training data set.

Question

I am interested in getting a grasp of which machine learning methods and algorithms that would be able to "learn" how to classify an item, based on a large set of examples with human-classified attributes, as indicated above.

I am a complete beginner to machine learning, except for some basic knowledge of statistics, so I would just like to be pointed in a general direction.

Can/should this be accomplished with something like multiple regression, or are we looking at something else? Is some sort of natural language processing needed – or is basic keyword pattern recognition enough?

Lastly, which terminology or labeled area of expertise would summarize this problem description?


texts: Predict the target audience for a new brand using data from other brands and customers buying behavior

texts: Assume a company has a large database about wine, including brand, the taste of the wine, year, place of production, etc, and data of customers' purchase behavior. Now if there is a new brand coming to the company and asking who should the brand sell the product to, how should I approach this question? What kind of data/features should I use from the database? Thank you very much!!


texts: The meaning of random word dropout in NLP

texts: I have been reading the early paper on pre-training in NLP (https://arxiv.org/abs/1511.01432) and I can't understand what random word dropout means. The authors completely ignore explaining this method as if it was a standard thing. Can someone explain what they really do and what is the purpose of that?


texts: Share the experiences of regularization of a LSTM model

texts:
There are five parameters from an LSTM layer for regularization if I am correct.
To deal with overfitting, I would start with

reducing the layers
reducing the hidden units
Applying dropout or regularizers.

There are kernel_regularizer, recurrent_regularizer, bias_regularizer, activity_regularizer, dropout and recurrent_dropout.
They have their definitions on the Keras's website, but can anyone share more experiences on how to reduce overfitting?
And how are these five parameters used? For example, which parameters are most frequently used and what kind of value should be input?
Is there other effective ways to regularize an LSTM model?


texts:
I'm using LSTM to predict financial data. As input data I use log returns and I want to predict the next day market movement. Do I need to retrain the ANN every day in order to keep time consistency or I can simply train ANN once for example with the data from 2010 to 2018 and predict market movement in 2019?

I'm using daily data


texts: Unable to transform (greatly performing) Autoencoder into Variational Autoencoder

texts:
Following the procedure described in this SO question, I am trying to transform my (greatly performing) convolutional Autoencoder into a Variational version of the same Autoencoder. As explained in the cited post, this essentially boils down to adding the KL divergence in the loss and to adding a layer for the sampling.

My data is composed of 1D time series.

This is my code:

from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
from keras.layers import Input, Dense, Lambda, Reshape, Conv1D, MaxPooling1D, UpSampling1D, Flatten
from keras.models import Model
from keras import backend as K
from keras import losses
from keras.callbacks import Callback

X_train = np.loadtxt(..)# here I load my data
X_test = np.loadtxt(..)

# dimension of input
n_x = X_train.shape[1]

X_train = np.reshape(X_train, [1500, n_x, 1])
X_test = np.reshape(X_test, [500, n_x, 1])

# dimension of latent space (batch size by latent dim)
batch_size = 100
latent_dim = 2

x = Input(shape=(n_x, 1))
conv_1 = Conv1D(8, kernel_size=16,
                padding='valid', activation='tanh')(x)
maxp1 = MaxPooling1D(2)(conv_1)
conv_2 = Conv1D(16, kernel_size=64,
                padding='valid', activation='tanh')(maxp1)
maxp2 = MaxPooling1D(2)(conv_2)
flatten = Flatten()(maxp2)
hidden = Dense(700, activation='tanh')(flatten)
hidden2 = Dense(400, activation='tanh')(hidden)

z_mean = Dense(latent_dim)(hidden2)
z_log_var = Dense(latent_dim)(hidden2)

epsilon_std = 1.0
def sampling(args):
    z_mean, z_log_var = args
    epsilon = K.random_normal(shape=(batch_size, latent_dim),
                              mean=0., stddev=epsilon_std)
    return(z_mean + K.exp(z_log_var/2) * epsilon)

z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])

decoder = Dense(700, activation='tanh')(z)
decoder = Reshape((700, 1))(decoder)
de_conv_1 = Conv1D(16, kernel_size=64,
                   padding='valid', activation='tanh')(decoder)
upsamp = UpSampling1D(2)(de_conv_1)
de_conv_2 = Conv1D(8, kernel_size=16,
                   padding='valid', activation='tanh')(upsamp)
upsamp = UpSampling1D(2)(de_conv_2)
flatten = Flatten()(upsamp)
x_decoded_mean = Dense(n_x)(flatten)
x_decoded_mean = Reshape([n_x, 1])(x_decoded_mean)

epochs = 200

vae = Model(x, x_decoded_mean)
vae.summary()

## define loss (sum of reconstruction and KL divergence)
def vae_loss(y_true, y_pred):
    # E[log P(X|z)]
    recon = K.sum(K.square(y_true - y_pred), axis=[1, 2])
    # D_KL(Q(z|X) || P(z|X))
    kl = K.exp(z_log_var) + K.square(z_mean) - 1. - z_log_var
    kl = 0.5 * K.sum(kl, axis=-1)
    return K.mean(recon + kl)
def KL_loss(y_true, y_pred):
    return(0.5 * K.sum(K.exp(z_log_var) + K.square(z_mean) - 1. - z_log_var, axis=-1))
def recon_loss(y_true, y_pred):
    return K.sum(K.square(y_true - y_pred), axis=[1, 2])
# compile and fit
vae.compile(optimizer='adam', loss=vae_loss, metrics = [KL_loss, recon_loss])
vae_hist = vae.fit(X_train, X_train, batch_size=batch_size, epochs=epochs,
                            validation_data = (X_test, X_test))

This results in horrible reconstructions. I am wondering if this has to do with the balance between KL and reconstruction loss, which is a notorious problem in Variational Autoencoder. However, even trying with different variations of using mean, sum etc in the definition of the KL and Reconstruction losses, the performance remains very poor. I have also tried with KL divergence Annealing, as explained e.g. here, but basically the problem I face in that case is that once the KL loss kicks in, the total loss stops decreasing.


texts: How to handle a valuable feature that is missing on 99\% of the samples in the data set?

texts: Suppose we have an input feature that is highly predictive of the outcome we want to predict. However, the feature is missing on 99% of the samples in the data set. What is the best way to use this feature in building a prediction model? Any ideas would be greatly appreciated!


texts: What is the appropriate statistical test to compare the MAUC scores from two machine learning classifiers?

texts:
I would like to compare the scores of two multi-class classifiers. I have calculated the MAUC score for each of the algorithms, and now I want to see whether there is a statistical difference between the results.
From what I have read so far, the McNemar test seems to be a good alternative, however, I am not sure how exactly to use it. In this article, there is an example o how to use McNemar's test to compare the accuracy between algorithms.
The scores I would like to compare are 0.809 and 0.812. By trying to follow the tutorial, I came up with this table on which I want to apply the McNemar test implemented here.
              model 1(correct)  |   model 1 (wrong)

model 2 (correct)        0.809        |     0.003
                                      |
model 2 (wrong)          0.000        |      0.191

Could someone please help me out in here? I'm very confused. Thank you!!


texts: Feature extraction for sentiment analysis

texts:
I am working on a group project for my capstone course and we have been tasked with creating a sentiment analysis tool with Python business logic and (L/W)AMP everything else.

We have good feedback for every part of our project plan except for feature extraction. One of our advisors insists that we should have ~15 different kinds of features.

Currently we only use unigrams and are having a hard time finding others that are practical to implement with our small set of data (~50 items) and within our time limit (~2 weeks to fully implement).

What are feature extraction techniques that are useful for sentiment analysis and work on smaller datasets? They should be able to be implemented quickly or already exist in a Python library.


texts: How to run R scripts without closing X11

texts:
I would like to run an R script using a single command (e.g. bat file or shortcut).

This R script asks the user to choose a file and then plots information about that file. All is done via dialog boxes.

I don't want the user to go inside R - because they don't know it at all.

So, I was using r cmd and other similar stuffs, but as soon as the plots are displayed, R exits and closes the plots.

What can I do?

Thanks for your help.


texts: Unsupervised learning-Feature selection in python

texts:
I need to select the most important features from my data frame before starting with nearest neighbours problem.

Which methods are the best to do this? My data frame has around 8 categorical features and 2 continuous features but no target variable. The problem is that I have three categorical features which can only be one hot encoded and once I do that it explodes the data into 47 OHE variables.

So considering these situations, what would be the best method to do the feature selection?


texts: How to encode high cardinality categorical data?

texts:
I have a dataset of 1600 rows and 28 columns. Only one column is partially complete with 1300 records. The rest is NaN. I did a value count of this columns and it has 84 different categories that are nominal. What is the best way to impute this column.
I need to convert these in numbers impute it and then convert back.
I understand that One-Hot encoding does not work in this case because of the high cardinality.
What is the best way to approach this problem?


texts:
I'm training a bidirectional lstm with attention on a dataset with text data and six target classes.

The F1 score on the test set by class is about 0.7 for four of the classes, and about 0.35 for the other two. Not, only that, the model is confusing those two classes and I can't find the pattern in the data by which it's making the prediction for them.

I'm using
fasttext
pretrained word vectors.

I couldn't find a proper solution for such a problem.

So, my question is: Is there a research on this topic that you could guide me to, or a solution that I could use?


texts: Output of evaluation metric for XGBoost - is it cumulative?

texts: On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been created so far?


texts:
I am working on an assignment that is teaching how to plot and label using matplotlib using Python. Science or math is not my background. I have been given the formula for calculating the geostrophic wind and we are to plot it (on the y-axis) versus the latitude on the x-axis.

I know how to plot give an x and a y. Beyond that, the formula is not making sense to me given my lack of background in the area.

Furthermore, there is a key to translate the symbols to the correct python code, but I don't know the English names to the symbols, making it very difficult.

For example, I am given

r'$x^{10}$'
r'$R_^{final}$'
r'$alpha^{\eta}$'

The first two are superscript and subscript. That I understand. But how this helps with the formula calculations I do not know.

I am given the values to put into the formula as well. An explanation of the order of operations would help.
If,
g0=9.81 ms-‐2;
ΔZ=60m;
Δn=2x10^5m;
and
f=2Ωsin(φ)

My question is how do I put the values into the formula and then plot them in matplotlib? is it as easy as x and y?


texts: CreateML mlModel fails to infer a few classes

texts:
I have a Image Classification problem (black and white stick figures)
The issue is that two classes never get inferred.
The training dataset has 500 classes with 100 or more (299x299,1) samples per class. The classes that fail to be recognized have been augmented from 100 samples to 140+ samples. Most recently I have trained CreateML with autosplit, but have used a separately generated Validation dataset.
No difference in results.  The internal "FeaturePrint" model in CoreML just seems to refuse to recognize these two glyphs.
Of note, a simple TensorFlow/Keras model (8 layer CNN/ANN) works just fine for all classes. However, the tf model has poorer performance on untrained, 'wild' input images, and the tf model when converted to ML format for deployment is 160 MB, whereas the CreateML model is 8 MB.
Any suggestions on wrangling CreateML?