texts: Scikit Learn OneHotEncoded Features causing error in classifier

texts:
I’m trying to prepare data for input to a Decision Tree and Multinomial Naïve Bayes Classifier.
This is what my data looks like (pandas dataframe):
Label  Feat1  Feat2  Feat3  Feat4

0        1     3       2      1
1        0     1       1      2
2        2     2       1      1
3        3     3       2      3

I have split the data into dataLabel and dataFeatures.
Prepared dataLabel using dataLabel.ravel()
I need to discretize features so the classifiers treat them as being categorical not numerical.
I’m trying to do this using OneHotEncoder:
enc = OneHotEncoder()

enc.fit(dataFeatures)
chk = enc.transform(dataFeatures)
from sklearn.naive_bayes import MultinomialNB

mnb = MultinomialNB()

from sklearn import metrics
from sklearn.cross_validation import cross_val_score
scores = cross_val_score(mnb, Y, chk, cv=10, scoring='accuracy')

I get this error: bad input shape (64, 16)
This is the shape of label and input:
dataLabel.shape = 72
chk.shape = 72,16
Why won't the classifier accept the onehotencoded features?
EDIT: Adding how I got dataFeatures
dataFeatures = data[['Accpred', 'Gyrpred', 'Barpred', 'altpred']]
Y = dataLabel.ravel()


texts: I have a dataset in which some columns have header but the other ones don't have. Therefore, I want to add column name only to those who doesn't have column name. I know how to add column names using header but  I would like to know how to do this using index so that I can add column names only to the empty ones. Any suggestions will be appreciated.


texts: Which algorithm can i use for predicting length of stay in coming year based on historical claims data?


texts: Precisely how does the cox.zph function in the R package "survival" scale Schoenfeld residuals?

texts: I am using cox.zph() from the R package "survival" to find baseline covariates which contravene proportional hazards. zph() gives a two-tailed p-value of a chi_square test applied to Schoenfeld residuals; the scaled residuals from zph() can also be plotted. I have also used coxph.residuals() from "survival" to obtain the raw Schoenfeld residuals, and when I plot these they are usually similar to the zph() plots. However this similarly is not always the case. What formula does zph() use to scale the residuals?


texts: Fastest batch perspective transform for image augmentation

texts:
I need to do some augmentation for my training images for a neural networks.
The problem is that even when loading batches in parallel, the augmentation is taking longer to perform than the network training.

The problem is in the perspective transforms I'm applying.
Both cv2.warpPerspective and torchvision.transforms.functional.perspective (with PIL images) are taking roughly the same time.

But both of them do it image by images (which seems very inefficient to me).

So, what are the best (fastest and batch-wise) ways of performing perspective transformation?

I think of doing it directly with PyTorch functions, but I don't know the algorithm. (Accepting a good algorithm as answer)
Any other package that can do it faster?


texts: Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

texts:
I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling.
I am pretty comfortable with the sklearn Pipeline object that I use for pre-processing but I am not sure if I should include data cleaning, data extraction and feature engineering steps that are typically more specific to the dataset I am working on.
My general thinking is that the pre-processing phase would include operations on the data that need to be done after splitting it so as to avoid data leakage. These would typically be:

Scaling
Imputing (if not by constant value)
Encoding

On the other hand, data cleaning or feature engineering are operations that could be performed on the whole data set, for example:
def clean_price(data):
    """Clean the price feature."""
    data['Prezzo_EUR'] = data['Prezzo'].str.split('€').str[1].str.replace('.', '').astype('float')
    return data

def create_energy_class(data):
    """Create energy class feature."""
    data['Classe_energetica'] = data['Efficienza energetica'].str.extract(r'([A-G]\d?)')
    return data

These operations are very specific to the dataset and don't cause data leakage, and I don't really see any value in including them in an sklearn pipeline using the FunctionTransformer object for example. I could easily have 20+ simple operations like the ones shown above and what I do instead is build a custom pipeline connecting all of the functions:
def create_pipeline(list_functions):
    """Pipeline function for data cleaning steps."""

    def pipeline(data):
        out = data
        for function in list_functions:
            out = function(out)
        return out

    return pipeline

So I basically end up having two separate pipelines, one for operations to be performed on the whole dataset before splitting it (and that are very specific to the dataset), and one for operations after splitting (what I call pre-processing).
What is the approach to building scalable data pipelines that include data cleaning, data extraction, feature engineering, pre-processing before modeling? Are there better or more standard ways of going about this?
Also please correct me if you think my terminology is not accurate.


texts: Predicting Sequence based on Tabular Features

texts:
I have a dataset regarding a specific junction on a piece of road and the cars that will cross that junction. I am trying to predict the order that cars will pass the specific junction given a set of tabular features. My dataset looks similar to the following:

Target      car1_type  car2_type  car3_type      car1_positionId   car2_positionId   car3_positionId ...
3,1,2       1          2          3              3                 6                 8
2,1,3       8          4          9              1                 4                 2

My features inlcude the type of car (car_type) as well as the position of each car (car_positionId). The position represents a region of road that the car is on. Each row above represents an observed case of 3 cars crossing a junction.

I am trying to predict the target column (the order that the cars will pass over the junction) based on the features given. There is additional complexity in that I also do not know how many cars there will be. There could be just one (trivial case) or there could be up to 20.

My question is what machine learning algorithm could I use to help me predict the order.


texts: Simplest way to build a semantic analyzer

texts:
I want to build a semantic analyzer i.e., to find how similar the meaning of two sentences are.
For example-
English: Birdie is washing itself in the water basin.
English Paraphrase: The bird is bathing in the sink.
Similarity Score: 5 ( The two sentences are completely equivalent, as they mean the same thing.)

I have to find the similarity between the meaning of those sentences.

Here is a github repo of what I want to implement.
https://github.com/anantm95/Semantic-Textual-Similarity

Is there any simpler approach?


texts: Multiple digits MNIST and transfer learning

texts:
I have a sample of 50,000 images, some of which are shown below:

 $\qquad$
  $\qquad$  $\qquad$  $\qquad$

Associated to these images are labels for the digit with the largest pixel size. My goal is to build a machine learning model to predict the largest digit in an image by pixel size.

To that end, I used transfer learning on the resnext model, but only found an accuracy of 60%.

Given that this implementation uses transfer learning to train a model to predict MNIST digits, I would now like to crop each training image to retain only the largest digit and then train the model using the linked implementation.

So, my question is, how I do crop the training images to retain only the digit in each image with the largest size.


texts: Recommending movies with additional features using collaborative filtering

texts:
I am trying to build a recommendation system using collaborative filtering. I have the usual [user, movie, rating] information. I would like to incorporate an additional feature like 'language' or 'duration of movie'. I am not sure what techniques I could use for such a problem.

Please suggest references or packages in python/R.


texts: How to adapt the softmax layer for multiple labels?

texts:
Question: A image has multiple labels. Given a set of image with labels, how to adapt the softmax layers?

My idea:

Encode multiple labels to 0-1 variables. Use logistics regression as the output layer.
Choose the top X labels in softmax output. But I do not know

how to determine the threshold.
how to modify the code in a pretrained net. Let's say

{
      "op": "null",
      "param": {},
      "name": "softmax_label",
      "inputs": [],
      "backward_source_id": -1
    },
    {
      "op": "Softmax",
      "param": {"grad_scale": "1"},
      "name": "softmax",
      "inputs": [[510, 0], [511, 0]],
      "backward_source_id": -1
    }

Any recommendations on tutorials are appreciated. I have some theory background from the book ESL.


texts: Is there a way to measure correlation between two similar datasets?

texts:
Let's say that I have two similar datasets with the same size of elements, for example 3D points :

Dataset A : { (1,2,3), (2,3,4), (4,2,1) }
Dataset B : { (2,1,3), (2,4,6), (8,2,3) }

And the question is that is there a way to measure the correlation/similarity／Distance between these two datasets ?

Any help will be appreciated.


texts: Rating Sports Teams: How to perform least squares minimization with a constraint?

texts:
I am trying to rate NFL teams by minimizing the sum of squared errors subject to a constraint. My data looks like:
dat = {"Home_Team": ["KC Chiefs", "LA Chargers", "Baltimore Ravens"],
       "Away_Team": ["Houston Texans", "Miami Dolphins", "KC Chiefs"],
       "Home_Score": [34, 20, 20],
       "Away_Score": [20, 23, 34],
       "Margin": [14, -3, -34]
      }
df = pd.DataFrame(dat)
df

    Home_Team        Away_Team      Home_Score  Away_Score  Margin
0   KC Chiefs        Houston Texans 34          20          14
1   LA Chargers      Miami Dolphins 20          23          -3
2   Baltimore Ravens KC Chiefs      20          34          -34

The Margin is Margin = Home_Score - Away_Score. My goal is to come up with a numerical rating for each team such that the average of all the teams' ratings is zero. Hence, if the Chiefs have a rating of 3.0, then they are 3 points better than the average team.
Given these ratings, we generate predictions in this way: the home team's predicted margin of victory is Home_Edge + Home_Rating - Away_Rating, where Home_Edge is the home field advantage (a constant for all home teams), Home_Rating is the home team's rating, and Away_Rating the away team's rating.
The error in a prediction is prediction - Margin, and I want to minimize the sum of squares of these errors. I am trying to do this using scipy.optimize in the following way:
# Our objective function, where x is our array of parameters,
# x[0] is the home edge, x[1] the home rating, and x[2] the away rating
# Y is the true, observed margin
def obj_fun(x, Y):
    y = x[0] + x[1] - x[2]
    return np.sum((y - Y)**2)

# Define the constraint function. We have that the ratings average to 0
def con(x):
    return np.mean(x[1])

# Constraint dictionary
cons = {'type': 'eq', 'fun': con}

# Minimize sum of squared errors
from scipy import optimize

# Initial guesses (numbers I randomly thought of in my head)
home_edge = 0.892
home_ratings = np.array([1.46, 9.67, -0.82])
away_ratings = np.array([-3.10, -6.57, 1.46])
x_init = [np.repeat(home_edge, 3), home_ratings, away_ratings]

# Minimize
results = optimize.minimize(fun = obj_fun, args = (df["Margin"]),
x0 = x_init, constraints = cons)

print(results.x)
[-2.9413615   0.          4.72534244  1.46        9.67       -0.82
 -3.1        -6.57        1.46      ]

I'm not exactly sure what's going wrong, but I want my output to have 6 solutions, not 9. We should have one solution for the home edge and five more (one for each team). What's going wrong? Thank you!


texts:
How do you calculate the probability that a certain number of hyper-parameter combinations contains the optimum combination?
Side Note: After doing more research, I have decided that Randomized Search is best used against a very large set of values. For example, when I want to test a ludicrously large range of split/bin percentages for Random Forest Ensemble. Is that correct?
Summary
I use SKLearn's RandomizedSearchCV module. It will test a certain number of  hyper-parameter combinations out of the supplied "dictionary" of hyper-parameters and their possible settings. I want to calculate the probability that the number of repetitions I specify will include the best set of hyper-parameters. I assume that the best set is included in the "dictionary" or mapping I supplied.
The number of repetitions - and thus the number of hyper-parameter combinations tested by RandomizedSeachCV - is specified using its n_repetitions parameter. I supply a "dictionary" or map with a machine-learning algorithm's hyper-parameters and, for each hyper-parameter, a list of possible settings I want to test.
Here is a link to the documentation.

RandomizedSearchCV: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html
MLPClassifier: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html

As a side note, is there any standard probability of success that I should stick to, for example 80% probability that the optimum set of hyper-parameters would be included in the tests?
Example
I assumed that to calculate the total possible combinations of hyper-parameters, you count all possible unique sets without regard for order. In other words, you use the formula for combinations (and not permutation) using the total number of hyper-parameter settings to be tested.
For example, assume I were calculating the maximum number of combinations for MLPClassifier, SKLearn's implementation of supervised neural networks. I want to test all four available activation functions (‘identity’, ‘logistic’, ‘tanh’, ‘relu’) and all three weight optimization "solvers" (‘lbfgs’, ‘sgd’, ‘adam’).
Again, given that:

For that test having two hyper-parameters with 4 and 3 possible options, the total number of possible hyper-parameter "sets" or "combinations" can be represented as written below:
The optimal set/combination of hyper-parameters is included in the supplied list of values.

nCr = 7C2

C(n, r) = 7! / [2! (7 - 2)!]

How do you determine the probability for a given number of "samples", i.e count of hyper-parameter  combinations to test, to include the optimal hyper-parameter settings?

Is there any standard probability of success that I should stick to, for example 80% probability that the optimum set of hyper-parameters would be included in the tests?


texts:
I have spent multiple days trying to grasp the concept of the PCA and SVD. But I still have a couple of confusion about the difference between SVD and PCA.
I watched Steve Brunton Youtube SVD playlist to learn about PCA and SVD. In the video, he said that the formula for SVD is matrix $X = U \Sigma V^T$. And $U$ captures most variance for each column while $V^T$ captures most variance for each row (Please correct me if I am wrong). And the example he used for SVD each training example is a column vector.
Then when he talked about PCA, he said PCA is built on top of SVD, a couple of differences he also said that all the training examples for PCA is a row vector. Is that a true difference between PCA and SVD or the data representation is not important? Then another difference is the purpose of PCA, since for each matrix we can do a SVD to get the U matrix and V transpose matrix and to find out the vector or direction that gives out the biggest variance, then why do we take more time to do the PCA?


texts: is numpy isclose function returning bad answer?

texts:
Using numpy isclose in the following example:

import numpy as np
np.isclose(1533761040,1533748023.0, atol=1)

Returns True, which is False.


texts: Deep learning: Training from COCO/imagenet weights or from a class-specific trained weights when classify 2 similar objects

texts:
I have a question about the possible outcome of a trained model. Imagine that I would like to classify 2 different models of Ferrari and the dataset of these 2 models is small (for example, a few hundred images per model).

In the Keras blog, the issue was discussed but in the example, they are classifying dogs and cats, 2 class that are very general, distinct between them, and there are many cats and dogs already included in the original imagenet model (lets say the car/bus/truck are not included in the imagenet output classes).

Will it be better start training the model from Coco/Imagenet weights or start training using the weights of a model previously trained to classify cars only?


texts: How does the bounding box regressor work in Fast R-CNN?

texts:
In the fast R-CNN paper (https://arxiv.org/abs/1504.08083) by Ross Girshick, the bounding box parameters are continuous variables. These values are predicted using regression method. Unlike other neural network outputs, these values do not represent the probability of output classes. Rather, they are physical values representing position and size of a bounding box.

The exact method of how this regression learning happens is not clear to me. Linear regression and image classification by deep learning are well explained separately earlier. But how the linear regression algorithm works in the CNN settings is not explained so clearly.

Can you explain the basic concept for easy understanding?


texts: How to compare the performance of feature selection methods?

texts:
There are several feature selection / variable selection approaches (see for example Guyon & Elisseeff, 2003; Liu et al., 2010):

filter methods (e.g., correlation-based, entropy-based, random forest importance based),
wrapper methods (e.g., forward-search, hill-climbing search), and
embedded methods where the feature selection is part of the model learning.

Many published algorithms are also implemented in the machine learning tools like R, Python, etc.

What would be an appropriate method to compare different feature selection algorithms and to select the best method for a given problem / dataset?
A further question would be, whether there are any metrics known that measure the performance of feature selection algorithms?


texts: What type of machine learning are able to return feature importance?

texts:
I've been using random forest to calculate feature importance however I've asked myself why random forest? so my question became what other machine learning models can calculate feature importance? and is there pros and cons between each models?
I've did some research and here are some that I've found

Statistical correlation score -> calculating each X variable with Y. (statistical method)
coefficients in linear models -> Logistic, linear regression, regularizations(L1, L2, elastic net)
Tree based algorithms  -> ex: decision tree, RF, boosting algorithms
permutation importance score

I would like to know if there are any other methods.
Thanks in advance!


texts: Why do we share parameters between two different inputs in the embeddings layer?

texts:
I noticed in some deep learning networks that have two inputs to the network, they use one embeddings layer to share the parameters between these two different inputs.

As an example, in Keras:

input_target = Input((1,))
input_context = Input((1,))
embedding = Embedding(vocab_size, embed_size, input_length=1, name='embedding')
target = embedding(input_target)
context = embedding(input_context)

Why do they use this way?

To make everything clear, the other case is: for each input we have different embeddings layer before moving to the RNN or CNN layers.


texts: How to formulate reward of an rl agent with two objectives

texts:
I have started learning reinforcement learning and trying to apply it for my use case. I am developing an rl agent which can maintain temperature at a particular value, and minimize the energy consumption if equipment by taking different actions that are available for it to take.

I am trying to formulate a reward function for it.

energy and temp_act can be measured

energy_coeff = -10
temp_coeff = -10

temp_penalty = np.abs(temp_setpoint - temp_act)

reward = energy_coeff * energy + temp_coeff * temp_penalty

This is the reward function I am using, but intuitively , I feel it should be better. because absolute value of enenrgy and temp_penalty are on different scales. How do i take into count the scaling problem, while structuring a reward.


texts: Maths question on mean squared error being dervied to bias and variance

texts:
I am reading a book and have difficulty in understanding the math on bias-
variance tradeoff. Below is the section that I am having trouble with:

  Given a set of training samples $x_1, x_2, ..., x_n$ and their targets $y_1, y_2, ..., y_n$, we want to find a regression function, $\hat{y}(x)$, which estimates the true relation $y(x)$ as correctly as possible. We measure the error of estimation, how good (or bad) the regression model is by mean squared error ($MSE$):

I can derive mean squared error with partial derivative and the concept of slope. I also understand that $MSE$ is to minimize the total error. I also understand basics statics on expected value.

Yet, I have been stuck in finding the relevant math and statistical concepts behind this formula for a week.

The question is, what are the relevant math and statistical concepts behind this formula?

For example, how

$MSE = E[(y-\hat{y} )^2]$

becomes:

$= E[(y-E[\hat{y} ] + E[\hat{y}] - \hat{y} )^2]$

Thank you! I can see that the first component after adding and subtracting E[y^] is unchanged. Then the formula operated according to
$(a+b)^2 = a^2 + 2ab + b^2 $
where

$2ac = +E[2(y - E[\hat{y}])(E[\hat{y}] - \hat{y})]$

Why 2ac becomes

$2(y - E[\hat{y}])(E[\hat{y}] - E[\hat{y}]) $


texts: How to find which features have been selected by PCA algorithm?

texts:
I used PCA function in MATLAB to decrease features on my data set.

By this code I can reduce features from 12 to 8(as an example). It works good but my question is that how can I found with feature have been removed or which feature are selected on the result?

[~, pca_scores, ~, ~, var_explained] = pca(myDataSet, 'NumComponents', 8);


texts:
I wonder which of these two books is better to read for a beginner in RL and which are the pros and cons of them.

Also, if you know any book that in your opinion is better for a beginner in RL, feel free to suggest.


texts: What is the proper proportion for train and test set for classification system?

texts:
I have recently googled the best proportion for training and test set for classifying physiological data between normal and abnormal. Much of the source tells that the proportion should be 70:30 or 80:20 for train:test. One of the sources could be visited here.
I am a little unsure because I haven't found the ground scientific truth behind this proportion. Is that any science that explains this proportion?
Thanks


texts:
I want to build a model that can localize occurrences of a particular word in an audio file. For example, I want to find the word "pizza" in a ~5min recording. The program should return an array with (start, stop) objects describing the start and stop boundaries of that word in the file.

Can I just use classic object localization with some kind of CNN, where the object is the wanted word in the spectrogram? If I could: how would I need to  prepare the data for training—records with the word "pizza" and the same number of other words or more other words?

Is there perhaps a better method for word-searching in recordings?


texts: What do you use to generate a dashboard in R?

texts:
I need to generate periodic (daily, monthly) web analytics dashboard reports. They will be static and don't require interaction, so imagine a PDF file as the target output. The reports will mix tables and charts (mainly sparkline and bullet graphs created with ggplot2). Think Stephen Few/Perceptual Edge style dashboards, such as:

but applied to web analytics.

Any suggestions on what packages to use creating these dashboard reports?

My first intuition is to use R markdown and knitr, but perhaps you've found a better solution. I can't seem to find rich examples of dashboards generated from R.


texts: How can I predict student enrolment in September based on independent data available earlier in the year?

texts:
I'd like to predict how many students will enroll in a college in September, based on independent variables which are known earlier in the annual student recruitment cycle.

For example, one independent variable would be the number of accepted offers; I know how many students have accepted an offer from the college for each week from January to August for the last five years, and I know how many of these students actually enrolled in September (generally about 80% of students who accept an offer actually enroll, but this varies year-on-year).

Other independent variables might include: how many students have already paid a deposit on their tuition fee, or even the current exchange rate for students travelling from abroad.

So far, I've used historical conversion rates to predict enrollment, but I am wondering whether I can do something more sophisticated which would allow me to add in other independent variables.

Any ideas of which approaches I should investigate would be very welcome. I'd be particularly interested to hear of any Python libraries which might be relevant.


texts: What exactly does generator produce in DCGANs?

texts: Does it generate the set of the same image classes in the same order on each iteration? If yes, what's the usufullness of that ?


texts: Interpretation of Similarity Number generated by LogLikehood in Mahout

texts:
I have a pretty basic question and I was hoping someone could help me. I’m not a math person and I’m fairly new to mahout so I’m looking for a poor’s man explanation.

It is a typical order recommendation system.

I have a database with around 699,445 orders.
These orders have items which were “purchased”.

I ran the following mahout command:

  mahout itemsimilarity --input /mnt/p1.csv --output ./output --similarityClassname SIMILARITY_LOGLIKELIHOOD --booleanData TRUE --threshold 0.9

I decided to spot-check the results.

I took the following line from the output file:

58331   120216  0.9705375406679205

In my input file:

1540 orders  have product 58331
35 orders have product 120216
10 orders have both (58331 and 120216)

Putting in Ted Dunning term’s

k_11 = 10   b_12 = 25
k_21 = 1531 k_22= 697889

The similarity number generated by the loglikehood algorithm between 58331 and 120216 is 0.9705375406679205.

1)  What does that mean?

2)  Should I recommend 58331 when someone orders 120216 ? Should I recommend 120216 when someone orders 58331?

3) How do I calculate the entropy used in the LLR formula?

Thx a lot


texts: How do you evaluate a predictive model that is deployed in production that is suppose to prevent certain scenerio?

texts:
Let's consider I trained a model that gives the probability if an apple will rot and deployed it. Once it's deployed, how can I measure/evaluate its performance. At first I thought, we can just check against the actual outcome and see if the prediction was made correctly. However, this model is an preventative model so that person can take action to make sure apple doesn't rot. Since person took some action to prevent the apple from rotting, the accuracy or any other metric will do poorly.

In this case, what would be the best way to evaluate the model as time goes by? How do we check if model is still correctly predicting the outcome of apple?

Thanks,

Shone


texts: Retrieving information from 2 or more approaches

texts:
I'm trying to extract information from documents. There are two approaches currently which produces the following cases where both approaches
Case 1: gives same output, "ABC XYZ Corporation"
Case 2: gives almost similar output, "ABD XYZ Corporation", "ABC XYC Corporation"
Case 3: gives partial output, "ABC XYZ", "Corporation"

I tried matching the outputs to see if those are similar, and then combining the outputs only when they're different from each other. I'm looking for best way to combine the two approaches so that maximum information is extracted. Also, what if there's a third approach as well? Suggestions are welcome!


texts: How to change voice features in python without affecting speech/language features?

texts: I am trying to build a CNN model which should be able to identify the language being spoken in an  audio file. I have extracted the MFCC matrix (for 13 coefficients) for each audio file and trained it. The audio files are call recordings. The accuracy is too high over a small data sample. The problem I have now is that in many audio files, there are common voices. I have a suspicion that the model is giving high importance to these voice features instead of features which are important to identify language (i.e. since 1 person speaks in only one language, the model might think the voice features of those files are important for classification - this will make it fail while testing for new voices). How do I manipulate the data such that each audio files has a different voice? (example: changing frequency is one way)


texts: Foreign exchange market forecasting with neural networks

texts:
I would like to use ANN to automate trading currencies, preferably USD/EUR or USD/GBP. I know this is hard and may not be straightforward. I have already read some papers and done some experiments but without much luck. I would like to get advice from EXPERTS to make this work.

Here is what I did so far:

I got tick by tick data for the month of july 2013. It has bid/ask/bid volume/ask volume.
Extracted all ticks for the time frame 12PM to 14PM for all days.
From this data, created a data set where each entry consists of n bid values in sequence.
Used that data to train an ANN with n-1 inputs and the output is the forecasted nth bid value.
The ANN had n-1 inputs neurons, (n-1)*2 + 1 hidden and 1 output neuron. Input layer had linear TF, hidden had log TF and output had linear TF.
Trained the network with back propagation with n-125 first and then 10.

For both n, the MSE did not drop below 0.5 and stayed at that value during full training. Assuming that this could be due to the time series being totally random, I used the R package to find partial autocorrelation on the data set (pacf). This gave non zero values for 2 and 3 lags only.

Question 1: What does this mean exactly?

Then I used hurst exponent to evaluate the randomness. In R, hurst(values) showed values above 0.9.

Question 2: It is supposed to be nearly random. Should it have a value closer to 0.5?

I repeated the training of the ANN with n=3. The ANN was trained and was able to obtain a pretty low value for MSE. However, the calculated output from this ANN does not differ much from the (n-1)th bid value. It looks like ANN just takes the last bid as the next bid! I tried different network structures (all multilayer perceptions), different training parameters, etc, but results are same.

Question 3: How can I improve the accuracy? Are there any other training methods than backpropagation?


texts: No accuracy in Keras RNN Model with Bitcoin Data

texts:
I am very new to machine-learning and have made an RNN-LSTM model with no accuracy. My data has been normalized with MinMaxScaler from Sklearn and has a shape of has an input of shape (3, 2)...

My normalization steps:

def get_data(currency):
    url=f'https://coinmarketcap.com/currencies/{currency}/historical-data/?start=20130428&end={time.strftime("%Y%m%d")}'
    data=pd.read_html(url, flavor='html5lib')[0]

    data=data.assign(Date=pd.to_datetime(data['Date']))
    data['Volume']=(pd.to_numeric(data['Volume'], errors='coerce').fillna(0))

    data.columns=[col.lower() for col in data.columns]
    data.columns=[col.strip('*') for col in data.columns]

    return data

df=get_data('bitcoin')
df=df.sort_values(by='date')

def split_data (data, trainsize):
    return np.array(data[:int(trainsize*len(data))]), np.array(data[int(trainsize*len(data)):])

scaler=MinMaxScaler(feature_range=(0,1))

def create_inputs(data, window):
    inputs=[]
    for i in range(len(data)-window):
        inputs.append(data[i:(i + window)].values)

    close,volume=[],[]
    for x in range(len(inputs)):
        close.append(inputs[x][:,0])
        volume.append(inputs[x][:,1])

    close=np.array(close)
    close=scaler.fit_transform(close)

    volume=np.array(volume)
    volume=scaler.fit_transform(volume)

    inputs=[]
    for i in range(len(close)):
        rows=[]
        for x in range(len(close[i])):
            row=[close[i][x], volume[i][x]]
            rows.append(row)
        inputs.append([rows])
    inputs=np.vstack(inputs)
    return inputs

def create_outputs(data, window):
    return scaler.fit_transform(data['close'][window:].values.reshape(-1,1))

# VARIABLES
df=df.filter(['date', 'close', 'volume'], axis=1)
df=df.sort_values(by='date')
df[df.columns] = df[df.columns].apply(pd.to_numeric, errors='coerce')

train,test=split_data(df,0.8)

train=pd.DataFrame(train, columns=df.columns)
test=pd.DataFrame(test, columns=df.columns)
train=train.drop('date',1)
test=test.drop('date',1)

xtrain,ytrain=create_inputs(train, 3), create_outputs(train, 3)
xtest,ytest=create_inputs(test, 3), create_outputs(test, 3)

Here is part of my training data (1607, 3, 2) fetched from CoinMarketCap's Bitcoin History after scaling:

    [[[0.01363717 0.        ]
      [0.01577874 0.        ]
      [0.01463021 0.        ]]

     [[0.01577874 0.        ]
      [0.01463021 0.        ]
      [0.01006721 0.        ]]

     [[0.01463021 0.        ]
      [0.01006721 0.        ]
      [0.00762504 0.        ]]...]

My model has 3 Layers with 1024 LSTM Cells, and Dense layer with 1 neuron:

model = keras.models.Sequential()

model.add(keras.layers.CuDNNLSTM(1024, input_shape=(3,2), return_sequences=True, name='input'))
model.add(keras.layers.Dropout(0.2))

model.add(keras.layers.CuDNNLSTM(1024, return_sequences=True, name='lstm1'))
model.add(keras.layers.Dropout(0.2))

model.add(keras.layers.CuDNNLSTM(1024, name='lstm2'))
model.add(keras.layers.Dropout(0.2))

model.add(keras.layers.Dense(1, activation='tanh', name='output'))

# Compile model
model.compile(
    loss='mse',
    optimizer='adam',
    metrics=['accuracy'],
)

history=model.fit(xtrain, ytrain, batch_size=64, epochs=1000, validation_data=(xtest, ytest), verbose=1)


texts:
I have a dataset having 7 attributes viz., time, C1, ... C7 pertaining to earth quake reports where each column/attribute represents a certain aspect of damage viz., power, sewer_and_water, shake_intensity, etc. Each of these attributes have a rating from 0 to 10, where 0 represents no damage and 10 represents maximum damage.

Since, some of the ratings in each attribute might not be trustworthy, hence I am trying to compute a weighted score as follows-
(C1 * W1) + (C2 * W2) + ... + (C7 * W7); where Ci is ith attribute and Wi is the feature importance score for it.

In order to compute a weighted score of each row/data point, I am trying to train a classifier such as Random Forest, LightGBM or XGBoost which will give me a feature importance score for each attribute. However, since I don't know which attribute is supposed to be target, I used a brute force approach where I chose each attribute as target and trained a classifier to see which gave me the highest accuracy.

However, this approach has the problem that out of n attributes, I will only get feature importance score for (n - 1) attributes as the nth attribute is the target variable.

Can you suggest a way in which I can get the feature importance scores for each attribute in dataset.

Thanks!


texts: How/What to initialize the hidden states in RNN sequence-to-sequence models?

texts:
In an RNN sequence-to-sequence model, the encode input hidden states and the output's hidden states needs to be initialized before training.

What values should we initialize them with? How should we initialize them?

From the PyTorch tutorial, it simply initializes zeros to the hidden states.

Is initializing zero the usual way of initializing hidden states in RNN seq2seq networks?

How about glorot initialization?

For a single-layer vanilla RNN wouldn't the fan-in and fan-out be equals to $(1 + 1)$ which gives a variance of $1$ and the gaussian distribution with $mean=0$ gives us a uniform distribution of $0$s.

for-each input-hidden weight
  variance = 2.0 / (fan-in +fan-out)
  stddev = sqrt(variance)
  weight = gaussian(mean=0.0, stddev)
end-for

For single layer encoder-decoder architecture with attention, if we use glorot, we'll get a very very small variance when initializing the decoder hidden state since the fan-in would include the attention which is mapped to all possible vocabulary from the encoder output. So we result in a gaussian mean of ~= 0 too since stdev is really really small.

What other initialization methods are there, esp. for the use on RNN seq2seq models?


texts: Homemade deep learning library: numerical issue with relu activation

texts:
For the sake of learning the finer details of a deep learning neural network, I have coded my own library with everything (optimizer, layers, activations, cost function) homemade.

It seems to work fine when benchmarking in on the MNIST dataset, and using only sigmoid activation functions.

Unfortunately I seem to get issues when replacing these with relus.

This is what my learning curve looks like for 50 epochs on a training dataset of ~500 examples:

Everything is fine for the first ~8 epochs and then I get a complete collapse on the score of a dummy classifier (~0.1 accuracy). I checked the code of the relu and it seems fine. Here are my forward and backward passes:

def fprop(self, inputs):
    return np.maximum( inputs, 0.)

def bprop(self, inputs, outputs, grads_wrt_outputs):
    derivative = (outputs > 0).astype( float)
    return derivative * grads_wrt_outputs

The culprit seems to be in the numerical stability of the relu. I tried different learning rates and many parameter initializers for the same result. Tanh and sigmoid work properly. Is this a known issue? Is it a consequence of non-continuous derivative of the relu function?


texts: python - What is the format of the WAV file for a Text to Speech Neural Network?

texts:
I am creating a Text to Speech system for a phonetic language called "Kannada" and I plan to train it with a Neural Network. The input is a word/phrase while the output is the corresponding audio.

While implementing the Network, I was thinking the input should be the segmented characters of the word/phrase as the output pronunciation only depends on the characters that make up the word, unlike English where we have slient words and Part of Speech to consider. However, I do not know how I should train the output.

Since my Dataset is a collection of words/phrases and the corrusponding MP3 files, I thought of converting these files to WAV using pydub for all audio files.

from pydub import AudioSegment
sound = AudioSegment.from_mp3("audio/file1.mp3")
sound.export("wav/file1.wav", format="wav")

Next, I open the wav file and convert it to a normalized byte array with values between 0 and 1.

import numpy as np
import wave

f = wave.open('wav/kn3.wav', 'rb')
frames = f.readframes(-1)

#Array of integers of range [0,255]
data = np.fromstring(frames, dtype='uint8')

#Normalized bytes of wav
arr  = np.array(data)/255

How Should I train this?

From here, I am not sure how to train this with the input text. From this, I would need a variable number of input and output neurons in the First and Last layers as the number of characters (1st layer) and the bytes of the corresponding wave (Last layer) change for every input.

Since RNNs deal with such variable data, I thought it would come in handy here.

Correct me if I am wrong, but the output of Neural Networks are actually probability values between 0 and 1. However, we are not dealing with a classification problem. The audio can be anything, right? In my case, the "output" should be a vector of bytes corresponding to the WAV file. So there will be around 40,000 of these with values between 0 and 255 (without the normalization step) for every word. How do I train this speech data? Any suggestions are appreciated.

EDIT 1 : In response to arduinolover's Answer

From what I understand, Phonemes are the basic sounds of the language.  So, why do I need a neural network to map phoneme labels with speech? Can't I just say, "whenever you see this alphabet, pronounce it like this". After all, this language, Kannada, is phonetic: There are no silent words. All words are pronounced the same way they are spelled. How would a Neural Network help here then?

On input of a new text, I just need to break it down to the corresponding alphabets (which are also the phonemes) and retrieve it's file (converted from WAV to raw byte data).  Now, merge the bytes together and convert it to a wav file.

Is this this too simplistic? Am I missing something here? What would be the point of a Neural Network for this particular language (Kannada) ?


texts: Questions about a multivariate timeseries forecasting model - keras

texts:
I have trouble understanding the model I'm trying to create.
I have few questions so I'll explain my model first and what I'm trying to do:

I have created sequences of data (input and ouput of the model) 7 timesteps each so the input would be the values of the days of a certain week and the output is the values of the days the following week (**so input1 of my model has an imput shape of (7,1) same as the output**).
I also have prepared another input list that has some extra features like holiday flag and weather condition for the **following week** so the model's second input2 has the shape (7,7) each example.

This is the full model summary:

After the lstm layers and a fully connected NN, I tried to concatenate the two inputs together, basically, I want to concatenate input2 to the output of the layer dense_1.
So I'm concatenating (7,1) shape with the second input 'input_2' of shape (7,7)
My questions:
1- The outputs of my dense layers confuse me I thought they would be in the shape (None, number of units) but they seem to be (None, 7 , number of units) there is always the 2nd dimension "7" which i don't understand.

2- For the concatenation part, since i'm adding a 7x7 input to a 7x1 input i was expecting to have 49+7=56 units connected to the next dense layer but the number of parameters tells me it's not the case being just 14*120+120=1800

3- I was thinking of having 7 units in the last layer ( 1 for each day to predict ) but had to make it 1 so it would output (7,1) and for it to work, there is something i'm clearly missing.

This is the code for model definition:
input1 = tf.keras.layers.Input(shape=(7,1),name="input_1")
x = tf.keras.layers.Conv1D(120, kernel_size=5,strides=1,activation="relu",
                           padding="causal",input_shape=[7, 1])(input1)
x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(210, return_sequences=True),name="LSTM_1")(x)
x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(210, return_sequences=True,name="LSTM_2"))(x)
x = tf.keras.layers.Dense(60,activation="relu",name="dense_1_1")(x)
x = tf.keras.layers.Dense(30,activation="relu",name="dense_1_2")(x)
x = tf.keras.layers.Dense(7,name="dense_1_3")(x)

input2 = tf.keras.layers.Input(shape=(7,7),name="input_2")
concat = tf.keras.layers.concatenate([x, input2],name="concat_1")
x = tf.keras.layers.Dense(120,activation="selu",name="dense_2_1")(concat)
x = tf.keras.layers.Dense(90,activation="selu",name="dense_2_2")(x)
x = tf.keras.layers.Dense(60,activation="selu",name="dense_2_3")(x)
output = tf.keras.layers.Dense(1)(x)

model = tf.keras.Model([input1, input2], output)

These are the dimensions of the inputs (input1 & input2) and the output


texts: Neural Network for Multiple Float Output

texts:
I have a dataset containing 40 input columns and 12 output columns (float values).

I supposed it's a regression problem and I am wondering of how to choose the ideal architecture of the Neural Network i didn't find examples of neural Network regression with multiple output
What is the right way to do it?


texts: Recognizing emerging topic within ongoing topic

texts: I have been collecting a large amount of tweets from Twitter for a few weeks related to a series of specific keywords, and would like to address a specific problem. Say I collected all tweets mentioning the word "Monsters", and it averaged about 50k tweets a day, but this week, James Blunt released a new song "Monsters" which becomes a particular hit and that amount increases to maybe 80k. What would be a good approach to help distinguish between the usual usages of the word "Monsters" and the new tweets that are presumably related to James Blunt? I would imagine there are quite a few options, so any ideas would be much appreciated.


texts: What is wrong with my logistic regression implementation?

texts:
Recently, I implemented the LR algorithm in Python. The main part of the code is as following(I didn't use mini batch in my code. Instead, I use the whole batch to compute gradients every time):

class Logistic():
def __init__(self):
    self.w = None
    self.lr = 10.
    pass
def train(self, xs, ys):
    m, n = xs.shape
    ones = np.ones([m, 1])
    xs = np.hstack([xs, ones])
    ys = np.expand_dims(ys, -1)
    self.w = np.ones([n+1, 1], dtype=np.float64) * 1.0
    epochs = 100
    for epoch in xrange(epochs):
        y_ = self.sigmoid(-np.dot(xs, self.w))
        # loss = -1.0/m * np.sum(ys * np.log(y_) + (1 - ys) * np.log(1 - y_))
        tmp1 = np.sum(np.log(y_[np.where(ys==1)]))
        tmp2 = np.sum(np.log(1 - y_[np.where(ys==0)]))
        loss = - (tmp1 + tmp2) / m
        print("epoch: %d, loss: %f" % (epoch, loss))
        print("y_: %f, %f" % (np.min(y_), np.max(y_)))
        grad =  np.sum((y_ - ys) * xs, axis=0) / m
        self.w -= self.lr * np.expand_dims(grad, -1)
        print("grad: %f, %f" % (np.min(grad), np.max(grad)))
        print("w: %f, %f" % (np.min(self.w), np.max(self.w)))
        print ""

The dataset I used is MNIST. I marked all digits 0 as class 0, and all other digits as class 1. Then I get this binary classification problem. I test my algorithm with many different learning rate, from 1e-6 to 10, and it turns out all of them produces good results(about 98% accuracy on test set). As far as I know, if the learning rate is to big, LR will not converge. But here although I used very big learning rate, the algorithm still converge to about 98% accuracy. Is there an explanation for this?


texts: Input with variable length Classification problem

texts:
I have a dataset with patient information with discrete labels (labels are stages of a particular disease) which needs to be predicted (Basically a classification problem).

The dataset looks something like below:

Patient#|Visit#|Other medical features related to patient and visit|label (disease stage)

So, I am interested in using patient's past visit data inorder to predict the current disease stage. But, the problem is that all the patients don't have equal number of visits. So, I can't just append the past visit information to predict the future visit label like below:

concat(Patient #n 1st visit (X = all input features)|label of this visit| Patient #n 2nd visit (X = all input features)) and then try to predict the label for 2nd visit using previous visit information.

In the above problem, the number of visit =1, but I have a varying number of visit for each patient. How can I tackle this problem?


texts: In XGBoost, how is a leaf index corresponding to the particular leaf node in actual base learner trees?

texts:
I've trained a XGBoost model for regression, where the max depth is 2.
# Create the ensemble
ensemble_size = 200
ensemble = xgb.XGBRegressor(n_estimators=ensemble_size, n_jobs=4, max_depth=2, learning_rate=0.1,
                            objective='reg:squarederror')

ensemble.fit(train_x, train_y)

I've plotted the first tree in the ensemble:
# Plot single tree
plot_tree(ensemble, rankdir='LR')

Now I retrieve the leaf indices of the first training sample in the XGBoost ensemble model:
ensemble.apply(train_x[:1]) # leaf indices in all 200 base learner trees

array([[6, 6, 4, 6, 4, 6, 5, 5, 4, 5, 4, 3, 5, 4, 5, 3, 6, 3, 5, 5, 3,
3,
3, 5, 4, 4, 3, 4, 3, 6, 6, 6, 4, 6, 6, 3, 5, 3, 5, 4, 6, 4, 4, 6,
3, 3, 6, 3, 6, 3, 4, 3, 6, 6, 3, 6, 5, 3, 6, 6, 3, 4, 6, 5, 3, 3,
3, 6, 3, 4, 3, 6, 3, 6, 3, 3, 3, 4, 6, 3, 4, 4, 6, 3, 3, 6, 3, 6,
6, 3, 3, 4, 4, 4, 3, 3, 6, 6, 3, 3, 6, 3, 3, 3, 6, 6, 6, 4, 4, 3,
5, 3, 3, 3, 4, 5, 3, 3, 6, 3, 3, 6, 3, 4, 5, 3, 6, 3, 5, 3, 4, 4,
3, 3, 4, 6, 6, 6, 6, 3, 4, 4, 3, 5, 6, 6, 3, 5, 3, 3, 6, 6, 3, 3,
6, 3, 3, 4, 4, 3, 4, 3, 5, 3, 3, 3, 3, 3, 4, 4, 6, 3, 6, 4, 4, 5,
6, 3, 4, 5, 6, 3, 4, 3, 4, 5, 6, 6, 5, 4, 3, 3, 6, 6, 3, 6, 5, 4,
3, 3]], dtype=int32)

Here is my question:

Since there are four leaf nodes in the first tree, how come there is
index 6 for the first training sample?

In the official doc for apply(), it says "Leaves are numbered within [0; 2**(self.max_depth+1)), possibly with gaps in the numbering." So if max_depth is 2, the leaves are numbered between 0 and 7. Since there are only four leaves in a binary tree of depth 2, shouldn't the leaves numbered within [0, 4)? What is the reason behind the design $[0; 2^{(self.max\_depth+1)})$?

Related question: https://stackoverflow.com/questions/58585537/how-to-interpret-the-leaf-index-in-xgboost-tree


texts:
Hello, I am very new to data science, machine learning, and stack overflow. Excuse me for being unclear or asking naive questions.
 My question is as follows:
From any given document, I am trying to classify it according to the emotions it evokes in readers, using a neural network. However, I am having difficulties with feature selection. I'm thinking of using NLTK and RAKE to extract keywords, but I don't know how I can translate them into features. Should I hash the keywords for one feature? Or, should I find a dictionary of english words (i.e. Wordnet), and use every word in the dictionary as a feature.


texts: Identifying importance of each feature in deep model

texts:
I have a deep model and I want to figure out which feature has the maximum influence on predicted result. For this I train the model with all the features I think are important, during prediction I set all features =0 one by one keeping the rest unchanged so I could figure out which is the least important of all. On predicting the results of these tampered test set on the trained model I get the same(changes at 5th-6th decimal place) F1 score, Recall and Precision.

Can some one explain me where I am wrong?


texts: How match output (pred value) to input value

texts:
I'm working with data(with 4 columns which are p(product), M(name of the store)), I want predict the demand of store for that I sued SVR on the data by theses formulation:
dfn = pd.get_dummies(df)
x = dfn.drop(["demand"],axis=1)
y = dfn.demand
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(0,1))
dfn = scaler.fit_transform(dfn)
.
.
.
from sklearn.metrics import r2_score
pred = regressor.predict(testX)
SVM_R2 = print('r2= ' +str(r2_score(testY,pred)))
print(pred)
# array example is between 0 and 1
array = np.array(pred)
#scaled from 200 to 800
minimo = 200
maximo = 800
output=array * minimo + (maximo - minimo)
print(output)
df2=pd.DataFrame(output)
df2.to_excel(r'/content/Book1.xlsx', index = False)

and now I get the output of this prediction. My question is, how can I match these outputs to inputs, or how can I found which demands are related to each market?


texts: PyTorch CNN network outputs homogeneous results

texts:
I am a beginner at data science and I got a project where I want to do nlp via a convolutional neural network in PyTorch. The problem is that regardless of what comes out of the convolutional layers, the output from the perceptron is always repeated. It is as if it wasn't seeing anything.

I already tried to change the dataset, number of variables, loss function, optimization method and add or remove layers and nothing seems to happen. It's been a whole week that I'm working on this and I can't figure out what's wrong.

The loss function in this case is torch.nn.MultiLabelSoftMarginLoss, which is why I removed the Sigmoid activation on the very last layer. The optimization method is SGD. It converges within 15 runs or so.

Here's a picture of what the data looks like after it goes through the convolutional layer (I put the tensor layers side-by-side):

The height is the number of channels and the width is the number of features times the batch size.

Data for testing

Here are the y,X pairs I'm using for the training. I used torch.save to create the files.

y:  https://file.io/JiLnim

x:  https://file.io/NKecI1

Reproducible Example

import torch
from torch import nn

T = torch.load("X_ex")
y = torch.load("y_ex")

class model(nn.Module):
    def __init__(self,Test,n_cat=1):
        super(model, self).__init__()
        self.last_kernel = 3
        self.convolutions = nn.Sequential(
            nn.Conv1d(Test.shape[1], 500, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(500, 400, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(400, 300, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(300, 200, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(200, 100, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=self.last_kernel),
            )
        self.test_conv = self.convolutions(Test) #I know it's dumb but I just wanted to get it working
        self.p_channels = self.test_conv.shape[1]*self.test_conv.shape[2]
        self.perceptron = nn.Sequential(
            nn.Linear(self.p_channels,100),
            nn.Sigmoid(),
            nn.Linear(100,16),
            nn.Sigmoid(),
            nn.Linear(16,n_cat),
            #nn.Sigmoid()
        )
    def forward(self, x):
        x = self.convolutions(x)
        x = x.view(x.size(0), -1)
        x = self.perceptron(x)
        return x

m=model(T,y.shape[1])
loss_fn = torch.nn.MultiLabelSoftMarginLoss(reduction="sum")

r = 0.001 #learning rate

optimizer = torch.optim.SGD(m.parameters(), lr=r, momentum=0.9)

for t in range(15):
    y_pred = m.forward(T)
    loss = loss_fn(y_pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

#Final y:
y_final = torch.sigmoid(y_pred)

Output (first few)

In [1]: y_actual
Out[1]:
tensor([[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        ...,
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698]],
       grad_fn=<SigmoidBackward>)
In [2]: y_actual.max(axis=1).indices
Out[2]:
tensor([24, 24, 24, 24, <keeps going>, 24, 24])


texts: Can someone explain why we can not feed LDA topic model with TFIDF?  What is wrong with this approach conceptually?


texts: Autoencoder works but VAE does not - KL divergence loss drops to zero

texts:
I have an image encoding problem, to which I have successfully applied an autoencoder. But when I try to convert this into a VAE framework by introducing KL divergence loss and sampling from a unit Gaussian, it fails catastrophically.
In the early phases of training, the KLD loss drops really quickly, and within a couple of epochs, the encoder essentially collapses, always returning exactly a unit Gaussian. If I 'turn off' the KL-divergence loss, network works well again (as it's equivalent to the normal autoencoder again). How do I diagnose the cause of this issue and fix it?


texts: How to deal with new features values in my classification model?

texts:
Lets say i have a categorical feature having a set of values equal to ['Single','Married','Divorced','Unknown']. Okay, so with the help of the other features, i create my model, i test it, all is fine and i deploy it.

Now , some user using my model as a webservice, types in 'Widow' for that feature value.

What's the proper way of dealing with it? Is there a way to ignore it? and predict based on the rest of the features only? or do i have to handle unknown feature values by assigning Unknown to them too? or worst case scenario, i handle the exception, return a message saying : hey man (or woman) use one of the available values on that feature! ??

Note : I'm trying to avoid assigning Unknown to it, because 'Unknown' has its weight. ( My real question is : if there is a way to set a feature's weight to 0 when doing inference )


texts: What is the State of the art method for full body gesture recognition in images

texts: I am working on gesture recognition in images and the best way that I am aware of, is whether using end to end approaches with deep neural networks or extracting body joint positions in an image and then doing some pre-processing on these joints positions and finally, feeding them to a classifier. I wanted to now what is the state of the art method for full body gesture recognition in images and if there are some good papers or a blog post reviewing different methods, as I have problem finding one.


texts:
I have just started getting my hands wet in ML and every time I try delving deeper into the concepts/code, I face the challenges of the mathematics and its cryptic notations.
Coming from a Computer Science background, I do understand bit of them but majority goes tangent.
Say, for example below formulae from this page -

I try and really want to understand them but somehow get confused and leave it everytime.
Can you please suggest how to start with it? Any starting pointers or advise please.


texts: 99% validation accuracy but 0% prediction results (UNET Architecture)

texts:
I am debugging results from the UNET architecture that I am using for identifying corneal reflection in eye images. While I am getting over 99% training accuracy and also very high (over 99%) validation accuracy, when I run the validation images myself, I am getting nothing but blank images from model prediction. When I used the same architecture with exact same parameters for training with a mask set for pupil, I get again high accuracy numbers but running the validation set again in the prediction gave great results. Here is the sample output for the data set where I am having trouble, mismatch of validation and prediction results:

Train on 326 samples, validate on 140 samples
Epoch 1/1
277s - loss: 0.1961 - dice_coef: 0.0012 - acc: 0.9834 - val_loss: 0.0338 - val_dice_coef: 4.8418e-11 - val_acc: 0.9979
326/326 [==============================] - 79s

Here is my code:

#-------------------------------------------------------
# Define UNET model
#-------------------------------------------------------
print("Compiling UNET Model.....")

def dice_coef(y_true, y_pred):
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = K.sum(y_true_f * y_pred_f)
    coef = (2. * intersection + K.epsilon()) / (K.sum(y_true_f) + K.sum(y_pred_f) + K.epsilon())
    return coef

x_data = x_data[:,:,:,np.newaxis]
y_data = y_data[:,:,:,np.newaxis]
x_train, x_val, y_train, y_val = train_test_split(x_data, y_data, test_size = 0.3)

input_layer = Input(shape=x_train.shape[1:])
c1 = Conv2D(filters=8, kernel_size=(3,3), activation='relu', padding='same')(input_layer)

l = MaxPool2D(strides=(2,2))(c1)
c2 = Conv2D(filters=16, kernel_size=(3,3), activation='relu', padding='same')(l)

l = MaxPool2D(strides=(2,2))(c2)
c3 = Conv2D(filters=32, kernel_size=(3,3), activation='relu', padding='same')(l)

l = MaxPool2D(strides=(2,2))(c3)
c4 = Conv2D(filters=32, kernel_size=(1,1), activation='relu', padding='same')(l)

l = concatenate([UpSampling2D(size=(2,2))(c4), c3], axis=-1)
l = Conv2D(filters=32, kernel_size=(2,2), activation='relu', padding='same')(l)

l = concatenate([UpSampling2D(size=(2,2))(l), c2], axis=-1)
l = Conv2D(filters=24, kernel_size=(2,2), activation='relu', padding='same')(l)

l = concatenate([UpSampling2D(size=(2,2))(l), c1], axis=-1)
l = Conv2D(filters=16, kernel_size=(2,2), activation='relu', padding='same')(l)

l = Conv2D(filters=64, kernel_size=(1,1), activation='relu')(l)
l = Dropout(0.5)(l)

output_layer = Conv2D(filters=1, kernel_size=(1,1), activation='sigmoid')(l)

model = Model(input_layer, output_layer)

model.compile(optimizer=Adam(1e-4), loss='binary_crossentropy', metrics=[dice_coef, 'acc'])

#-------------------------------------------------------
# Train UNET MOdel
#-------------------------------------------------------
if train_opt:
    print("Training UNET Model.....")
    weight_saver = ModelCheckpoint(weights_file, monitor='val_dice_coef', save_best_only=True, save_weights_only=True)
    annealer = LearningRateScheduler(lambda x: 1e-3 * 0.8 ** x)
    hist = model.fit(x_train, y_train, batch_size = 8, validation_data = (x_val, y_val), epochs=1, verbose=2, callbacks = [weight_saver, annealer])
    model.evaluate(x_train, y_train)

Thanks a lot for your help!


texts: Sklearn PCA with zero components example

texts:
I'm simply trying to repeat a benchmark from the sklearn's docs.

The unclear part is: n_components = np.arange(0, n_features, 5).
They are applying a PCA transform with 0 components!

Can somebody, please, explain, what's the mathematical meaning of this transform?


texts: How to devise multicategory classifiers employing linear discriminant functions?

texts:
We might reduce the problem to $c$ two-class problems, where the $i^{th}$ problem is solved by a linear discriminant function that separates points assigned to $w_i$ from those not assigned to $w_1$.
A more extravagant approach would be to use $\frac{c(c-1)}{2}$ linear discriminants, one for every pair of classes.

For these two methods, I don't understand how they divide the area. If we remove all the lines in Figure 9.3, how should we start?


texts: ValueError: Error when checking input: expected input_5 to have 4 dimensions, but got array with shape ()

texts:
I am trying to load a densenet-121 model pretrained on imagenet weights and train on my dataset. I have two files namely train.csv and validation.csv and I have splitted the train.csv file into 80-20 train set and validation set and using validation.csv file for testing. However, when I am trying my model to fit the dataset, I am getting the above error. Here is my code snippet:

import os
import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras import Model
from tensorflow.keras.applications.densenet import DenseNet121
import collections
from sklearn.model_selection import train_test_split
import pandas as pd
import tensorflow_datasets as tfds
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.data import TFRecordDataset
model = DenseNet121(include_top=False, weights='imagenet',  input_shape=(input_shp_img),
        classes=class_names)
_DATA_DIR = "C:/Users/1024/Documents/my_file"
_TRAIN_DIR = os.path.join(_DATA_DIR, "train")
TEST_DIR = os.path.join(_DATA_DIR, "valid")
_TRAIN_LABELS_FNAME = os.path.join(_DATA_DIR, "train.csv")
TEST_LABELS_FNAME = os.path.join(_DATA_DIR, "valid.csv")
_LABELS = collections.OrderedDict({
"-1.0": "uncertain",
"1.0": "positive",
"0.0": "negative",
"": "unmentioned",
})
len(data_csv)
train_data_size= int(0.8 * len(data_csv))
val_data_size = int(0.2 * len(data_csv))
full_dataset = tf.data.TFRecordDataset(_TRAIN_LABELS_FNAME)
full_dataset = full_dataset.shuffle(buffer_size=800)
train_dataset = full_dataset.take(train_data_size)
val_dataset = full_dataset.skip(train_data_size)
val_dataset = val_dataset.skip(val_data_size)
print(train_dataset)
model.compile(optimizer='sgd',
      loss='mean_squared_error',
      metrics=['accuracy'])
history = model.fit(train_dataset, batch_size=1000, validation_data=val_dataset,
          steps_per_epoch=100, epochs=100, use_multiprocessing=True)

Below is the error I am getting:

ValueError   Traceback (most recent call last)
    <ipython-input-117-bf00f479a1f4> in <module>
      2           loss='mean_squared_error',
      3           metrics=['accuracy'])
----> 4 history = model.fit(train_dataset, batch_size=1000, validation_data=val_dataset,
    steps_per_epoch=100, epochs=100, use_multiprocessing=True)
      5

~\.conda\envs\manpreet_env\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self,
 x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle,
 class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size,
 workers, use_multiprocessing, **kwargs)
774         steps=steps_per_epoch,
775         validation_split=validation_split,
--> 776         shuffle=shuffle)
777
778     # Prepare validation data.

~\.conda\envs\manpreet_env\lib\site-packages\tensorflow\python\keras\engine\training.py in
_standardize_user_data(self, x, y, sample_weight, class_weight, batch_size, check_steps, steps_name,
steps, validation_split, shuffle)
2380         feed_input_shapes,
2381         check_batch_axis=False,  # Don't enforce the batch size.
-> 2382         exception_prefix='input')
2383
2384     if y is not None:

~\.conda\envs\manpreet_env\lib\site-packages\tensorflow\python\keras\engine\training_utils.py in
standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
351                            ': expected ' + names[i] + ' to have ' +
352                            str(len(shape)) + ' dimensions, but got array '
--> 353                            'with shape ' + str(data_shape))
354         if not check_batch_axis:
355           data_shape = data_shape[1:]

ValueError: Error when checking input: expected input_5 to have 4 dimensions, but got ar


texts: Trying to change pandas column dtype from str to float

texts:
I am trying to convert a pandas column from a str to a float. Before I convert the strings to floats using astype(float), I need to remove characters from the string which cannot be converted into floats, like a comma or a space. In trying to remove these characters, I am using list compression and the str.replace method like this:

data.loc[:,'column_name'] = [char.replace(',', '') for char in
                                data.loc[:,'column_name']]

but I am getting an error saying:

AttributeError: 'float' object has no attribute 'replace'.

The confusing part, though, is that when I look at the dtypes, the dtype for that column shows up as 'o', or object. So I have no idea what is going on.


texts: Ideas for using polynomial regression for multivariate time series prediction

texts:
I would like to predict next 12 months of employee count based on 3 or more years of data that includes salaries, profit and employee count.

I have also interpolated monthly values for these three variables for known years to better suit to the problem and generate more training examples.

Based on analysis I would like to apply polynomial prediction model for predicting employee count at step t from t-1, then from t to t+1 and up to t+11 which corresponds to December. I do not have available salaries and profit data at prediction times so I cannot use them as X vector to predict employee count y.

I tried using Arima and it works fine, but it works with univariate series.
I tried also to reframe these three time series in pattern sequences with window or lag 1 and prediction steps 12 which gives (13,39) matrix.

Do you have some ideas how this problem can be approached, how it can be trained and how prediction can be triggered? Also what could be other options for reframing the time series?


texts: How to set a newtwork with two objectives?

texts:
Suppose I have a x_train, y1_train and y2_train.

I want to construct a network (such as simple MLP) to fit y1_train and to be low correlated with y2_train (or to fit -y2_train) simultaneously.

How could I achieve this goal? Is the custom loss function a good solution?

I use keras as my tool.


texts: Naive Bayes Classifier - Discriminant Function

texts:
To classify my samples, I decided to use Naive Bayes classifier, but I coded it, not used built-in library functions.

If I use this equality, I obtain nice classification accuracy: p1(x) > p2(x) => x belongs to C1

However, I could not understand why discriminant functions produce negative values. If they are probability functions, I think they must generate a value between 0 and 1.

Is there anyone who can explain the reason ?


texts:
I want to extract name from CV.
I need high level if accuracy more than 95 %. I have started with taking assumptions that it is highly likely to be found in 10% top lines or if not there then in some section similar to Personal details.
Can u plz  provide any help/guidance ?


texts: Timestamps in Ridge Regression Scikit Learn

texts:
I am trying to transform data for use in regression, most likely the Ridge or Lasso technique implemented in sklearn.linear_model.

My training data contains time stamps , which I believe may have predictive power. The time stamps reflect the time that a user placed an order for pizza. Here is an example:

Edit: Including labels in field elapsed_time, which is in seconds.

import pandas as pd
import sklearn.linear_model as linear_model

delivery_data = {
    'order_time' : ['2018-09-12 21:43:08', '2018-09-13 06:33:04', '2018-09-13 09:12:18'],
    'price' : [34.54, 8.63, 21.24],
    'miles' : [6, 3, 7],
    'home_type' : ['apartment', 'house', 'apartment'],
    'elapsed_time' : [2023, 1610, 1918]
}

df = pd.DataFrame(delivery_data)
df['order_time'] = pd.to_datetime(df['order_time'])

The resulting DataFrame looks like this:

           order_time  price  miles  home_type  elapsed_time
0 2018-09-12 21:43:08  34.54      6  apartment          2023
1 2018-09-13 06:33:04   8.63      3      house          1610
2 2018-09-13 09:12:18  21.24      7  apartment          1918

I am trying to predict the time to deliver pizza (elapsed_time) given timestamp, quantitative, and categorical data.

I suspect that time of day is predictive but that date is less predictive.

So far, I am considering extracting only the hour from the time stamp. In this example, order_time would become [21, 6, 9]. My first concern is that 23:59 has an hour of 23 and 00:01 has an hour of 0. The two values are far apart, even though the order times are two minutes apart.

Is there a better way to transform this datetime data?

Does it make a difference that the dataset contains other quantitative data (price, miles_from_store) and categorical data (home_type)?


texts: What are alternatives to MLP, when you have rectangular, structured data?

texts:
I have a rectangular numeric dataset, and I'm applying a multilayer perceptron to it. I'm having success, but I'm now looking to see what other architectures I can apply.

Much of deep learning seems applied to loosely-structured data -- sequences, text, images -- and everyone is having a lot of fun working with a variety of interesting models...at least when they have a problem that fits these models.

What about basic, row/column datasets. What are some of the canonical models to be used with this kind of data, apart from tweaking the layers of a basic MLP?


texts: Trouble with grouping by and counting in r

texts:
I have a data.frame called world. This data frame lists countries in column name.y, I have a column for languages named languages.  I need to List the names of countries where more than 4 languages are spoken.

My first take on this is to aggregate country and languages.

language <- data.frame(table(World$name.y,World$language))

when I run head(language) it doesn't work out.


texts: How to extract paragraphs from text document?

texts:
I have extracted text data from pdf files of annual reports of companies using pdftotext. The extracted file content looks like:
Sample pdf file is here

FORWARD-LOOKING STATEMENTS

In this Annual Report, we have disclosed forward-looking information to enable investors to

comprehend our prospects and take investment decisions. This report and other statements written and oral – that we periodically share contain

forward-looking statements that set out ...//Content Continued like in this format until the paragraph ends//

HERO
O
FOREVER
Now if check out the actual report on above link the text under Notes to Account and Independent Auditor's Report follows the almost similar structure in all companies reports in India. Only the Chairman's message or Boards reports vary but usually talks about growth, performance, future, investments etc.
So is there any way to extract only the paragraphs/multiple paragraphs combines into single(if continuation of same information) which contains useful information. I have searched but i find most of work on paragraph/document summarization but donot find something like extraction of actual continuous blocks of text data from documents.
Note : There is lot of noisy data(actual data in pdf but they are like page number, if company added some text in header/footer apart from title of page etc) between paragraphs which is generated while converting from pdf to text like:

Multi-line paragraph
R

N

O

I

WE W DS REMA INDS

BRANVANT IN M

RELE MARKETSDES

AND SS DECA

ACRO

6

Next Multiline paragraph


texts: Do I need to encode the target variable for sklearn logistic regression

texts:
I'm trying to get familiar with the sklearn library, and now I'm trying to implement logistic regression for a dataframe containing numerical and categorical values to predict a binary target variable.
While reading some documentation I found the logistic regression should be used to predict binary variables presented by 0 and 1.
My target variable is "YES" and "NO", should I code it to 0 and 1 for the algorithm to work properly, or there is no difference?
Maybe I just didn't get the idea but can someone confirm this to me.


texts: Restrictions on my skewed validation data

texts:
I have a severely skewed data sets consisting of 20 something classes where the smallest class contains on the order of 1000 samples and the largest several millions.

Regarding the validation data, I understand that I should make sure that it represent a similar ratio between classes compared to the one in my original raw data. Hence, I shouldn't do any under- or over-sampling on that validation data, but can do it on the training data.

Because I have such greatly skewed data set, is it still viable to add some restriction to the selection of my validation data set? Say I want there to be at least 1000 samples from each class in order to accept it, as I want to have a reasonable accuracy on the metrics of all classes.

Would this ruin my validation as the ratio between the largest and smallest class could then go from ~0.01-0.1% to ~1.0%, or is it still safe as the validation data still is significantly skewed?


texts: Different runs of feature selection algorithm giving different set of selected feature. How to choose the best set among them?

texts:
I am using the forward feature selection algorithm from MATLAB. The code is as follows:

 X=combine_6_non;
 y=target;
 c = cvpartition(y,'k',10);
 opts = statset('display','iter');
 [fs,history] = sequentialfs(@fun,X,y,'cv',c,'options',opts)

The function fun is as follows:

function err = fun(XT,yT,Xt,yt)
  model = svmtrain(XT,yT, 'Kernel_Function', 'rbf', 'boxconstraint', 1);
  err = sum(svmclassify(model, Xt) ~= yt);
end

Now for different runs of the selection algorithm I am getting different feature sets. How should I zero down to the best feature set?


texts: GAN - why doesn't the generator nullify the noise input?

texts:
In GAN architecture, during training, what keeps the generator's output dependant on the input noise? Why don't the weights of the noise input become zero (plus a bias)?
I would expect the generator to converge to outputting a single picture which is extremely real and non-distinguishable from a real picture, and ignore the noise all together, since this is a "cheaper" way (in convergence time and in number of parameters used) to decrease the generator's loss.


texts: How to pass a sequence of 4 images into LSTM and CNN-LSTM

texts:
I got an assignment and stuck with it while going down the rabbithole of learning PyTorch, LSTM and cnn.
Provided the well known MNIST library I take combinations of 4 numbers and per combination it falls down into one of 7 labels.
eg:
1111 label 1 (follow a constant trend)
1234 label 2 increasing trend
4321 label 3 decreasing trend
...
7382 label 7 decreasing trend - increasing trend - decreasing trend
The shape of my tensor after loading of the tensor become (3,4,28,28) where the 28 comes from the MNIST image's width and height. 3 is the batch size and 4 is the channels (4 images).
I'm somewhat stuck with how to pass this into a PyTorch backed LSTM and CNN as basically all Google searches lead to articles where simply one image is passed in.
I was thinking of reshaping it to 1 long array of (pixel values) where I put all of the values of the first image row by row (28) after each other, then appended by the same approach for the second, third and fourth image. So that would make 4 * 28 * 28 = 3136.
Is my way of thinking on how to tackle this a correct one or should I rethink? I'm rather new to this all and looking for some guidance on how to go forward. I've been reading loads of articles, YT videos, ... but all seem to touch the basic stuff or alternatives of the same subject.


texts:
After reading a while, I am confused now about my LSTM data structure. Assuming that I have a supervised learning problem with 1000 samples and 40 features as input. Now I want to create 10 timesteps of x. My resulting dimension of Keras data structure is

x: (1000, 10, 40)

with every two-dimensional matrix (1000, 40) shifted row-wise and ten times.

The question is now:

Which dimension has my target y to be?

My dimension is y=[1000,1] for one resulting target, but I also read y=[1000,10,1]. I suppose, the second dimension with value 10 should be the row-wise shifted target-vector. But, do I really need this? Shouldn't my solution be the right one?


texts:
I have a list of credits, let's assume they are only two for ease:
Credit #1 and credit #2: One has 3 months of payments missing and the other has 6 months left of payments.
It would like to obtain a table which can produce a dataset with the due date of the following payments:
Original dataset:

Desired dataset:


texts:
I found packages being used to calculating "Information Gain" for selecting main attributes in C4.5 Decision Tree and I tried using them to calculating "Information Gain".

But the results of calculation of each packages are different like the code below.

> IG.CORElearn <- attrEval(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi, dataUSE1, estimator = "InfGain")
> IG.RWeka     <- InfoGainAttributeEval(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi, dataUSE1)
> IG.FSelector <- information.gain(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi,dataUSE1)

> IG.CORElearn
   In_Temp    In_Humi     In_CO2    In_Illu      In_LP   Out_Temp   Out_Humi
0.04472928 0.02705100 0.09305418 0.35064927 0.44299167 0.01832216 0.05551973
> IG.RWeka
   In_Temp    In_Humi     In_CO2    In_Illu      In_LP   Out_Temp   Out_Humi
0.11964771 0.04340197 0.12266724 0.38963327 0.44299167 0.03831816 0.07705798
> IG.FSelector
         attr_importance
In_Temp       0.08293347
In_Humi       0.02919697
In_CO2        0.08411316
In_Illu       0.27007321
In_LP         0.30705843
Out_Temp      0.02656012
Out_Humi      0.05341252

Why do the results of calculation of each packages be different?
And Which one is right?


texts: How to find relation between N components and predict the value of any one component using the predicted relation?

texts:
I am new to Machine learning and trying to learn by practicing.

I have a situation where I am reading a set of N data. Each of N data will have independent state at any moment of time. I want to use these N data to find a relation between them. In case if I am not receiving one or more sets of data, I should be able to use available relation between N data to predict the missing data.

What type of machine learning algorithm can be used?


texts: Is the distance in Nearest Neighbors a good measure of similarity?

texts:
Let's train a Nearest Neighbor model with just one sample in it:

In [48]: nn = NearestNeighbors().fit([[0, 1, 0, 0]])

So this one sample has just one significant feature. Querying the model with the same sample returns 0 distance in the first array as expected:

In [50]: nn.kneighbors([[0, 1, 0, 0]], 1)
Out[50]: (array([[0.]]), array([[0]]))

But queries with samples of [0,2,0,0] and [0,1,1,0] both return the same distance value of 1:

In [51]: nn.kneighbors([[0, 2, 0, 0]], 1)
Out[51]: (array([[1.]]), array([[0]]))

In [52]: nn.kneighbors([[0, 1, 1, 0]], 1)
Out[52]: (array([[1.]]), array([[0]]))

This is counter-intuitive, because one would expect [0,2,0,0] to be more similar to [0,1,0,0] than [0,1,1,0]. Using Jaccard metric slightly improves on this issue:

In [56]: nn = NearestNeighbors(metric=scipy.spatial.distance.jaccard).fit([[0, 1, 0, 0]])

In [57]: nn.kneighbors([[0, 1, 0, 0]], 1)
Out[57]: (array([[0.]]), array([[0]]))

In [58]: nn.kneighbors([[0, 2, 0, 0]], 1)
Out[58]: (array([[1.]]), array([[0]]))

In [59]: nn.kneighbors([[0, 1, 1, 0]], 1)
Out[59]: (array([[0.5]]), array([[0]]))

But for my dataset Jaccard metric makes the kNN queries taking very long time, perhaps it is more suited for binary features. I have a set of readings from 52 sensors per row, nicely normalized with zero mean. I stumbled upon this issue when I fitted this set into sklearn.neighbors.NearestNeighbors and queried giving the first row of the training set as a sample and K=2, so it returned 0th index at 0 distance as expected and some other index with 0.02 distance. When I checked that other one I could not see ANY similarities, in fact most of the features where very dissimilar by value and/or by sign. I could get the same distance with a made-up sample of the first row from the training set where any one feature was increased by 0.02.

I am wondering now how to overcome this problem and if there is an easy way (ie. by tweaking parameters of NearestNeighbors) or a hacky way (ie. custom metrics, feature weights etc.) or should I rather use a different model? KMeans for example can converge clusters from my dataset pretty fast, but it uses NN internally and I do not fully like it because of the randomness in the init.


texts: Why real-world output of my classifier has similar label ratio to training data?

texts:
I trained a neural network on balanced dataset, and it has good accuracy ~85%. But in real world positives appear in about 10% of the cases or less. When I test network on set with real world distribution it seems to assign more positive labels than needed tending to balanced proportion as in training set.
What can be the reason of such behavior and what should I do solve it?
I'm using Keras and combination of LSTM and CNN layers.


texts: Machine Learning: creating a prediction without previous trends

texts: Is there a machine learning algorithm that allows one to give an input a classification without a grouping or previous classification? Example: with a dataset of 5 digit numbers, the user inputs a random 5 digit number (12345), can the machine output "alteration" when 12345 is similar to something in the dataset, or output "completely new" when 12345 or anything similar doesn't exist? To clarify, the machine would do something like "alteration" when it finds something similar to 12345 such as 12445 where its only number difference, and output "completely new" when no similar numbers exist.


texts: Simple Q-Table Learning: Understanding Example Code

texts:
I'm trying to follow a tutorial for Q-Table learning from this source, and am having difficulty understanding a small piece of the code. Here's the entire block:

import gym
import numpy as np

env = gym.make('FrozenLake-v0')

#Initialize table with all zeros
Q = np.zeros([env.observation_space.n,env.action_space.n])
# Set learning parameters
lr = .8
y = .95
num_episodes = 2000
#create lists to contain total rewards and steps per episode
#jList = []
rList = []
for i in range(num_episodes):
    #Reset environment and get first new observation
    s = env.reset()
    rAll = 0
    d = False
    j = 0
    #The Q-Table learning algorithm
    while j < 99:
        j+=1
        #Choose an action by greedily (with noise) picking from Q table
        a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))
        #Get new state and reward from environment
        s1,r,d,_ = env.step(a)
        #Update Q-Table with new knowledge
        Q[s,a] = Q[s,a] + lr*(r + y*np.max(Q[s1,:]) - Q[s,a])
        rAll += r
        s = s1
        if d == True:
            break
    #jList.append(j)
    rList.append(rAll)

print "Score over time: " +  str(sum(rList)/num_episodes)

print "Final Q-Table Values"
print Q

The code runs well and I'm able to print my results, but here is where I'm having difficulties:

a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))

My question is, why are we multiplying by 1/(i+1)? Is this supposed to be an implementation of epsilon annealing? Any help is appreciated.


texts: F1_score(average='micro') is equal to calculating accuracy for multiclasification

texts:
Is f1_score(average='micro') always the same as calculating the accuracy. Or it is just in this case?
I have tried with different values and they gave the same answer but I don't have the analytical demonstration.
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 0, 1]
print(f1_score(y_true, y_pred, average='micro'))
print(accuracy_score(y_true,y_pred))

# 0.3333333
# 0.3333333


texts: Mathematics: Writing down a three-class classifier confusion matrix

texts:
Confusion matrix 2A three-class classifier is evaluated on a test set
of 900 samples which containsall three classes in equal
proportions.
• Classes 2 and 3 are always classified correctly
• Class 1 is confused with class 2 in 50% of the cases,
  and with class 3 in 20% of the cases.

Write down the confusion matrix!

Like the name "confusion matrix" already says, I'm confused.
I'm using a binary classification confusion matrix which looks like that

I show you this matrix, because there are a lot of different looking classification matrices with a different order of the true positives, true negatives and so on.
According to this task, I must use a three-class classifier matrix. So first I divided the 900 Samples to 3, so I get 300 for each class. Then I substracted 50% and 20% of 300 to get 90 samples for the first class.
I read, the row index is for the true class and the column index is, what the classifier says.
My confusion Matrix looks like this:

I think, the confusions of class 1 are false positives.
Did I write the matrix correctly?


texts:
I have worked in classification problems, and stratified cross-validation is one of the most useful and simple techniques I've found. In that case, what it means is to build a training and validation set that have the same prorportions of classes of the target variable.

I am wondering if such an strategy exists in regression. A simple approach would be to split the data in quartiles or deciles and make sure that the proportions of training and validation instances in the respective quartiles and deciles are the same.

The question is: is there a standard way to do this? if so, is there an implementation in sklearn?


texts: Assigning vector of weights to Neural Network

texts:
I am using nnet package in R for training Artificial Neural
Network. Some tell that weights are assigned to each case in training set,
and some say that no. of weights are equal to the no. of connections in
structure of neural network. I didn't understand this. Can anybody help me
out.

I looked at the R code of nnet i.e.

What is the difference between weights & Wts in the nnet code below?

nnet(x, y, weights, size, Wts, mask, linout = FALSE, entropy = FALSE, softmax = FALSE, censored = FALSE, skip = FALSE, rang = 0.7, decay = 0, maxit = 100, Hess = FALSE, trace = TRUE, MaxNWts = 1000, abstol = 1.0e-4, reltol = 1.0e-8, ...)


texts: Global Average Pooling for Multi-label Segmentation

texts: From my understanding, GAP models the image-level classification information. Does it make any sense to use GAP for predicting a segmentation mask when you have more than 1 class? Won't GAP only favour the most frequently occurring class and defeat the purpose of multi-class segmentation? If it does make sense to use GAP, how would you go about doing so?


texts: How can this CNN for the portfolio management problem be implemented in keras?

texts:
In the paper "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem" the CNN in the picture below is applied on the portfolio management problem.

I am trying to understand how this network works and can be implemented in keras.

What I don't understand is, why do we have 20+1 feature maps after the second convolution? And how I am supposed to include the portfolio w from last period in keras?

My current implementation is as follows:

def network(self):
    input1 = Input(shape=(3, 50, 4))
    conv1 = Conv2D(2, (1, 3), activation='relu',)(input1)
    conv2 = Conv2D(10, (1, 48), activation='relu')(conv1)
    conv3 = Conv2D(1, (1, 1), activation='relu')(conv2)
    flat1 = Flatten()(conv3)
    preds = Activation('softmax')(flat1)
    model = Model(input1, preds)
    model.compile(optimizer='rmsprop', loss='mse') # todo loss function
    return model

Thanks for any help.


texts: In Orange, is it possible to analyze or visualize real time, streaming data?

texts: Using orange, I would like to be able to do real time analysis or visualizations on streaming data. I would appreciate any input on the matter!


texts: Why does Lasso behave "erratically" when the number of features is greater than the number of training instances?

texts:
From the book "Hands-on Machine Learning with Scikit-Learn and TensorFlow 2nd edition" chapter 4:

  In general, Elastic Net is preferred over Lasso since Lasso
  may behave erratically when the number of features is greater than the number of
  training instances or when several features are strongly correlated.

How and why exactly does Lasso behave "erratically" when no. of features > no. of training instances or when features are strongly correlated? Does it have to do with redundant features?


texts: Retrieve dropped column names from `sklearn.impute.SimpleImputer`

texts:
The SimpleImputer class takes pandas dataframes and returns unlabeled numpy arrays. Which means that the SimpleImputer drops some features at will, but has no way to communicate which features have been dropped to the caller

I've been trying to come up with a workaround, but they all are extremely hackish and unreliable. Is there something I'm missing?


texts: what is the one hot encoding for cancer data classification

texts:
I am working on a project to classify lung CT dataset using CNN and tensorflow, I know that the order for the category is cancer/no-cancer (only 2 classes), in more than one Github repository I see that they did one hot encoding like the code below:

if label == 1:
    label = np.array([0, 1])
elif label == 0:
    label = np.array([1, 0])

what makes me confused is: 1 means cancer and 0 means no-cancer, as I understand it should be:

if label == 1:
    label = np.array([1, 0])
elif label == 0:
    label = np.array([0, 1])

but why they did one hot encoding like this, I don't know I am wrong or there is another thing that I did not understand, can anyone explain it for? or give me a better way to do encoding for my data, but with code?


texts: Transformer masking during training or inference?

texts:
I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not yet predicted), so it can be used autoregressively during inference.
I don't understand how masking is used during inference. When the encoder is given an unseen sample with no ground truth output or prediction, it seems to me that there is nothing to mask, since there aren't any output tokens beyond what the decoder has already produced. Is my understanding of masking correct?
Thanks!


texts:
Pretty new to Python, but as an SEO I'm looking at the benefits of using notebooks in my workflow.

I've got two excel files which I've cleaned and imported into a new notebook using pandas.

I'm trying to compare position changes and create a new dataframe with new columns to show previous, new, and changes in positions.

Have a look at the screengrabs[! of the data below. Thanks in advance.


texts: Keras Model Predict is not predicting all images flowing from directory?

texts:
I have the following code where I have done all the training and passed the testing set as a flow from directory. After that when I pass that object into the model.predict option, the array received is not of the same length as the test set length. Code:

PATH = '/content/testing'
testGen.reset()
testGen = valAug.flow_from_directory(
    PATH,
    class_mode="categorical",
    target_size=(75, 75),
    color_mode="rgb",
    shuffle=False,
    batch_size=BS)

predIdxs = model1.predict_generator(testGen,
    steps=(totalTest // 32))
print(len(predIdxs))
# for each image in the testing set we need to find the index of the
# label with corresponding largest predicted probability
predIdxs = np.argmax(predIdxs, axis=1)
print(len(predIdxs))
import pandas as pd
from glob import glob
test_df = pd.DataFrame()
id = []
for x in glob('/content/testing/0/*'):
    id.append(x)
for x in glob('/content/testing/1/*'):
    id.append(x)
test_df['id'] = id
test_df['category'] = predIdxs
print(test_df)
test_df.to_csv('submission.csv', index=False)

After that the output I got is as follows:

Found 55505 images belonging to 2 classes.
55488
55488
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-81-09c1488e5ac6> in <module>()
     25     id.append(x)
     26 test_df['id'] = id
---> 27 test_df['category'] = predIdxs
     28 print(test_df)
     29 test_df.to_csv('submission.csv', index=False)

3 frames
/usr/local/lib/python3.6/dist-packages/pandas/core/internals/construction.py in sanitize_index(data, index, copy)
    609
    610     if len(data) != len(index):
--> 611         raise ValueError("Length of values does not match length of index")
    612
    613     if isinstance(data, ABCIndexClass) and not copy:

ValueError: Length of values does not match length of index

Lenght of test set(totalTest = 55505) but only 55488 data is predicted. Why is data lost here?
P.S: The model I have used is a pretrained Inception V3 model where I have downloaded the weights beforehand and run the model. I got about 85% accuracy. And I have tried the same method using Resnet block also and I have received the results without error. Why am I getting an error here? Any help would be appreciated.


texts: How can I measure the reliability of the specificity of a model with very small train, test, and validation datasets?

texts:
Stats newbie here. I have a small dataset of 646 samples that I've trained a reasonably performant model on (~99% test and val accuracy). To complicate things a little bit, the classes are somewhat unbalanced. It's a binary classification problem.

Here is my confusion matrix on training data

[[387   1]
 [  1  73]]

on testing data:

[[74  1]
 [ 0 10]]

on validation data:

[[85  1]
 [ 0 13]]

Training Specificity: .986
Testing Specificity: .909
Validation Specificity: .928

My thoughts are that testing and validation have a very low specificity while training has a comparatively high specificity. However, given that only one sample is missed in both the testing and validation datasets, what is my real-world specificity? Is there a better generalizability measure? Is there something akin to a p-value that relates the reliability of the specificity given the size of the negative sample class?

Thanks!